SOLVED Only half gigabit on Supermicro A1SAi-2750F

[NerdingByDoing]

I recently upgraded my HP n54l to a custom build using the above board (one that is frequently recommended for FreeNAS) as the n54l had a hard time with an encrypted 6-disk RaidZ2. The upgrade went smoothly; assemble everything and finally plug the old USB stick booting FreeNAS into the new system. Worked right out of the box, so to say. Details of my build are in my signature.

What doesn't work (as expected), however, is networking. Contrary to my expectations, I only get around 500mbit/s on those NICs. I have spent the better part of the last week trying to debug the issue, but to no avail. I consider myself somewhat experienced in the Linux world, albeit my BSD knowledged is limited. Maybe I'm just not seeing something that's right before my eyes for lack of experience. Hopefully someone here can help me. Here's the issue:

# iperf -c freenas
------------------------------------------------------------
Client connecting to freenas, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 4] local 192.168.0.131 port 54562 connected with 192.168.0.99 port 5001
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 526 MBytes 441 Mbits/sec

I see this reflected in all applications. Transfer rates to and from the NAS converge on ~45-55MB/s after the initial spike and simultaneous read and write from an SMB network share pulls that down to values in the single digits at times. I have tried SMB, FTP, NFS, and even SSH (-c none); they all perform differently, but the all cap at ~500mbit/s.
This does not change when I plug NAS and test machine directly together without an intermediate switch. I have used both Cat.5e and Cat.6 cables (the latter trying to configure 1000BaseTX). At the same time, an old core2duo-based FreeBSD test machine I set up for reference performs like a champ, delivering constant gigabit, even under heavy load.

From the looks of it, though, everything is set up correctly.

# ifconfig igb0
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=2400b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6>
ether 0c:c4:7a:ac:80:d2
inet 192.168.0.99 netmask 0xffffff00 broadcast 192.168.0.255
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet 1000baseT <full-duplex>
status: active


Iperf or smbd don't nearly saturate a single core when testing. I've done tests with an up-to-date Arch Linux, using iperf 2.0.9, cygwin version on Windows 7 using iperf 2.0.5 (with pthreads), and FreeBSD 11-RELEASE-p1 using iperf 2.0.9. All results are similar. Meanwhile I can scrub at more than 350MB/s (which is surprising as two of those drives are using 3Gbit/s SATA ports) and iozone throughput tests indicate write rates in the 60-70MB/s range.

I'm at a complete loss by now. It appears like the NICs are being capped at 0.5Gbit/s for some reason. I'll appreciate any help and input.

 

[NerdingByDoing]

I have now tested with six different cables, two of which Cat.6, the rest Cat.5e, with and without a gigabit switch in between.
I have also set up a fresh install of FreeNAS 9.10 just to rule out any misconfiguration on my end. Results were the same, unfortunately.
Tests with an Arch Linux live image, however, revealed a perfectly working gigabit connection, ruling out the board, or a BIOS setting I may have missed as cause.
Since the board is from 2013 and FreeBSD 10.3, the base for FreeNAS 9.10 came out about a year ago, I don't have high hopes on igb driver updates, come the next version. I'll try installing FreeBSD 11 on a USB stick tomorrow and will report back with tests, just to check anyway.

I may be under the wrong impression that the board is widely used among FreeNAS users, but since it's on the recommended hardware list, I was hoping for at least a few people running a setup similar to mine who could maybe provide some input on how they tuned it. I must be overlooking something very obvious if I'm the only person having problems like these.

Meanwhile here's my system's dmesg output after a fresh boot and some other hopefully helpful data:

Show : dmesg

Copyright (c) 1992-2016 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 10.3-STABLE #0 41eb257(9.10.2-STABLE): Mon Mar 6 17:03:14 UTC 2017
root@gauntlet:/freenas-9.10-releng/_BE/objs/freenas-9.10-releng/_BE/os/sys/FreeNAS.amd64 amd64
FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
CPU: Intel(R) Atom(TM) CPU C2750 @ 2.40GHz (2400.07-MHz K8-class CPU)
Origin="GenuineIntel" Id=0x406d8 Family=0x6 Model=0x4d Stepping=8
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x43d8e3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,MOVBE,POPCNT,TSCDLT,AESNI,RDRAND>
AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
AMD Features2=0x101<LAHF,Prefetch>
Structured Extended Features=0x2282<TSCADJ,SMEP,ERMS,NFPUSG>
VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
TSC: P-state invariant, performance statistics
real memory = 36507222016 (34816 MB)
avail memory = 33232211968 (31692 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <INTEL TIANO >
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
FreeBSD/SMP: 1 package(s) x 8 core(s)
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 2
cpu2 (AP): APIC ID: 4
cpu3 (AP): APIC ID: 6
cpu4 (AP): APIC ID: 8
cpu5 (AP): APIC ID: 10
cpu6 (AP): APIC ID: 12
cpu7 (AP): APIC ID: 14
random: <Software, Yarrow> initialized
WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
ioapic0 <Version 2.0> irqs 0-23 on motherboard
kbd1 at kbdmux0
module_register_init: MOD_LOAD (vesa, 0xffffffff80e48970, 0) error 19
aesni0: <AES-CBC,AES-XTS> on motherboard
padlock0: No ACE support.
cryptosoft0: <software crypto> on motherboard
acpi0: <ALASKA A M I > on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
cpu4: <ACPI CPU> on acpi0
cpu5: <ACPI CPU> on acpi0
cpu6: <ACPI CPU> on acpi0
cpu7: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 350
Event timer "HPET1" frequency 14318180 Hz quality 340
Event timer "HPET2" frequency 14318180 Hz quality 340
atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
atrtc0: Warning: Couldn't map I/O.
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> mem 0xdf3c0000-0xdf3dffff irq 16 at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> at device 0.0 on pci1
pci2: <ACPI PCI bus> on pcib2
vgapci0: <VGA-compatible display> port 0xd000-0xd07f mem 0xde000000-0xdeffffff,0xdf000000-0xdf01ffff irq 16 at device 0.0 on pci2
vgapci0: Boot video device
pcib3: <ACPI PCI-PCI bridge> mem 0xdf3a0000-0xdf3bffff irq 16 at device 2.0 on pci0
pci3: <ACPI PCI bus> on pcib3
xhci0: <XHCI (generic) USB 3.0 controller> mem 0xdf200000-0xdf201fff irq 17 at device 0.0 on pci3
xhci0: 64 bytes context size, 32-bit DMA
usbus0 on xhci0
pcib4: <ACPI PCI-PCI bridge> mem 0xdf380000-0xdf39ffff irq 20 at device 3.0 on pci0
pci4: <ACPI PCI bus> on pcib4
ahci0: <ASMedia ASM1061 AHCI SATA controller> port 0xc050-0xc057,0xc040-0xc043,0xc030-0xc037,0xc020-0xc023,0xc000-0xc01f mem 0xdf110000-0xdf1101ff irq 22 at device 0.0 on pci4
ahci0: AHCI v1.20 with 2 6Gbps ports, Port Multiplier supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
pci0: <base peripheral, IOMMU> at device 15.0 (no driver attached)
igb0: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0xe0c0-0xe0df mem 0xdf360000-0xdf37ffff,0xdf3ec000-0xdf3effff irq 20 at device 20.0 on pci0
igb0: Using MSIX interrupts with 9 vectors
igb0: Ethernet address: *REMOVED*
igb0: Bound queue 0 to cpu 0
igb0: Bound queue 1 to cpu 1
igb0: Bound queue 2 to cpu 2
igb0: Bound queue 3 to cpu 3
igb0: Bound queue 4 to cpu 4
igb0: Bound queue 5 to cpu 5
igb0: Bound queue 6 to cpu 6
igb0: Bound queue 7 to cpu 7
igb1: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0xe0a0-0xe0bf mem 0xdf340000-0xdf35ffff,0xdf3e8000-0xdf3ebfff irq 21 at device 20.1 on pci0
igb1: Using MSIX interrupts with 9 vectors
igb1: Ethernet address: *REMOVED*
igb1: Bound queue 0 to cpu 0
igb1: Bound queue 1 to cpu 1
igb1: Bound queue 2 to cpu 2
igb1: Bound queue 3 to cpu 3
igb1: Bound queue 4 to cpu 4
igb1: Bound queue 5 to cpu 5
igb1: Bound queue 6 to cpu 6
igb1: Bound queue 7 to cpu 7
igb2: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0xe080-0xe09f mem 0xdf320000-0xdf33ffff,0xdf3e4000-0xdf3e7fff irq 22 at device 20.2 on pci0
igb2: Using MSIX interrupts with 9 vectors
igb2: Ethernet address: *REMOVED*
igb2: Bound queue 0 to cpu 0
igb2: Bound queue 1 to cpu 1
igb2: Bound queue 2 to cpu 2
igb2: Bound queue 3 to cpu 3
igb2: Bound queue 4 to cpu 4
igb2: Bound queue 5 to cpu 5
igb2: Bound queue 6 to cpu 6
igb2: Bound queue 7 to cpu 7
igb3: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0xe060-0xe07f mem 0xdf300000-0xdf31ffff,0xdf3e0000-0xdf3e3fff irq 23 at device 20.3 on pci0
igb3: Using MSIX interrupts with 9 vectors
igb3: Ethernet address: *REMOVED*
igb3: Bound queue 0 to cpu 0
igb3: Bound queue 1 to cpu 1
igb3: Bound queue 2 to cpu 2
igb3: Bound queue 3 to cpu 3
igb3: Bound queue 4 to cpu 4
igb3: Bound queue 5 to cpu 5
igb3: Bound queue 6 to cpu 6
igb3: Bound queue 7 to cpu 7
ehci0: <Intel Avoton USB 2.0 controller> mem 0xdf3f3000-0xdf3f33ff irq 23 at device 22.0 on pci0
usbus1: EHCI version 1.0
usbus1 on ehci0
ahci1: <Intel Avoton AHCI SATA controller> port 0xe150-0xe157,0xe140-0xe143,0xe130-0xe137,0xe120-0xe123,0xe040-0xe05f mem 0xdf3f2000-0xdf3f27ff irq 19 at device 23.0 on pci0
ahci1: AHCI v1.30 with 4 3Gbps ports, Port Multiplier not supported
ahcich2: <AHCI channel> at channel 0 on ahci1
ahcich3: <AHCI channel> at channel 1 on ahci1
ahcich4: <AHCI channel> at channel 2 on ahci1
ahcich5: <AHCI channel> at channel 3 on ahci1
ahci2: <Intel Avoton AHCI SATA controller> port 0xe110-0xe117,0xe100-0xe103,0xe0f0-0xe0f7,0xe0e0-0xe0e3,0xe020-0xe03f mem 0xdf3f1000-0xdf3f17ff irq 19 at device 24.0 on pci0
ahci2: AHCI v1.30 with 2 6Gbps ports, Port Multiplier not supported
ahcich6: <AHCI channel> at channel 0 on ahci2
ahcich7: <AHCI channel> at channel 1 on ahci2
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
ichwd0: <Intel Avoton/Rangeley SoC watchdog timer> on isa0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xcffff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: CGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3d0-0x3db iomem 0xb8000-0xbffff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
est1: <Enhanced SpeedStep Frequency Control> on cpu1
coretemp2: <CPU On-Die Thermal Sensors> on cpu2
est2: <Enhanced SpeedStep Frequency Control> on cpu2
coretemp3: <CPU On-Die Thermal Sensors> on cpu3
est3: <Enhanced SpeedStep Frequency Control> on cpu3
coretemp4: <CPU On-Die Thermal Sensors> on cpu4
est4: <Enhanced SpeedStep Frequency Control> on cpu4
coretemp5: <CPU On-Die Thermal Sensors> on cpu5
est5: <Enhanced SpeedStep Frequency Control> on cpu5
coretemp6: <CPU On-Die Thermal Sensors> on cpu6
est6: <Enhanced SpeedStep Frequency Control> on cpu6
coretemp7: <CPU On-Die Thermal Sensors> on cpu7
est7: <Enhanced SpeedStep Frequency Control> on cpu7
random: unblocking device.
usbus0: 5.0Gbps Super Speed USB v3.0
usbus1: 480Mbps High Speed USB v2.0
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled
ugen1.1: <Intel> at usbus1
uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
ugen0.1: <0x1912> at usbus0
uhub1: <0x1912 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
uhub1: 8 ports with 8 removable, self powered
ugen0.2: <SanDisk> at usbus0
umass0: <SanDisk Cruzer Edge, class 0/0, rev 2.00/1.26, addr 1> on usbus0
umass0: SCSI over Bulk-Only; quirks = 0x8100
umass0:9:0:-1: Attached to scbus9
uhub0: 8 ports with 8 removable, self powered
ugen1.2: <vendor 0x8087> at usbus1
uhub2: <vendor 0x8087 product 0x07db, class 9/0, rev 2.00/0.02, addr 2> on usbus1
uhub2: 4 ports with 4 removable, self powered
ugen1.3: <vendor 0x0557> at usbus1
uhub3: <vendor 0x0557 product 0x7000, class 9/0, rev 2.00/0.00, addr 3> on usbus1
uhub3: 4 ports with 3 removable, self powered
ugen1.4: <vendor 0x0557> at usbus1
ukbd0: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00, addr 4> on usbus1
kbd2 at ukbd0
ada0 at ahcich1 bus 0 scbus1 target 0 lun 0
ada0: <INTEL SSDSA2CT040G3 4PC10362> ATA8-ACS SATA 2.x device
ada0: Serial Number *REMOVED*
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 38166MB (78165360 512 byte sectors)
ada0: Previously was known as ad6
ada1 at ahcich2 bus 0 scbus2 target 0 lun 0
ada1: <WDC WD40EFRX-68WT0N0 80.00A80> ACS-2 ATA SATA 3.x device
ada1: Serial Number *REMOVED*
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 3815447MB (7814037168 512 byte sectors)
ada1: quirks=0x1<4K>
ada1: Previously was known as ad8
ada2 at ahcich3 bus 0 scbus3 target 0 lun 0
ada2: <WDC WD40EFRX-68WT0N0 80.00A80> ACS-2 ATA SATA 3.x device
ada2: Serial Number *REMOVED*
ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada2: Command Queueing enabled
ada2: 3815447MB (7814037168 512 byte sectors)
ada2: quirks=0x1<4K>
ada2: Previously was known as ad10
ada3 at ahcich4 bus 0 scbus4 target 0 lun 0
ada3: <WDC WD40EFRX-68WT0N0 80.00A80> ACS-2 ATA SATA 3.x device
ada3: Serial Number *REMOVED*
ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada3: Command Queueing enabled
ada3: 3815447MB (7814037168 512 byte sectors)
ada3: quirks=0x1<4K>
ada3: Previously was known as ad12
ada4 at ahcich5 bus 0 scbus5 target 0 lun 0
ada4: <WDC WD40EFRX-68N32N0 82.00A82> ACS-3 ATA SATA 3.x device
ada4: Serial Number *REMOVED*
ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada4: Command Queueing enabled
ada4: 3815447MB (7814037168 512 byte sectors)
ada4: quirks=0x1<4K>
ada4: Previously was known as ad14
ada5 at ahcich6 bus 0 scbus6 target 0 lun 0
ada5: <WDC WD40EFRX-68WT0N0 80.00A80> ACS-2 ATA SATA 3.x device
ada5: Serial Number *REMOVED*
ada5: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada5: Command Queueing enabled
ada5: 3815447MB (7814037168 512 byte sectors)
ada5: quirks=0x1<4K>
ada5: Previously was known as ad16
ada6 at ahcich7 bus 0 scbus7 target 0 lun 0
ada6: <WDC WD40EFRX-68WT0N0 80.00A80> ACS-2 ATA SATA 3.x device
ada6: Serial Number WD-WCC4E3X6RSPF
ada6: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada6: Command Queueing enabled
ada6: 3815447MB (7814037168 512 byte sectors)
ada6: quirks=0x1<4K>
ada6: Previously was known as ad18
da0 at umass-sim0 bus 0 scbus9 target 0 lun 0
da0: <SanDisk Cruzer Edge 1.26> Fixed Direct Access SPC-4 SCSI device
da0: Serial Number 2005486132029DA204D7
da0: 40.000MB/s transfers
da0: 7633MB (15633408 512 byte sectors)
da0: quirks=0x2<NO_6_BYTE>
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #5 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #7 Launched!
SMP: AP CPU #6 Launched!
Timecounter "TSC-low" frequency 1200033036 Hz quality 1000
Trying to mount root from zfs:freenas-boot/ROOT/9.10.2-U2 []...
GEOM_RAID5: Module loaded, version 1.3.20140711.62 (rev f91e28e40bf7)
ipmi0: <IPMI System Interface> port 0xca2,0xca3 on acpi0
ipmi0: KCS mode found at io 0xca2 on acpi
ipmi0: IPMI device rev. 1, firmware rev. 2.23, version 2.0
ipmi0: Number of channels 2
ipmi0: Attached watchdog
hwpmc: SOFT/16/64/0x67<INT,USR,SYS,REA,WRI> TSC/1/64/0x20<REA> IAP/2/40/0x3ff<INT,USR,SYS,EDG,THR,REA,WRI,INV,QUA,PRC> IAF/3/40/0x67<INT,USR,SYS,REA,WRI>
igb0: link state changed to UP
ums0: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00, addr 4> on usbus1
ums0: 3 buttons and [Z] coordinates ID=0
GEOM_ELI: Device ada0p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI: Crypto: hardware
GEOM_ELI: Device ada6p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI: Crypto: hardware
GEOM_ELI: Device ada2p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI: Crypto: hardware
GEOM_ELI: Device ada1p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI: Crypto: hardware
GEOM_ELI: Device ada5p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI: Crypto: hardware
GEOM_ELI: Device ada4p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI: Crypto: hardware
GEOM_ELI: Device ada3p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI: Crypto: hardware
bridge0: Ethernet address: *REMOVED*
igb0: promiscuous mode enabled
bridge0: link state changed to UP
epair0a: Ethernet address: *REMOVED*
epair0b: Ethernet address: *REMOVED*
epair0a: link state changed to UP
epair0b: link state changed to UP
igb0: link state changed to DOWN
epair0a: promiscuous mode enabled
ng_ether_ifnet_arrival_event: can't re-name node epair0b
igb0: link state changed to UP
epair1a: Ethernet address: *REMOVED*
epair1b: Ethernet address: *REMOVED*
epair1a: link state changed to UP
epair1b: link state changed to UP
epair1a: promiscuous mode enabled
ng_ether_ifnet_arrival_event: can't re-name node epair1b
epair2a: Ethernet address: *REMOVED*
epair2b: Ethernet address: *REMOVED*
epair2a: link state changed to UP
epair2b: link state changed to UP
epair2a: promiscuous mode enabled
ng_ether_ifnet_arrival_event: can't re-name node epair2b

Show : pciconf -lv

00:00.0 Host bridge: Intel Corporation Atom processor C2000 SoC Transaction Router (rev 02)
Flags: bus master, fast devsel, latency 0

00:01.0 PCI bridge: Intel Corporation Atom processor C2000 PCIe Root Port 1 (rev 02) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 16
Memory at df3c0000 (64-bit, non-prefetchable)
Bus: primary=00, secondary=01, subordinate=02, sec-latency=0
I/O behind bridge: 0000d000-0000dfff
Memory behind bridge: de000000-df0fffff
Capabilities: [40] Express Root Port (Slot+), MSI 00
Capabilities: [80] Power Management version 3
Capabilities: [88] Subsystem: Intel Corporation Device 7270
Capabilities: [90] MSI: Enable- Count=1/1 Maskable+ 64bit-

00:02.0 PCI bridge: Intel Corporation Atom processor C2000 PCIe Root Port 2 (rev 02) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 16
Memory at df3a0000 (64-bit, non-prefetchable)
Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
Memory behind bridge: df200000-df2fffff
Capabilities: [40] Express Root Port (Slot+), MSI 00
Capabilities: [80] Power Management version 3
Capabilities: [88] Subsystem: Intel Corporation Device 7270
Capabilities: [90] MSI: Enable- Count=1/1 Maskable+ 64bit-

00:03.0 PCI bridge: Intel Corporation Atom processor C2000 PCIe Root Port 3 (rev 02) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 20
Memory at df380000 (64-bit, non-prefetchable)
Bus: primary=00, secondary=04, subordinate=04, sec-latency=0
I/O behind bridge: 0000c000-0000cfff
Memory behind bridge: df100000-df1fffff
Capabilities: [40] Express Root Port (Slot+), MSI 00
Capabilities: [80] Power Management version 3
Capabilities: [88] Subsystem: Intel Corporation Device 7270
Capabilities: [90] MSI: Enable- Count=1/1 Maskable+ 64bit-

00:0e.0 Host bridge: Intel Corporation Atom processor C2000 RAS (rev 02)
Subsystem: Intel Corporation Device 0000
Flags: fast devsel
Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

00:0f.0 IOMMU: Intel Corporation Atom processor C2000 RCEC (rev 02)
Subsystem: Intel Corporation Device 0000
Flags: bus master, fast devsel, latency 0, IRQ 16
Capabilities: [40] Express Root Complex Event Collector, MSI 00
Capabilities: [80] Power Management version 3
Capabilities: [90] MSI: Enable- Count=1/1 Maskable+ 64bit-

00:13.0 System peripheral: Intel Corporation Atom processor C2000 SMBus 2.0 (rev 02)
Subsystem: Intel Corporation Device 0000
Flags: bus master, fast devsel, latency 0, IRQ 16
Memory at df3f4000 (64-bit, non-prefetchable)
Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00
Capabilities: [80] Power Management version 3
Capabilities: [8c] MSI: Enable- Count=1/1 Maskable+ 64bit+

00:14.0 Ethernet controller: Intel Corporation Ethernet Connection I354 (rev 03)
Subsystem: Super Micro Computer Inc Device 1f41
Flags: bus master, fast devsel, latency 0, IRQ 20
Memory at df360000 (64-bit, non-prefetchable)
I/O ports at e0c0
Memory at df3ec000 (64-bit, non-prefetchable)
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [70] MSI-X: Enable+ Count=10 Masked-
Capabilities: [a0] Express Root Complex Integrated Endpoint, MSI 00

00:14.1 Ethernet controller: Intel Corporation Ethernet Connection I354 (rev 03)
Subsystem: Super Micro Computer Inc Device 1f41
Flags: bus master, fast devsel, latency 0, IRQ 21
Memory at df340000 (64-bit, non-prefetchable)
I/O ports at e0a0
Memory at df3e8000 (64-bit, non-prefetchable)
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [70] MSI-X: Enable+ Count=10 Masked-
Capabilities: [a0] Express Root Complex Integrated Endpoint, MSI 00

00:14.2 Ethernet controller: Intel Corporation Ethernet Connection I354 (rev 03)
Subsystem: Super Micro Computer Inc Device 1f41
Flags: bus master, fast devsel, latency 0, IRQ 22
Memory at df320000 (64-bit, non-prefetchable)
I/O ports at e080
Memory at df3e4000 (64-bit, non-prefetchable)
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [70] MSI-X: Enable+ Count=10 Masked-
Capabilities: [a0] Express Root Complex Integrated Endpoint, MSI 00

00:14.3 Ethernet controller: Intel Corporation Ethernet Connection I354 (rev 03)
Subsystem: Super Micro Computer Inc Device 1f41
Flags: bus master, fast devsel, latency 0, IRQ 23
Memory at df300000 (64-bit, non-prefetchable)
I/O ports at e060
Memory at df3e0000 (64-bit, non-prefetchable)
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [70] MSI-X: Enable+ Count=10 Masked-
Capabilities: [a0] Express Root Complex Integrated Endpoint, MSI 00

00:16.0 USB controller: Intel Corporation Atom processor C2000 USB Enhanced Host Controller (rev 02) (prog-if 20 [EHCI])
Subsystem: Intel Corporation Device 7270
Flags: bus master, medium devsel, latency 0, IRQ 23
Memory at df3f3000 (32-bit, non-prefetchable)
Capabilities: [50] Power Management version 3
Capabilities: [58] Debug port: BAR=1 offset=00a0
Capabilities: [98] PCI Advanced Features

00:17.0 SATA controller: Intel Corporation Atom processor C2000 AHCI SATA2 Controller (rev 02) (prog-if 01 [AHCI 1.0])
Subsystem: Intel Corporation Device 7270
Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 19
I/O ports at e150
I/O ports at e140
I/O ports at e130
I/O ports at e120
I/O ports at e040
Memory at df3f2000 (32-bit, non-prefetchable)
Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
Capabilities: [70] Power Management version 3
Capabilities: [a8] SATA HBA v1.0

00:18.0 SATA controller: Intel Corporation Atom processor C2000 AHCI SATA3 Controller (rev 02) (prog-if 01 [AHCI 1.0])
Subsystem: Intel Corporation Device 7270
Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 19
I/O ports at e110
I/O ports at e100
I/O ports at e0f0
I/O ports at e0e0
I/O ports at e020
Memory at df3f1000 (32-bit, non-prefetchable)
Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
Capabilities: [70] Power Management version 3
Capabilities: [a8] SATA HBA v1.0

00:1f.0 ISA bridge: Intel Corporation Atom processor C2000 PCU (rev 02)
Subsystem: Intel Corporation Device 7270
Flags: bus master, medium devsel, latency 0
Capabilities: [e0] Vendor Specific Information: Len=0c <?>

00:1f.3 SMBus: Intel Corporation Atom processor C2000 PCU SMBus (rev 02)
Subsystem: Intel Corporation Device 7270
Flags: medium devsel, IRQ 18
Memory at df3f0000 (32-bit, non-prefetchable)
I/O ports at e000

01:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 03) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=01, secondary=02, subordinate=02, sec-latency=32
I/O behind bridge: 0000d000-0000dfff
Memory behind bridge: de000000-df0fffff
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [78] Power Management version 3
Capabilities: [80] Express PCI-Express to PCI/PCI-X Bridge, MSI 00
Capabilities: [a4] Subsystem: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge

02:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 30) (prog-if 00 [VGA controller])
Subsystem: Super Micro Computer Inc Device 0813
Flags: bus master, medium devsel, latency 0, IRQ 16
Memory at de000000 (32-bit, non-prefetchable)
Memory at df000000 (32-bit, non-prefetchable)
I/O ports at d000
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/4 Maskable- 64bit+

03:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03) (prog-if 30 [XHCI])
Subsystem: Super Micro Computer Inc Device 0813
Flags: bus master, fast devsel, latency 0, IRQ 17
Memory at df200000 (64-bit, non-prefetchable)
Capabilities: [50] Power Management version 3
Capabilities: [70] MSI: Enable+ Count=1/8 Maskable- 64bit+
Capabilities: [90] MSI-X: Enable- Count=8 Masked-
Capabilities: [a0] Express Endpoint, MSI 00

04:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) (prog-if 01 [AHCI 1.0])
Subsystem: ASMedia Technology Inc. Device 1060
Flags: bus master, fast devsel, latency 0, IRQ 22
I/O ports at c050
I/O ports at c040
I/O ports at c030
I/O ports at c020
I/O ports at c000
Memory at df110000 (32-bit, non-prefetchable)
Expansion ROM at df100000 [disabled]
Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit-
Capabilities: [78] Power Management version 3
Capabilities: [80] Express Legacy Endpoint, MSI 00

sysctl -a

Any input is appreciated.

 

[Ericloewe]

This is weird. It may be a driver regression in recent builds. Definitely try some other versions of FreeNAS/FreeBSD (the FreeNAS nightlies are currently on FreeBSD 11, so I'd try those as a first step).
That should give us a better idea of what's going on.

 

[millst]

I have a A1SRi-2758F and get ~110MB/s copying over SMB. I didn't do any particular tuning other than the sticky at the top of the Sharing section, which helps with browsing speed. It pretty much worked great out of the box.

-tm

 

[Ericloewe]

I have a A1SRi-2758F and get ~110MB/s copying over SMB. I didn't do any particular tuning other than the sticky at the top of the Sharing section, which helps with browsing speed. It pretty much worked great out of the box.

-tm

Which version of FreeNAS?

 

[millst]

FreeNAS-9.10.2-U2 (e1497f2)

To NAS
Speed : 79254984 Bytes/sec.
Speed : 4535.006 MegaBytes/min.
Ended : Tuesday, April 18, 2017 8:47:27 PM

From NAS
Speed : 118358132 Bytes/sec.
Speed : 6772.506 MegaBytes/min.
Ended : Tuesday, April 18, 2017 8:48:43 PM

-tm

 

[NerdingByDoing]

I have a A1SRi-2758F and get ~110MB/s copying over SMB. I didn't do any particular tuning other than the sticky at the top of the Sharing section, which helps with browsing speed. It pretty much worked great out of the box.

-tm

That's practically the same board. May I ask your exact RAM configuration? Just to rule out any weird hardware combination issues FreeBSD might have. I'll try a nightly tonight. Now running 9.10.2-STABLE. Maybe it really is just that.

 

[millst]

I just upgraded to 32GB since I had to tear down and rebuild the system to RMA it for the C2000 bug. I had the same read performance with 16GB. I'm not sure about the write performance. It's a mirror, so that might explain the lower number, but I'm a little suspicious. Another possibility, SMB performance generally sucks compared to NFS (work experience).

-tm

 

[NerdingByDoing]

More testing:

• unplugged all HDDs + SATA expansion card & SSD -> no avail
  
• removed all memory banks, tried individual modules one-by-one -> no avail
  
• booted Freenas Corral-RELEASE, based on FreeBSD 11.0-STABLE

Took me a while to find out how to get to a shell from corral's CLI
Iperf now reaches about 850Mbit/s but only every n-th attempt:

# iperf -c freenas
------------------------------------------------------------
Client connecting to freenas, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 4] local 192.168.0.131 port 42444 connected with 192.168.0.99 port 5001
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 580 MBytes 486 Mbits/sec

# iperf -c freenas
------------------------------------------------------------
Client connecting to freenas, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 4] local 192.168.0.131 port 42788 connected with 192.168.0.99 port 5001
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 901 MBytes 756 Mbits/sec

# iperf -c freenas
------------------------------------------------------------
Client connecting to freenas, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 4] local 192.168.0.131 port 42790 connected with 192.168.0.99 port 5001
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 970 MBytes 814 Mbits/sec

# iperf -c freenas
------------------------------------------------------------
Client connecting to freenas, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 4] local 192.168.0.131 port 42792 connected with 192.168.0.99 port 5001
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 1010 MBytes 847 Mbits/sec

# iperf -c freenas
------------------------------------------------------------
Client connecting to freenas, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 4] local 192.168.0.131 port 42794 connected with 192.168.0.99 port 5001
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 885 MBytes 742 Mbits/sec


Again unsure how to interpret this.
Meanwhile Google stumbled me on the fact that @MrToddsFriends runs pretty much the exact same build I do now, almost with identical RAM modules, even. Maybe we can compare benchmarks.
@millst I was asking for your RAM brand and model, sorry. I don't have another DDR3 SODIMM module to test if that is the culprit. One way to find out is by exclusion :)

 

[millst]

Just Kingston value RAM, 4x8. It's not even on the approved list.

-tm

 

[NerdingByDoing]

I think I'm getting closer.

A colleague of mine suggested to try OpenBSD, pretty much just for the lulz. That I did (OpenBSD 6.1), and the gigabit link worked as expected, right away.
I noticed at that point that OpenBSD loads em instead of igb. So I thought I might simply prevent FreeNAS from loading igb. Apparently, though, network drivers are compiled into FreeNAS' kernel, so no dice (unless I'm not aware of a piece of FreeBSD in-kernel driver voodoo, in that case, please let me know).

I went on digging and again, I'm confused:
The network chipset on the A1SAi-2750F is an Intel i354 (rev 03 in my particular case). This should be the same as in the A1SRi-2758F that @millst owns (incidentally, could you check your chipset revision?). However, neither the igb, nor the em manpages in FreeNAS 9.10.2 suggest their respective driver supports the i354. So i googled this and came up with this, suggesting i354 in FreeBSD is supported by em, and this, suggesting i354 in FreeBSD is supported by igb. Both pertaining to the latest version of CURRENT, I suspect.

So here's what I'm looking for now:

1. Can I prevent FreeNAS from loading igb and load em instead?
2. Can I definitely find out whether em or igb is the right driver?
   
3. Could I build a more recent version of either driver myself (on a separate machine would be fine) and install it in FreeNAS?
4. Could anyone else with this board post their network chipset's revision, loaded driver, and network throughput, please?
5. If this is a bug, is it a bug in FreeNAS or FreeBSD? Where should I look for further help?

 

[Ericloewe]

Okay, now I'm seriously confused.

I was certain that some controllers used igb and the others em. You might want to file a bug report and see if iX can clear this up with Intel.

 

[NerdingByDoing]

It would appear that the problem is just an outdated driver.
I tested a clean FreeBSD 11 installation today, and—much as in the latest OpenBSD version—things just immediately worked.

I'm still puzzled as to why I am the only one experiencing this and all other users of the A1SAi-2750f (a board from 2013!) and its sibling models, whose numbers I imagine are quite substantial, aren't affected at all.

So my two choices now are either to wait until a FreeBSD 11-based version of FreeNAS hits stable (the option I'll likely go for), or hit the bleeding edge and start experimenting with nightlies. Not really looking forward to the latter.

Is there even the least chance for me to backport the newer driver to FreeBSD 10.3? Would this still warrant a bug report with iX? And what would you guys think ist the ETA for a FreeBSD 11-based stable FreeNAS build?

 

[Ericloewe]

So my two choices now are either to wait until a FreeBSD 11-based version of FreeNAS hits stable (the option I'll likely go for)

That'll be nine days, if all goes well.

Is there even the least chance for me to backport the newer driver to FreeBSD 10.3? Would this still warrant a bug report with iX?

Not going to happen, it would take longer than to just wait for FreeNAS 11.

 

[NerdingByDoing]

That'll be nine days, if all goes well.

Oh really? That's awesome.
It never occurred to me to google a roadmap. I'll try the most recent nightly as a test then, just to make sure. Since the 11-based version of Corral I tested didn't perform according to expectation, maybe I can throw in a last minute bug report if this one also won't.
Thanks for the help.

 

[Ericloewe]

FreeNAS 11 uses a FreeBSD release slightly ahead of Corral. It was frozen about a week or two ago.

 

[NerdingByDoing]

You're right. The latest FreeNAS 11 nightly manages a solid gigabit on those NICs. I'm still puzzled how the board is four years old and I appear to be the only one having these problems. But seeing that a solution isn't far ahead, I've marked the thread as solved. Thanks for the help.

 

[Ericloewe]

It is very weird indeed...