SOLVED 10Gbe performance issue in FreeNAS 11

mrchisholm

Cadet
Joined
Feb 6, 2017
Messages
5
I should probably explain my setup before explaining my performance issue. I have a server with FreeNAS11 on it and a Mellanox ConnectX-2 card and another server with two Mellanox ConnectX-2 cards bridged via vyOS in Hyper-V on Server 2016 and my Client is a normal gaming rig with a Mellanox ConnectX-2 card in it.

When I check with iperf I get 9Gbit from my FreeNAS to my Client which is just fine but when i have iperf as server on the FreeNAS box and check speed from my client to the FreeNAS box i only get around 3Gbit which bottlenecks my transfers to the storage. All cards are setup with jumbo frames 9000, if i don't change to jumbo frames from the defaults on the FreeNAS box i only get 1.5Gbit on average.

I figured maybe it was a problem with Mellanox cards on FreeNAS so I put in my Chelsio CC2-S320E-SR card in there but it has the same 3Gbit max to the server and from the server i get around 8Gbit.

I installed windows 10 on another disk on the freenas server and when i use the latest mellanox drivers there i get 9Gbit on both ways so it doesn't seem to be a hardware issue since it works in windows but not on FreeNAS.

Both the FreeNAS server and my Bridge server has the S2600CP motherboard, freenas has a E5-2640 v1 and bridge a E5-2670 v2.

EDIT: I updated to the latest stable FreeNAS 11 build as of today and it did not make a difference.
 
Last edited by a moderator:

joeschmuck

Old Man
Moderator
Joined
May 28, 2011
Messages
10,970
A few questions...

How much RAM do you have?
Is FreeNAS running as a VM?
What network protocol are you using?
Have you tried a direct connection? (I doubt this is the problem but I got to ask)
Post the output of dmesg

At face value it sure looks like something is up with FreeNAS since you were able to use Win10 on that same hardware and it worked fine. I would think it were a driver issue as well. Since I don't have the pleasure of 10Gb network at my house, I'm sure someone will chime in with a suggestion or two.
 
Joined
May 10, 2017
Messages
838
Don't have any suggestions but I have the same NIC on FreeNAS and can get close to line speed.
 

mrchisholm

Cadet
Joined
Feb 6, 2017
Messages
5
A few questions...

How much RAM do you have?
Is FreeNAS running as a VM?
What network protocol are you using?
Have you tried a direct connection? (I doubt this is the problem but I got to ask)
Post the output of dmesg

At face value it sure looks like something is up with FreeNAS since you were able to use Win10 on that same hardware and it worked fine. I would think it were a driver issue as well. Since I don't have the pleasure of 10Gb network at my house, I'm sure someone will chime in with a suggestion or two.

Running with 32GB Ram
It's not running as a VM
using SMB3 protocol
I tried a direct connection to my other server and i got the same speed issue there. and as u say since it works fine under win10 i can't see it as a hardware issue but if Johnnie has it working there must be something that's having an affect be it hardware or software.

heres's the rather long log :)

Code:
Copyright (c) 1992-2017 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
		The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.0-STABLE #0 r313908+d7d07647f69(freenas/11.0-stable): Thu Jul 20 19:01:05 UTC 2017
	root@gauntlet:/freenas-11-releng/freenas/_BE/objs/freenas-11-releng/freenas/_BE/os/sys/FreeNAS.a
FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0)
CPU: Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz (2494.04-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x206d7  Family=0x6  Model=0x2d  Stepping=7
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUS
  Features2=0x1fbee3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DC
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  XSAVE Features=0x1<XSAVEOPT>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 35433480192 (33792 MB)
avail memory = 33199476736 (31661 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <INTEL  S2600CP>
FreeBSD/SMP: Multiprocessor System Detected: 12 CPUs
FreeBSD/SMP: 1 package(s) x 6 core(s) x 2 hardware threads
WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
ACPI BIOS Warning (bug): Invalid length for FADT/Pm1aControlBlock: 32, using default 16 (20170303/tb
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
random: entropy device external interface
kbd1 at kbdmux0
nexus0
cryptosoft0: <software crypto> on motherboard
aesni0: <AES-CBC,AES-XTS,AES-GCM,AES-ICM> on motherboard
padlock0: No ACE support.
acpi0: <INTEL S2600CP> on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
cpu4: <ACPI CPU> on acpi0
cpu5: <ACPI CPU> on acpi0
cpu6: <ACPI CPU> on acpi0
cpu7: <ACPI CPU> on acpi0
cpu8: <ACPI CPU> on acpi0
cpu9: <ACPI CPU> on acpi0
cpu10: <ACPI CPU> on acpi0
cpu11: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 350
Event timer "HPET1" frequency 14318180 Hz quality 340
Event timer "HPET2" frequency 14318180 Hz quality 340
Event timer "HPET3" frequency 14318180 Hz quality 340
Event timer "HPET4" frequency 14318180 Hz quality 340
Event timer "HPET5" frequency 14318180 Hz quality 340
Event timer "HPET6" frequency 14318180 Hz quality 340
Event timer "HPET7" frequency 14318180 Hz quality 340
atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
atrtc0: Warning: Couldn't map I/O.
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pcib0: _OSC returned error 0x10
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 47 at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> irq 47 at device 2.0 on pci0
pci2: <ACPI PCI bus> on pcib2
mps0: <Avago Technologies (LSI) SAS2008> port 0x2000-0x20ff mem 0xd2640000-0xd264ffff,0xd2600000-0xd
mps0: Firmware: 20.00.07.00, Driver: 21.01.00.00-fbsd
mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
pcib3: <ACPI PCI-PCI bridge> irq 47 at device 2.2 on pci0
pci3: <ACPI PCI bus> on pcib3
mps1: <Avago Technologies (LSI) SAS2008> port 0x1000-0x10ff mem 0xd2440000-0xd244ffff,0xd2400000-0xd
mps1: Firmware: 20.00.07.00, Driver: 21.01.00.00-fbsd
mps1: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
pcib4: <ACPI PCI-PCI bridge> irq 47 at device 3.0 on pci0
pci4: <ACPI PCI bus> on pcib4
pci4: <network, ethernet> at device 0.0 (no driver attached)
pcib5: <ACPI PCI-PCI bridge> irq 47 at device 3.2 on pci0
pci5: <ACPI PCI bus> on pcib5
cxgbc0: <Chelsio T320, 2 ports> mem 0xd2090000-0xd2090fff,0xd1800000-0xd1ffffff,0xd2080000-0xd2080ff
PHY 0 i2c read of dev.addr a0.0 timed out
PHY 1 i2c read of dev.addr a0.0 timed out
cxgbc0: using MSI-X interrupts (9 vectors)
cxgb0: <Port 0 10GBASE-R> on cxgbc0
cxgb0: Using defaults for TSO: 65518/35/2048
cxgb0: Ethernet address: 00:07:43:07:9e:c3
cxgb1: <Port 1 10GBASE-R> on cxgbc0
cxgb1: Using defaults for TSO: 65518/35/2048
cxgb1: Ethernet address: 00:07:43:07:9e:c4
cxgbc0: Firmware Version 7.11.0
pcib6: <ACPI PCI-PCI bridge> irq 16 at device 17.0 on pci0
pci6: <ACPI PCI bus> on pcib6
pci0: <simple comms> at device 22.0 (no driver attached)
pci0: <simple comms> at device 22.1 (no driver attached)
ehci0: <Intel Patsburg USB 2.0 controller> mem 0xd2720000-0xd27203ff irq 22 at device 26.0 on pci0
usbus0: EHCI version 1.0
usbus0 on ehci0
pcib7: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci7: <ACPI PCI bus> on pcib7
pcib8: <ACPI PCI-PCI bridge> irq 19 at device 28.7 on pci0
pci8: <ACPI PCI bus> on pcib8
vgapci0: <VGA-compatible display> mem 0xfa000000-0xfaffffff,0xd1010000-0xd1013fff,0xd0800000-0xd0fff
vgapci0: Boot video device
ehci1: <Intel Patsburg USB 2.0 controller> mem 0xd2710000-0xd27103ff irq 20 at device 29.0 on pci0
usbus1: EHCI version 1.0
usbus1 on ehci1
pcib9: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci9: <ACPI PCI bus> on pcib9
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
ahci0: <Intel Patsburg AHCI SATA controller> port 0x3070-0x3077,0x3060-0x3063,0x3050-0x3057,0x3040-031.2 on pci0
ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich5: <AHCI channel> at channel 5 on ahci0
acpi_button0: <Sleep Button> on acpi0
pcib10: <ACPI Host-PCI bridge> on acpi0
pci10: <ACPI PCI bus> on pcib10
pci10: <dasp, performance counters> at device 14.1 (no driver attached)
pci10: <dasp, performance counters> at device 19.1 (no driver attached)
pci10: <dasp, performance counters> at device 19.4 (no driver attached)
pci10: <dasp, performance counters> at device 19.5 (no driver attached)
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
ichwd0: <Intel Patsburg watchdog timer> on isa0
ichwd0: ICH WDT present but disabled in BIOS or hardware
device_attach: ichwd0 attach returned 6
ichwd0: <Intel Patsburg watchdog timer> at port 0x430-0x437,0x460-0x47f on isa0
ichwd0: ICH WDT present but disabled in BIOS or hardware
device_attach: ichwd0 attach returned 6
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc9800-0xca7ff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
est1: <Enhanced SpeedStep Frequency Control> on cpu1
coretemp2: <CPU On-Die Thermal Sensors> on cpu2
est2: <Enhanced SpeedStep Frequency Control> on cpu2
coretemp3: <CPU On-Die Thermal Sensors> on cpu3
est3: <Enhanced SpeedStep Frequency Control> on cpu3
coretemp4: <CPU On-Die Thermal Sensors> on cpu4
est4: <Enhanced SpeedStep Frequency Control> on cpu4
coretemp5: <CPU On-Die Thermal Sensors> on cpu5
est5: <Enhanced SpeedStep Frequency Control> on cpu5
coretemp6: <CPU On-Die Thermal Sensors> on cpu6
est6: <Enhanced SpeedStep Frequency Control> on cpu6
coretemp7: <CPU On-Die Thermal Sensors> on cpu7
est7: <Enhanced SpeedStep Frequency Control> on cpu7
coretemp8: <CPU On-Die Thermal Sensors> on cpu8
est8: <Enhanced SpeedStep Frequency Control> on cpu8
coretemp9: <CPU On-Die Thermal Sensors> on cpu9
est9: <Enhanced SpeedStep Frequency Control> on cpu9
coretemp10: <CPU On-Die Thermal Sensors> on cpu10
est10: <Enhanced SpeedStep Frequency Control> on cpu10
coretemp11: <CPU On-Die Thermal Sensors> on cpu11
est11: <Enhanced SpeedStep Frequency Control> on cpu11
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
freenas_sysctl: adding account.
freenas_sysctl: adding directoryservice.
freenas_sysctl: adding middlewared.
freenas_sysctl: adding network.
freenas_sysctl: adding services.
ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled
nvme cam probe device init
usbus0: 480Mbps High Speed USB v2.0
usbus1: 480Mbps High Speed USB v2.0
ugen0.1: <Intel EHCI root HUB> at usbus0
uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0
ugen1.1: <Intel EHCI root HUB> at usbus1
uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
mps0: SAS Address for SATA device = 696e5303a06a7e73
mps0: SAS Address from SATA device = 696e5303a06a7e73
mps0: SAS Address for SATA device = 6a6854039f697f72
mps0: SAS Address from SATA device = 6a6854039f697f72
mps1: SAS Address for SATA device = d2635e46dadabd90
mps1: SAS Address from SATA device = d2635e46dadabd90
mps1: SAS Address for SATA device = d2636859e0bcbc6f
mps1: SAS Address from SATA device = d2636859e0bcbc6f
mps0: SAS Address for SATA device = 6a6d55039f687972
mps0: SAS Address from SATA device = 6a6d55039f687972
ugen1.2: <vendor 0x8087 product 0x0024> at usbus1
uhub2 on uhub1
uhub2: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus1
ugen0.2: <vendor 0x8087 product 0x0024> at usbus0
uhub3 on uhub0
uhub3: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus0
mps0: SAS Address for SATA device = 6a6b55039f687972
mps0: SAS Address from SATA device = 6a6b55039f687972
mps0: SAS Address for SATA device = 637054039f687971
mps0: SAS Address from SATA device = 637054039f687971
mps1: SAS Address for SATA device = d2655353dfc0ba93
mps1: SAS Address from SATA device = d2655353dfc0ba93
mps1: SAS Address for SATA device = d2635e46dab8bb7e
mps1: SAS Address from SATA device = d2635e46dab8bb7e
uhub3: 6 ports with 6 removable, self powered
uhub2: 8 ports with 8 removable, self powered
ugen1.3: <NOVATEK USB Keyboard> at usbus1
ukbd0 on uhub2
ukbd0: <NOVATEK USB Keyboard, class 0/0, rev 1.10/1.04, addr 3> on usbus1
kbd0 at ukbd0
mps0: SAS Address for SATA device = 6a655303a06a7e73
mps0: SAS Address from SATA device = 6a655303a06a7e73
ugen1.4: <American Megatrends Inc. Virtual Keyboard and Mouse> at usbus1
ukbd1 on uhub2
ukbd1: <Keyboard Interface> on usbus1
kbd2 at ukbd1
mps0: SAS Address for SATA device = 3b27412f7f6c5d30
mps0: SAS Address from SATA device = 3b27412f7f6c5d30
mps0: SAS Address for SATA device = 3b27412f7f6c5a4e
mps0: SAS Address from SATA device = 3b27412f7f6c5a4e
ada0 at ahcich5 bus 0 scbus7 target 0 lun 0
ada0: <TOSHIBA THNSNJ128GCSU JULA0101> ACS-2 ATA SATA 3.x device
ada0: Serial Number 54HS10DST8MW
ada0: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 122104MB (250069680 512 byte sectors)
da4 at mps0 bus 0 scbus0 target 7 lun 0
da4: <ATA ST2000DM001-1CH1 CC27> Fixed Direct Access SPC-4 SCSI device
da4: Serial Number Z340F3Y2
da4: 600.000MB/s transfers
da4: Command Queueing enabled
da4: 1907729MB (3907029168 512 byte sectors)
da4: quirks=0x8<4K>
da8 at mps1 bus 0 scbus1 target 6 lun 0
da8: <ATA WDC WD30EFRX-68E 0A82> Fixed Direct Access SPC-4 SCSI device
da8: Serial Number WD-WCC4N1NC204A
da8: 600.000MB/s transfers
da8: Command Queueing enabled
da8: 2861588MB (5860533168 512 byte sectors)
da8: quirks=0x8<4K>
da1 at mps0 bus 0 scbus0 target 4 lun 0
da1: <ATA SAMSUNG HD204UI 0001> Fixed Direct Access SPC-4 SCSI device
da1: Serial Number S2H7J1BB226870
da1: 300.000MB/s transfers
da1: Command Queueing enabled
da1: 1907729MB (3907029168 512 byte sectors)
da1: quirks=0x8<4K>
da6 at mps0 bus 0 scbus0 target 10 lun 0
da6: <ATA SAMSUNG HD204UI 0001> Fixed Direct Access SPC-4 SCSI device
da6: Serial Number S2H7J9CB101603
da6: 300.000MB/s transfers
da6: Command Queueing enabled
da6: 1907729MB (3907029168 512 byte sectors)
da6: quirks=0x8<4K>
da2 at mps0 bus 0 scbus0 target 5 lun 0
da2: <ATA ST2000DM001-1CH1 CC27> Fixed Direct Access SPC-4 SCSI device
da2: Serial Number Z340F3VP
da2: 600.000MB/s transfers
da2: Command Queueing enabled
da2: 1907729MB (3907029168 512 byte sectors)
da2: quirks=0x8<4K>
da7 at mps0 bus 0 scbus0 target 11 lun 0
da7: <ATA SAMSUNG HD204UI 0001> Fixed Direct Access SPC-4 SCSI device
da7: Serial Number S2H7J1BB226869
da7: 300.000MB/s transfers
da7: Command Queueing enabled
da7: 1907729MB (3907029168 512 byte sectors)
da7: quirks=0x8<4K>
da0 at mps0 bus 0 scbus0 target 3 lun 0
da0: <ATA SAMSUNG HD204UI 0001> Fixed Direct Access SPC-4 SCSI device
da0: Serial Number S2H7J1DB101776
da0: 300.000MB/s transfers
da0: Command Queueing enabled
da0: 1907729MB (3907029168 512 byte sectors)
da0: quirks=0x8<4K>
da3 at mps0 bus 0 scbus0 target 6 lun 0
da3: <ATA SAMSUNG HD204UI 0001> Fixed Direct Access SPC-4 SCSI device
da3: Serial Number S2H7J1DB101778
da3: 300.000MB/s transfers
da3: Command Queueing enabled
da3: 1907729MB (3907029168 512 byte sectors)
da3: quirks=0x8<4K>
da5 at mps0 bus 0 scbus0 target 9 lun 0
da5: <ATA SAMSUNG HD204UI 0001> Fixed Direct Access SPC-4 SCSI device
da5: Serial Number S2H7J1CB117773
da5: 300.000MB/s transfers
da5: Command Queueing enabled
da5: 1907729MB (3907029168 512 byte sectors)
da5: quirks=0x8<4K>
da9 at mps1 bus 0 scbus1 target 7 lun 0
da9: <ATA WDC WD30EFRX-68E 0A82> Fixed Direct Access SPC-4 SCSI device
da9: Serial Number WD-WCC4N3CP783V
da9: 600.000MB/s transfers
da9: Command Queueing enabled
da9: 2861588MB (5860533168 512 byte sectors)
da9: quirks=0x8<4K>
da10 at mps1 bus 0 scbus1 target 8 lun 0
da10: <ATA WDC WD30EFRX-68E 0A82> Fixed Direct Access SPC-4 SCSI device
da10: Serial Number WD-WCC4N1XV8452
da10: 600.000MB/s transfers
da10: Command Queueing enabled
da10: 2861588MB (5860533168 512 byte sectors)
da10: quirks=0x8<4K>
da11 at mps1 bus 0 scbus1 target 9 lun 0
da11: <ATA WDC WD30EFRX-68E 0A82> Fixed Direct Access SPC-4 SCSI device
da11: Serial Number WD-WCC4N1NC2R6S
da11: 600.000MB/s transfers
da11: Command Queueing enabled
da11: 2861588MB (5860533168 512 byte sectors)
da11: quirks=0x8<4K>
random: unblocking device.
mlx4_core0: <mlx4_core> mem 0xd2200000-0xd22fffff,0x38007f800000-0x38007fffffff irq 40 at device 0.0
mlx4_core: Initializing mlx4_core: Mellanox ConnectX VPI driver v2.1.6 (Jul 20 2017)

mlx4_en mlx4_core0: Activating port:1
mlxen0: Ethernet address: 00:02:c9:4f:00:ea
mlxen0: link state changed to DOWN
mlx4_en: mlx4_core0: Port 1: Using 12 TX rings
mlx4_en: mlx4_core0: Port 1: Using 8 RX rings
mlx4_en: mlxen0: Using 12 TX rings
mlx4_en: mlxen0: Using 8 RX rings
mlx4_en: mlxen0: Initializing port
SMP: AP CPU #1 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #7 Launched!
SMP: AP CPU #5 Launched!
SMP: AP CPU #10 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #9 Launched!
SMP: AP CPU #6 Launched!
SMP: AP CPU #8 Launched!
SMP: AP CPU #11 Launched!
Timecounter "TSC-low" frequency 1247018984 Hz quality 1000
Trying to mount root from zfs:freenas-boot/ROOT/11.0-U2 []...
GEOM_RAID5: Module loaded, version 1.3.20140711.62 (rev f91e28e40bf7)
ichwd0: <Intel Patsburg watchdog timer> at port 0x430-0x437,0x460-0x47f on isa0
ichwd0: ICH WDT present but disabled in BIOS or hardware
device_attach: ichwd0 attach returned 6
hwpmc: SOFT/16/64/0x67<INT,USR,SYS,REA,WRI> TSC/1/64/0x20<REA> IAP/4/48/0x3ff<INT,USR,SYS,EDG,THR,RE
cxgb0: link state changed to DOWN
uhid0 on uhub2
uhid0: <NOVATEK USB Keyboard, class 0/0, rev 1.10/1.04, addr 3> on usbus1
ums0 on uhub2
ums0: <Mouse Interface> on usbus1
ums0: 3 buttons and [Z] coordinates ID=0
GEOM_ELI: Device da0p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:	 Crypto: hardware
GEOM_ELI: Device da1p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:	 Crypto: hardware
GEOM_ELI: Device da2p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:	 Crypto: hardware
GEOM_ELI: Device da3p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:	 Crypto: hardware
GEOM_ELI: Device da4p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:	 Crypto: hardware
GEOM_ELI: Device da5p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:	 Crypto: hardware
GEOM_ELI: Device da6p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:	 Crypto: hardware
GEOM_ELI: Device da7p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:	 Crypto: hardware
GEOM_ELI: Device da10p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:	 Crypto: hardware
GEOM_ELI: Device da11p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:	 Crypto: hardware
GEOM_ELI: Device da8p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:	 Crypto: hardware
GEOM_ELI: Device da9p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:	 Crypto: hardware
mlxen0: link state changed to UP
mlx4_en: mlxen0: Link Down
mlxen0: link state changed to DOWN
mlx4_en: mlxen0: Link Up
mlxen0: link state changed to UP
root@freenas:~ #


 
Last edited:

joeschmuck

Old Man
Moderator
Joined
May 28, 2011
Messages
10,970
Have you tied to change to SMB2 protocol?

@Johnnie Black Did you have any tunables related to the NIC?
 
Joined
May 10, 2017
Messages
838
@Johnnie Black Did you have any tunables related to the NIC?

Yes, I did, I forgot to mention that, performance without them was considerably worse, I found these on the forum or some other site:

Code:
kern.ipc.maxsockbuf=16777216
net.inet.ip.intr_queue_maxlen=2048
net.inet.tcp.recvbuf_max=16777216
net.inet.tcp.recvspace=4194304
net.inet.tcp.recvbuf_inc=524288
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.sendspace=2097152
net.inet.tcp.sendbuf_inc=32768
net.route.netisr_maxqlen=2048
 

mrchisholm

Cadet
Joined
Feb 6, 2017
Messages
5
So it turns out when i setup my FreeNAS server I checked in autotune, the guide I was following told me it was the best thing to do but it seems it had altered those tunable buffer settings. I put in yours instead and low and behold the speeds improved significantly . :)

to client:
OrIqvS.jpg


from client:
CrLyEX.jpg


so i get an average of around 7.7Gbit up now which deff won't bottleneck my transfers anymore so that's great! i get the odd spike down to 5gbit now and then which seems odd, like the buffer runs out or something?`I tried to copy the send buffer settings to the receive buffer settings but that made it a bit worse, sending averaged 6.6Gbit instead of 7.7.
 
Last edited by a moderator:

joeschmuck

Old Man
Moderator
Joined
May 28, 2011
Messages
10,970
Glad things are working better and @Johnnie Black thanks for posting those tunables.
 

chris crude

Patron
Joined
Oct 13, 2016
Messages
210

mrchisholm

Cadet
Joined
Feb 6, 2017
Messages
5
cool, i'll play with the values see if i can get the recieving part closer to max, thanks everyone for the guidance :)
 

SMnasMAN

Contributor
Joined
Dec 2, 2018
Messages
177
fyi:, I enable one of these 3x tunable from above (not sure which one did it):
net.inet.tcp.sendspace=2097152
net.inet.tcp.sendbuf_inc=32768
net.route.netisr_maxqlen=2048

it made my NICs disappear (no ssh nor web gui). SM x9dr3-ln4f+ MB (so the 4x onboard nics) and a Chelsio tr520. FN 11.2u3.

so be ware, as usual with any tunables (this is a test system so no big deal). a real mess as i dont see anyway to disable these tuanbles from being auto applied at boot up, and i cant get into the web gui (just stuck at login).
 
Last edited:

msbxa

Contributor
Joined
Sep 21, 2014
Messages
151
fyi:, I enable one of these 3x tunable from above (not sure which one did it):
net.inet.tcp.sendspace=2097152
net.inet.tcp.sendbuf_inc=32768
net.route.netisr_maxqlen=2048

it made my NICs disappear (no ssh nor web gui). SM x9dr3-ln4f+ MB (so the 4x onboard nics) and a Chelsio tr520. FN 11.2u3.

so be ware, as usual with any tunables (this is a test system so no big deal). a real mess as i dont see anyway to disable these tuanbles from being auto applied at boot up, and i cant get into the web gui (just stuck at login).

I think its this one (net.inet.tcp.sendspace=2097152) that will course the system to get stuck because of this error message : No buffer space available.
 
Top