UPDATE: Poor software iSCSI and NFS latencies on Chelsio T420-CR

[Jason Keller]

Problem

I have a particularly vexing issue between my ESXi hosts and my FreeNAS 9.3 box. I am getting 500Mbit + latency hits upward of 50-100ms from ESXi > FreeNAS but almost 6Gbit from FreeNAS > ESXi.

When using a Windows 2012 R2 iSCSI initiator as a VM within the host (using normal host networking) I see very bursty throughputs both ways (sometimes it'll burst up to 6-7Gbit and immediately crash down to 500Mbit), but the throughput is still around 2Gbit average from Windows VM > FreeNAS. This is the big issue that has me wishing FreeNAS had SAS target support.

Any and all help appreciated! FYI before anyone says anything about sync I have it disabled (and I know full well the ramifications - looking at you Cyberjock!). I've been using ZFS in production for 4+ years so I am not new at all to ZFS. LZ4 compression default, 64k blocksize on the ZVOL. 512 byte standard blocks in iSCSI. MTU I've tried both ways 1500 and 9000, little if any difference.

FreeNAS head...
IBM X3650 M3
USB booted
BuildFreeNAS-9.3-STABLE-201503270027
Platform2x Intel(R) Xeon(R) CPU E5540 @ 2.53GHz
Memory49114MB
Chelsio T420-CR Dual-port 10Gbit SFP+ Unified Wire Accelerator
2x Avago LSI 9207-8i HBAs direct wired to backplanes (no expander)
10x 300GB 10000 RPM SAS drives and 2x 600GB 10000 RPM SAS drives all in standard mirror vdevs (600GB drives sized down to 300GB to match the others). Perftest attached.

2x ESXi 5.5 U2 hosts (latest patches installed)
HP DL380 G6 and HP DL380 G7
USB booted
2x Intel Xeon CPU E5649 @ 2.53GHz
72GB memory
Chelsio T420-CR Dual-port 10Gbit SFP+ Unified Wire Accelerator
8x 146GB 15000 RPM SAS disks in RAID10


Hosts have latest Chelsio firmware/drivers and are direct attached to the Freenas head via 10G twinaxial. ESXi hosts are using software iSCSI adapter. You will see a predictive failing drive - I am aware of it but as seen in the perf test this is not the cause of the hangups.

dmesg...

Code:

[root@ulduar] ~# dmesg
Copyright (c) 1992-2014 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 9.3-RELEASE-p12 #2 r275790+e006356: Thu Mar 26 17:28:49 PDT 2015
    root@build3.ixsystems.com:/tank/home/jkh/build/FN/objs/os-base/amd64/tank/home/jkh/build/FN/FreeBSD/src/sys/FREENAS.amd64 amd64
gcc version 4.2.1 20070831 patched [FreeBSD]
CPU: Intel(R) Xeon(R) CPU           E5540  @ 2.53GHz (2533.47-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x106a5  Family = 0x6  Model = 0x1a  Stepping = 5
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x9ce3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,SSE4.2,POPCNT>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  TSC: P-state invariant, performance statistics
real memory  = 53687091200 (51200 MB)
avail memory = 49750380544 (47445 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: <IBM    THURLEY >
FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs
FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 SMT threads
cpu0 (BSP): APIC ID:  0
cpu1 (AP): APIC ID:  1
cpu2 (AP): APIC ID:  2
cpu3 (AP): APIC ID:  3
cpu4 (AP): APIC ID:  4
cpu5 (AP): APIC ID:  5
cpu6 (AP): APIC ID:  6
cpu7 (AP): APIC ID:  7
cpu8 (AP): APIC ID: 16
cpu9 (AP): APIC ID: 17
cpu10 (AP): APIC ID: 18
cpu11 (AP): APIC ID: 19
cpu12 (AP): APIC ID: 20
cpu13 (AP): APIC ID: 21
cpu14 (AP): APIC ID: 22
cpu15 (AP): APIC ID: 23
WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
ACPI Warning: Invalid length for Pm1aControlBlock: 32, using default 16 (20111123/tbfadt-658)
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
ispfw: registered firmware <isp_1040>
ispfw: registered firmware <isp_1040_it>
ispfw: registered firmware <isp_1080>
ispfw: registered firmware <isp_1080_it>
ispfw: registered firmware <isp_12160>
ispfw: registered firmware <isp_12160_it>
ispfw: registered firmware <isp_2100>
ispfw: registered firmware <isp_2200>
ispfw: registered firmware <isp_2300>
ispfw: registered firmware <isp_2322>
ispfw: registered firmware <isp_2400>
ispfw: registered firmware <isp_2400_multi>
ispfw: registered firmware <isp_2500>
ispfw: registered firmware <isp_2500_multi>
kbd1 at kbdmux0
cryptosoft0: <software crypto> on motherboard
aesni0: No AESNI support.
padlock0: No ACE support.
acpi0: <IBM THURLEY> on motherboard
acpi0: Power Button (fixed)
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 450
Event timer "HPET1" frequency 14318180 Hz quality 440
Event timer "HPET2" frequency 14318180 Hz quality 440
Event timer "HPET3" frequency 14318180 Hz quality 440
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
cpu4: <ACPI CPU> on acpi0
cpu5: <ACPI CPU> on acpi0
cpu6: <ACPI CPU> on acpi0
cpu7: <ACPI CPU> on acpi0
cpu8: <ACPI CPU> on acpi0
cpu9: <ACPI CPU> on acpi0
cpu10: <ACPI CPU> on acpi0
cpu11: <ACPI CPU> on acpi0
cpu12: <ACPI CPU> on acpi0
cpu13: <ACPI CPU> on acpi0
cpu14: <ACPI CPU> on acpi0
cpu15: <ACPI CPU> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
atrtc0: Warning: Couldn't map I/O.
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x588-0x58b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pcib0: Length mismatch for 3 range: 1 vs 8100000000
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 28 at device 1.0 on pci0
pci11: <ACPI PCI bus> on pcib1
bce0: <QLogic NetXtreme II BCM5709 1000Base-T (C0)> mem 0x92000000-0x93ffffff irq 28 at device 0.0 on pci11
miibus0: <MII bus> on bce0
brgphy0: <BCM5709 10/100/1000baseT PHY> PHY 1 on miibus0
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bce0: Ethernet address: 5c:f3:fc:4c:16:c0
bce0: ASIC (0x57092003); Rev (C0); Bus (PCIe x2, 5Gbps); B/C (7.4.0); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW (NCSI 2.0.11)
Coal (RX:6,6,18,18; TX:20,20,80,80)
bce1: <QLogic NetXtreme II BCM5709 1000Base-T (C0)> mem 0x94000000-0x95ffffff irq 40 at device 0.1 on pci11
miibus1: <MII bus> on bce1
brgphy1: <BCM5709 10/100/1000baseT PHY> PHY 1 on miibus1
brgphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bce1: Ethernet address: 5c:f3:fc:4c:16:c2
bce1: ASIC (0x57092003); Rev (C0); Bus (PCIe x2, 5Gbps); B/C (7.4.0); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW (NCSI 2.0.11)
Coal (RX:6,6,18,18; TX:20,20,80,80)
pcib2: <PCI-PCI bridge> irq 29 at device 2.0 on pci0
pci16: <PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 24 at device 3.0 on pci0
pci21: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> irq 26 at device 5.0 on pci0
pci26: <ACPI PCI bus> on pcib4
pci26: <network, ethernet> at device 0.0 (no driver attached)
pci26: <network, ethernet> at device 0.1 (no driver attached)
pci26: <network, ethernet> at device 0.2 (no driver attached)
pci26: <network, ethernet> at device 0.3 (no driver attached)
pci26: <network, ethernet> at device 0.4 (no driver attached)
pci26: <mass storage, SCSI> at device 0.5 (no driver attached)
pci26: <serial bus, Fibre Channel> at device 0.6 (no driver attached)
pci26: <network, ethernet> at device 0.7 (no driver attached)
pcib5: <ACPI PCI-PCI bridge> irq 30 at device 7.0 on pci0
pci31: <ACPI PCI bus> on pcib5
mps0: <LSI SAS2308> port 0x2000-0x20ff mem 0x98a40000-0x98a4ffff,0x98a00000-0x98a3ffff irq 30 at device 0.0 on pci31
mps0: Firmware: 20.00.02.00, Driver: 16.00.00.00-fbsd
mps0: IOCCapabilities: 5285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
pcib6: <ACPI PCI-PCI bridge> irq 32 at device 9.0 on pci0
pci36: <ACPI PCI bus> on pcib6
mps1: <LSI SAS2308> port 0x1000-0x10ff mem 0x98940000-0x9894ffff,0x98900000-0x9893ffff irq 32 at device 0.0 on pci36
mps1: Firmware: 20.00.02.00, Driver: 16.00.00.00-fbsd
mps1: IOCCapabilities: 5285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
pci0: <base peripheral, interrupt controller> at device 16.0 (no driver attached)
pci0: <base peripheral, interrupt controller> at device 16.1 (no driver attached)
pci0: <base peripheral, interrupt controller> at device 17.0 (no driver attached)
pci0: <base peripheral, interrupt controller> at device 17.1 (no driver attached)
pci0: <base peripheral, interrupt controller> at device 20.0 (no driver attached)
pci0: <base peripheral, interrupt controller> at device 20.1 (no driver attached)
pci0: <base peripheral, interrupt controller> at device 20.2 (no driver attached)
pci0: <base peripheral, interrupt controller> at device 20.3 (no driver attached)
uhci0: <Intel 82801JI (ICH10) USB controller USB-D> port 0x30a0-0x30bf irq 17 at device 26.0 on pci0
usbus0 on uhci0
uhci1: <Intel 82801JI (ICH10) USB controller USB-E> port 0x3080-0x309f irq 18 at device 26.1 on pci0
usbus1 on uhci1
ehci0: <Intel 82801JI (ICH10) USB 2.0 controller USB-B> mem 0x98b21000-0x98b213ff irq 19 at device 26.7 on pci0
usbus2: EHCI version 1.0
usbus2 on ehci0
pcib7: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci1: <ACPI PCI bus> on pcib7
pcib8: <PCI-PCI bridge> irq 16 at device 28.4 on pci0
pci6: <PCI bus> on pcib8
pcib9: <PCI-PCI bridge> irq 16 at device 0.0 on pci6
pci7: <PCI bus> on pcib9
vgapci0: <VGA-compatible display> mem 0x96000000-0x96ffffff,0x98800000-0x98803fff,0x98000000-0x987fffff irq 16 at device 0.0 on pci7
vgapci0: Boot video device
uhci2: <Intel 82801JI (ICH10) USB controller USB-A> port 0x3060-0x307f irq 17 at device 29.0 on pci0
usbus3 on uhci2
uhci3: <Intel 82801JI (ICH10) USB controller USB-B> port 0x3040-0x305f irq 18 at device 29.1 on pci0
usbus4 on uhci3
uhci4: <Intel 82801JI (ICH10) USB controller USB-C> port 0x3020-0x303f irq 19 at device 29.2 on pci0
usbus5 on uhci4
ehci1: <Intel 82801JI (ICH10) USB 2.0 controller USB-A> mem 0x98b20000-0x98b203ff irq 17 at device 29.7 on pci0
usbus6: EHCI version 1.0
usbus6 on ehci1
pcib10: <PCI-PCI bridge> at device 30.0 on pci0
pci41: <PCI bus> on pcib10
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH10 SATA300 controller> port 0x3118-0x311f,0x312c-0x312f,0x3110-0x3117,0x3128-0x312b,0x30f0-0x30ff,0x30e0-0x30ef irq 16 at device 31.2 on pci0
ata2: <ATA channel> at channel 0 on atapci0
ata3: <ATA channel> at channel 1 on atapci0
atapci1: <Intel ICH10 SATA300 controller> port 0x3108-0x310f,0x3124-0x3127,0x3100-0x3107,0x3120-0x3123,0x30d0-0x30df,0x30c0-0x30cf irq 21 at device 31.5 on pci0
ata4: <ATA channel> at channel 0 on atapci1
ata5: <ATA channel> at channel 1 on atapci1
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
qpi0: <QPI system bus> on motherboard
pcib11: <QPI Host-PCI bridge> pcibus 255 on qpi0
pci255: <PCI bus> on pcib11
pcib12: <QPI Host-PCI bridge> pcibus 254 on qpi0
pci254: <PCI bus> on pcib12
ichwd0 on isa0
orm0: <ISA Option ROM> at iomem 0xc0000-0xc7fff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
est1: <Enhanced SpeedStep Frequency Control> on cpu1
coretemp2: <CPU On-Die Thermal Sensors> on cpu2
est2: <Enhanced SpeedStep Frequency Control> on cpu2
coretemp3: <CPU On-Die Thermal Sensors> on cpu3
est3: <Enhanced SpeedStep Frequency Control> on cpu3
coretemp4: <CPU On-Die Thermal Sensors> on cpu4
est4: <Enhanced SpeedStep Frequency Control> on cpu4
coretemp5: <CPU On-Die Thermal Sensors> on cpu5
est5: <Enhanced SpeedStep Frequency Control> on cpu5
coretemp6: <CPU On-Die Thermal Sensors> on cpu6
est6: <Enhanced SpeedStep Frequency Control> on cpu6
coretemp7: <CPU On-Die Thermal Sensors> on cpu7
est7: <Enhanced SpeedStep Frequency Control> on cpu7
coretemp8: <CPU On-Die Thermal Sensors> on cpu8
est8: <Enhanced SpeedStep Frequency Control> on cpu8
coretemp9: <CPU On-Die Thermal Sensors> on cpu9
est9: <Enhanced SpeedStep Frequency Control> on cpu9
coretemp10: <CPU On-Die Thermal Sensors> on cpu10
est10: <Enhanced SpeedStep Frequency Control> on cpu10
coretemp11: <CPU On-Die Thermal Sensors> on cpu11
est11: <Enhanced SpeedStep Frequency Control> on cpu11
coretemp12: <CPU On-Die Thermal Sensors> on cpu12
est12: <Enhanced SpeedStep Frequency Control> on cpu12
coretemp13: <CPU On-Die Thermal Sensors> on cpu13
est13: <Enhanced SpeedStep Frequency Control> on cpu13
coretemp14: <CPU On-Die Thermal Sensors> on cpu14
est14: <Enhanced SpeedStep Frequency Control> on cpu14
coretemp15: <CPU On-Die Thermal Sensors> on cpu15
est15: <Enhanced SpeedStep Frequency Control> on cpu15
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 480Mbps High Speed USB v2.0
usbus3: 12Mbps Full Speed USB v1.0
usbus4: 12Mbps Full Speed USB v1.0
usbus5: 12Mbps Full Speed USB v1.0
usbus6: 480Mbps High Speed USB v2.0
ugen0.1: <Intel> at usbus0
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ugen2.1: <Intel> at usbus2
uhub2: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2
ugen3.1: <Intel> at usbus3
uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3
ugen4.1: <Intel> at usbus4
uhub4: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
ugen5.1: <Intel> at usbus5
uhub5: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus5
ugen6.1: <Intel> at usbus6
uhub6: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus6
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub3: 2 ports with 2 removable, self powered
uhub4: 2 ports with 2 removable, self powered
uhub5: 2 ports with 2 removable, self powered
uhub2: 6 ports with 6 removable, self powered
uhub6: 6 ports with 6 removable, self powered
ugen3.2: <IBM> at usbus3
ugen6.2: <PNY Technologies> at usbus6
umass0: <PNY Technologies USB 2.0 FD, class 0/0, rev 2.00/11.00, addr 2> on usbus6
umass0:  SCSI over Bulk-Only; quirks = 0x0100
umass0:7:0:-1: Attached to scbus7
da2 at mps0 bus 0 scbus0 target 2 lun 0
da2: <HP EG0300FBDBR HPD7> Fixed Direct Access SCSI-5 device
da2: Serial Number         PMVTDBWB
da2: 600.000MB/s transfers
da2: Command Queueing enabled
da2: 286102MB (585937500 512 byte sectors: 255H 63S/T 36472C)
cd0 at ata2 bus 0 scbus2 target 0 lun 0
cd0: <MATSHITA DVD-RAM UJ8A0 SA82> Removable CD-ROM SCSI-0 device
cd0: Serial Number YK66 006634
cd0: 150.000MB/s transfers (SATA 1.x, UDMA5, ATAPI 12bytes, PIO 8192bytes)
cd0: Attempt to query device size failed: NOT READY, Medium not present - tray closed
da1 at mps0 bus 0 scbus0 target 1 lun 0
da1: <HP EG0300FAWHV HPDE> Fixed Direct Access SCSI-5 device
da1: Serial Number 6SE2CVZW000091262V20
da1: 600.000MB/s transfers
da1: Command Queueing enabled
da1: 286102MB (585937500 512 byte sectors: 255H 63S/T 36472C)
da3 at mps0 bus 0 scbus0 target 3 lun 0
da3: <HP EG0300FBDBR HPD7> Fixed Direct Access SCSI-5 device
da3: Serial Number         PMVS36JB
da3: 600.000MB/s transfers
da3: Command Queueing enabled
da3: 286102MB (585937500 512 byte sectors: 255H 63S/T 36472C)
da0 at mps0 bus 0 scbus0 target 0 lun 0
da0: <HP DG0300BARTQ HPD0> Fixed Direct Access SCSI-5 device
da0: Serial Number D002P9401BNV0914
da0: 300.000MB/s transfers
da0: Command Queueing enabled
da0: 286102MB (585937500 512 byte sectors: 255H 63S/T 36472C)
da4 at mps1 bus 0 scbus1 target 0 lun 0
da4: <HP EG0300FAWHV HPDD> Fixed Direct Access SCSI-5 device
da4: Serial Number 3SE20NX100009037HVFZ
da4: 600.000MB/s transfers
da4: Command Queueing enabled
da4: 286102MB (585937500 512 byte sectors: 255H 63S/T 36472C)
da5 at mps1 bus 0 scbus1 target 1 lun 0
da5: <HP DG0300BARTQ HPD0> Fixed Direct Access SCSI-5 device
da5: Serial Number D002P930101D0913
da5: 300.000MB/s transfers
da5: Command Queueing enabled
da5: 286102MB (585937500 512 byte sectors: 255H 63S/T 36472C)
da6 at mps1 bus 0 scbus1 target 3 lun 0
da6: <HP EG0300FBDSP HPD2> Fixed Direct Access SCSI-5 device
da6: Serial Number ECA1PAC00M5Y1051
da6: 600.000MB/s transfers
da6: Command Queueing enabled
da6: 286102MB (585937500 512 byte sectors: 255H 63S/T 36472C)
da8 at mps1 bus 0 scbus1 target 9 lun 0
da8: <HP EG0300FBDBR HPD7> Fixed Direct Access SCSI-5 device
da8: Serial Number         PMVKZDTB
da8: 600.000MB/s transfers
da8: Command Queueing enabled
da8: 286102MB (585937500 512 byte sectors: 255H 63S/T 36472C)
da9 at mps1 bus 0 scbus1 target 13 lun 0
da9: <HP EG0300FBDBR HPD7> Fixed Direct Access SCSI-5 device
da9: Serial Number         PMVTG2MB
da9: 600.000MB/s transfers
da9: Command Queueing enabled
da9: 286102MB (585937500 512 byte sectors: 255H 63S/T 36472C)
da7 at mps1 bus 0 scbus1 target 8 lun 0
da7: <HP DG0300BARTQ HPD0> Fixed Direct Access SCSI-5 device
da7: Serial Number D002P930101E0913
da7: 300.000MB/s transfers
da7: Command Queueing enabled
da7: 286102MB (585937500 512 byte sectors: 255H 63S/T 36472C)
da10 at mps1 bus 0 scbus1 target 15 lun 0
da10: <IBM-ESXS ST9600204SS B549> Fixed Direct Access SCSI-5 device
da10: Serial Number 6WN0NYL50000B140JFWX
da10: 600.000MB/s transfers
da10: Command Queueing enabled
da10: 572325MB (1172123568 512 byte sectors: 255H 63S/T 72961C)
da11 at mps1 bus 0 scbus1 target 16 lun 0
da11: <IBM-ESXS ST9600204SS B549> Fixed Direct Access SCSI-5 device
da11: Serial Number 6WN0S4J40000B141JFBG
da11: 600.000MB/s transfers
da11: Command Queueing enabled
da11: 572325MB (1172123568 512 byte sectors: 255H 63S/T 72961C)
da12 at umass-sim0 bus 0 scbus7 target 0 lun 0
da12: <PNY USB 2.0 FD 1100> Removable Direct Access SCSI-4 device
da12: Serial Number 04162500000066847437
da12: 40.000MB/s transfers
da12: 7788MB (15950592 512 byte sectors: 255H 63S/T 992C)
da12: quirks=0x2<NO_6_BYTE>
SMP: AP CPU #1 Launched!
SMP: AP CPU #14 Launched!
SMP: AP CPU #6 Launched!
SMP: AP CPU #15 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #12 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #11 Launched!
SMP: AP CPU #7 Launched!
SMP: AP CPU #13 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #10 Launched!
SMP: AP CPU #5 Launched!
SMP: AP CPU #9 Launched!
SMP: AP CPU #8 Launched!
Timecounter "TSC-low" frequency 1266736490 Hz quality 1000
Trying to mount root from zfs:freenas-boot/ROOT/FreeNAS-9.3-STABLE-201503270027 []...
GEOM_RAID5: Module loaded, version 1.3.20140711.62 (rev f91e28e40bf7)
t4nex0: <Chelsio T420-CR> mem 0x97900000-0x9793ffff,0x97000000-0x977fffff,0x979c8000-0x979c9fff irq 26 at device 0.4 on pci26
t4nex0: PCIe x8, 2 ports, 22 MSI-X interrupts, 71 eq, 21 iq
cxgbe0: <port 0> on t4nex0
cxgbe0: Ethernet address: 00:07:43:12:5f:f0
cxgbe0: 16 txq, 8 rxq (NIC); 8 txq, 2 rxq (TOE)
cxgbe1: <port 1> on t4nex0
cxgbe1: Ethernet address: 00:07:43:12:5f:f8
cxgbe1: 16 txq, 8 rxq (NIC); 8 txq, 2 rxq (TOE)
bce0: Gigabit link up!
bce0: Gigabit link up!
vboxdrv: fAsync=0 offMin=0x2fb offMax=0x3a66
[root@ulduar] ~#

ifconfig...

Code:

[root@ulduar] ~# ifconfig
bce0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether 5c:f3:fc:4c:16:c0
        inet 192.168.68.20 netmask 0xffffff00 broadcast 192.168.68.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
bce1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether 5c:f3:fc:4c:16:c2
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect
ipfw0: flags=8801<UP,SIMPLEX,MULTICAST> metric 0 mtu 65536
        nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0xb
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
cxgbe0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:07:43:12:5f:f0
        inet 172.31.10.1 netmask 0xfffffffc broadcast 172.31.10.3
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet 10Gbase-Twinax <full-duplex>
        status: active
cxgbe1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:07:43:12:5f:f8
        inet 172.31.11.1 netmask 0xfffffffc broadcast 172.31.11.3
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet 10Gbase-Twinax <full-duplex>
        status: active
[root@ulduar] ~#

 

[jgreco]

Nothing sticks out at me. What's happening on the FreeNAS box when the write performance is sagging?

Look especially at the output of top and gstat to see if something is saturating.

I notice your PTP configuration on the cxgbe's (netmask), so is that one link to each ESXi host, then? Just trying to clarify because multipath iSCSI is tricky and it is better to get the singlepath up and running first.

 

[Jason Keller]

One link to each host that's correct (no fancy multipathing etc, figured let's walk before we run). Also just the datastore creation is even taking forever (over gigabit it happens within seconds; i just kicked it off over 10G again and five minutes later I'm still waiting).

Outputs...tried to catch each during txg commits. This is in flight from ESXi > FreeNAS.

Code:

last pid:  8687;  load averages:  0.62,  0.41,  0.22    up 0+01:15:00  11:08:25
41 processes:  1 running, 40 sleeping
CPU:  0.0% user,  0.0% nice,  1.8% system,  0.3% interrupt, 97.9% idle
Mem: 370M Active, 138M Inact, 21G Wired, 2288K Cache, 25G Free
ARC: 20G Total, 211M MFU, 19G MRU, 40M Anon, 97M Header, 136M Other
Swap:

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
 3183 root          1  20    0   217M 96556K select  8   0:10   0.00% python2.7
 3255 root         12  20    0   158M 18860K uwait   0   0:08   0.00% collectd
 3178 root          6  21    0   372M   166M usem    3   0:04   0.00% python2.7
 3483 root          4  52    0   173M 60588K select  9   0:01   0.00% python2.7
 4071 root          1  52    0   165M 61244K ttyin  10   0:01   0.00% python2.7
 7948 root          1  20    0 18628K  2668K CPU0    0   0:01   0.00% top
 2155 root          2  20    0   174M 11888K kqread 13   0:00   0.00% syslog-ng
 7979 root          1  20    0 25236K  2952K nanslp  2   0:00   0.00% gstat
 2913 root          1  20    0 28096K  4416K nanslp 10   0:00   0.00% smartd
 2629 root          1  20    0 22224K  4020K select  6   0:00   0.00% ntpd
 2904 root          1  20    0   276M 17364K select 13   0:00   0.00% smbd
 7896 root          1  20    0 71916K  6244K select  9   0:00   0.00% sshd
 7892 root          1  20    0 71916K  6244K select  6   0:00   0.00% sshd
 3060 www           1  20    0 26348K  5540K kqread  4   0:00   0.00% nginx
 2901 root          1  20    0   209M 12144K select  1   0:00   0.00% nmbd
 2932 root          1  31   10 18596K  3284K wait    8   0:00   0.00% sh

Code:

dT: 1.001s  w: 1.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    0      0      0      0    0.0      0      0    0.0    0.0| cd0
    0     65      0      0    0.0     63   6206    7.4   42.9| da0
    0     62      0      0    0.0     61   6390    6.7   40.9| da1
    0     61      0      0    0.0     60   6322    6.6   39.7| da2
    0     59      0      0    0.0     58   6230    6.6   38.4| da3
    0     65      0      0    0.0     63   6206    7.4   42.9| da0p1
    0     62      0      0    0.0     61   6390    6.7   41.0| da1p1
    0     59      0      0    0.0     58   6230    6.8   39.2| da4
    0     66      0      0    0.0     64   6274    7.3   43.0| da5
    1     66      0      0    0.0     65   5499    6.1   39.8| da6
    1     66      0      0    0.0     65   5499    6.6   42.7| da7
    1     72      0      0    0.0     70   5526    6.4   42.8| da8
    1     71      0      0    0.0     69   5638    6.8   43.3| da9
    1     71      0      0    0.0     69   5522    6.8   43.9| da10
    1     71      0      0    0.0     69   5638    6.7   43.7| da11
    0      0      0      0    0.0      0      0    0.0    0.0| da12
    0     61      0      0    0.0     60   6322    6.6   39.7| da2p1
    0     59      0      0    0.0     58   6230    6.6   38.5| da3p1
    0     65      0      0    0.0     63   6206    7.4   42.9| gptid/8cd38744-d8a9-11e4-8f6f-5cf3fc4c16c0
    0     62      0      0    0.0     61   6390    6.7   41.0| gptid/8d642fac-d8a9-11e4-8f6f-5cf3fc4c16c0
    0     59      0      0    0.0     58   6230    6.8   39.2| da4p1
    0     66      0      0    0.0     64   6274    7.3   43.1| da5p1

 

[Jason Keller]

Graph from GUI...gap is between two different transfers

 

[HoneyBadger]

Can I ask what the use case is that necessitates the use of the Windows VM as the iSCSI initiator rather than letting ESXi handle it and layering a VMDK on top or passing the LUN through via RDM?

 

[Jason Keller]

I do not intend to do that in production (Windows VM initiator > FreeNAS). I'm highlighting the fact that a Windows VM, using the ESXi network stack/driver to get to the target, is still several times faster than iSCSI from the host itself in that direction. I'm trying to narrow the scope of the issue.

The problem is the iSCSI performance in ESXi going from host > FreeNAS. That's what I'm trying to solve (and getting beyond frustrated doing so).

 

[HoneyBadger]

Gotcha. The Windows VM being the initiator might be an issue because the ESXi vSwitch might be doing something silly with the packets that would cause the sync/async part to get a little silly.

But re: the bursty writes, I'm wondering if maybe it's choking under sustained write load. 48GB of RAM with the default tunables I believe would generate a rather big default TXG size - 6GB IIRC - and without sync, a huge burst of writes could come into the pool and get backlogged trying to spin off to disk. 6 mirrored vdevs of 15K disks should be able to sustain a pretty high rate of dumping to disk but fragmentation and other access will also hurt that.

 

[jgreco]

It is reporting busier than I'd expect for a relatively small number of IOPS, but while that may just be poor estimation, the symptoms and numbers start to feel like a pool that's way too full.

What's the pool full percentage ("zpool list") and is fragmentation being reported ("zpool get fragmentation")? And whatcha been doin' on the pool?

 

[Jason Keller]

It is reporting busier than I'd expect for a relatively small number of IOPS, but while that may just be poor estimation, the symptoms and numbers start to feel like a pool that's way too full.

What's the pool full percentage ("zpool list") and is fragmentation being reported ("zpool get fragmentation")? And whatcha been doin' on the pool?

Unfortunately it's empty (the ZVOL is essentially the only thing on the pool) and it's the only thing that's running. I am curious about the busy % as well.

 

[Mlovelace]

It's a shot in the dark... but I know there have been very odd problems with this miss-match. Try flashing your HBA firmware to the P16 version and test the iSCSI after that.

mps0: Firmware: 20.00.02.00, Driver: 16.00.00.00-fbsd

 

[Jason Keller]

It's a shot in the dark... but I know there have been very odd problems with this miss-match. Try flashing your HBA firmware to the P16 version and test the iSCSI after that.

mps0: Firmware: 20.00.02.00, Driver: 16.00.00.00-fbsd

Ironically I had initially tried to do this (and I have that firmware already downloaded) but it is not allowing me to erase and downflash it. Is there a special procedure for doing this with sas2flash in BSD? Or do I need the DOS version? I've never had to downgrade firmware on the LSI HBAs before.

 

[Mlovelace]

When I downgraded my HBA's I used the uefi version of sas2flash and that worked. Some people say not to add back the bios but I did. It takes slightly longer to post but I like the utility of it.

 

[Jason Keller]

When I downgraded my HBA's I used the uefi version of sas2flash and that worked. Some people say not to add back the bios but I did. It takes slightly longer to post but I like the utility of it.

Is the command to erase sas2flash -e -o 6 and then feed it the index number? And am I downgrading both BIOS and firmware (assuming so) or just downgrade the firmware?

 

[Mlovelace]

Is the command to erase sas2flash -e -o 6 and then feed it the index number? And am I downgrading both BIOS and firmware (assuming so) or just downgrade the firmware?

Yes that is the command to erase, and yes downgrade the bios as well. Make note of the card's info with sas2flash -listall before the erase.

 

[Jason Keller]

Yes that is the command to erase, and yes downgrade the bios as well. Make note of the card's info with sas2flash -listall before the erase.

Help! Got them downgraded but now the server hangs on "Connecting Boot Devices and Adapters" in the BIOS. Do I need to remove the BIOS from the cards somehow? I needn't boot off of them.

 

[Jason Keller]

Ok had a little bit of a scare there. I have it up again and everything is green now in FreeNAS but nothing has changed (in anything, gstat, iscsi, nothing). Still stuck around 500 Mbit for ESXi iSCSI > FreeNAS.

 

[Jason Keller]

Anyone have any more ideas? I've reached out to Chelsio support and have had zero reply from them. Could this be a driver interaction between FreeNAS and ESXi with Chelsio cards?

 

[Jason Keller]

Ok a bit more information....

Tried tweaking around on iSCSI first/max burst and NOOP intervals to no avail. Then disabled Delayed Ack in VMware on the iSCSI adapter, rebooted and...WHOA! Hitting about 5Gbit pushing ESXi > FreeNAS. However, now FreeNAS > ESXi is down to about 1.5 - 2 Gbit. Also disabling Delayed Ack in FreeNAS causes ESXI > FreeNAS to dip down to 1Gbit; however pushes FreeNAS > ESXi back up to 5Gbit.

Hopefully this sparks some ideas from someone.

 

[Jason Keller]

Solved

Problem, as it stands, is Software iSCSI Adapter in ESXi seems to absolutely suck on Chelsio cards (and it appears that it is NOT dynamically negotiating it's iSCSI settings like burst with CTL when the Chelsio driver is used in FreeNAS). Solution was actually pretty simple - the csiostor driver for ESXi (written by Chelsio). Now, their release notes state that only the T5 cards are supported. This is false - my T4 was picked up and so far seems to be working just fine. Due to disk bottlenecks, I'm getting roughly 4-5Gbit both directions. I am also still using the networking driver (cxgb4) with it to provide 10gbit vMotion between hosts. Again in contravention to the readme, it doesn't appear to matter which loads first (csiostor or cxgb4). That issue was probably resolved in the latest networking driver that I'm using.

Now the T4 actually shows up as two iSCSI hardware independent adapters as well as two available networking NICs in VMware (the way I was looking to have it in the first place). Snooping through some of the iSCSI settings show some very different defaults from ESXi's software initiator (one of them notably is that delayed ack is grayed out completely). Though I doubt even if I managed to dig up all those settings that the software initiator would have performed acceptably anyway. Also of note was lack of send target support for dynamic discovery in csiostor - you'll have to statically assign the targets (the field is there but it doesn't do anything).

I must say that after fighting with this for a week, that I am beyond dismayed by the utter void of any support from Chelsio. For all intents and purposes, their support doesn't exist. Considering the volume of people using ESXi/FreeNAS I'd also have expected someone to run into this before (especially since iXsystems writes the driver for FreeBSD), but if they have they never posted anything about it. I seek to change that so the next poor soul to venture here has an easier time of it.

So for those looking to use Chelsio cards in their FreeNAS host and ESXi hosts...ye be warned, these are murky waters. You'll have to use both csiostor (VMware site; used in contravention to stated driver support) and the latest network driver (Chelsio site, though I bet you can find that on VMware's site as well if you dig enough). If you want to have a good experience anyway.

 

[Darren Myers]

So i'm using 2 ESXi hosts one with a Qlogic 10Gb card and the other with the same chelsio T420-CR card, i installed the drivers for the chelsio card upon installation of ESXi 5.5 u2 because the card wasnt seen what so ever. Are you saying youre getting better performance from your T420 when using the T520 iSCSI driver instead? I have yet to test the speed of either of my cards but i might just carve out some time to do that tomorrow.

EDIT: the "Chelsio Full Offload-iSCSI Initiator Driver v1.0.0.0 for ESXi5.5 " driver?