Hello everyone, long time lurker, new poster. Through forum information i have been running Free/TrueNAS for a while without needing to post for help.
TLDR summary, when using ConnectX-3 QSFP+ dual port NIC, iSCSI is nearly unusable. When using a pair of 10gBaseT copper interfaces everything is fine. What tunable's should i be trying to get the qsfp+ nics to work properly?
System Summary:
Super Micro X10DRi-T4+ mainboard
2x Xeon E5-2620v3. (2x e5-2667v3 in route to upgrade)
This has 4 on board 10gbaseT nics that work great
384g ddr4-2133 (384gig ordered for 768g total)
2x Intel DC series SATA SSD for OS in zfs mirror.
BPN-SAS3-846EL1 24-port 4U SAS3 12Gbps single-expander backplane (all spinners here)
BPN-SAS3-826EL1 12-port 2U SAS3 12Gbps single-expander backplane (flash & scratch drives here, two empty bays)
LSI-9400 16i IT mode, two physical connections, one to each expander backplane, two open/unused connections)
Mellanox ConnectX-3 CX354A MCX354A-FCBT Dual port QSFP+ nic This one causes me problems.
ZIL Optane 280g 900p and L2-Arc Samsung NVME currently offline/not installed due to bracket issues (need lo-pro brackets to install in the current rack mount case)
Pool1: qty 12, 10tb SAS3 4kn PMR HDD, 3 drive raid-z vdev's, 4 vdevs.
Pool2: qty 12, 10tb SAS3 4kn PMR HDD, 3 drive raid-z vdev's, 4 vdevs.
Pool3: qty 2, 4tb sata, PMR hdd, mirror (scratch drive for misc high fragmentation activities)
Pool4: qty 8, 3.84tb PM1633a, SAS3 SSD, 4 drive raidz vdev, 2 vdevs.
networking: HP 5930 4 Slot chassis switch, latest firmware, 2x 8 port qsfp+ cards, 1x SFP+, 1x 10gBaseT. Appropriate HP SR4 QSFP's in switch, appropriate Finisar QSFP's in Mellanox NIC's, MPO 12 fiber cables between, links in Up/Up state.
Local NTP source with ALL devices syncing to that source.
I use this NAS in my home lab for a lot of different tasks and use cases involving CIFS/Samba, NFS, iscsi, occasional local vm's, etc. Everything was working great except for often maxing out the 10g interfaces hence the desire to upgrade to the 40g nics which I installed a week ago.
I have been struggling to figure out how to get speed from the Mellanox NIC. It doesnt seem to matter if jumbo/MTU9000 is enabled or not, flow control, hard set 40g full duplex on both ends or auto negotiate, switching PCI slots (to change the cpu they are connected to) etc etc the best i have been able to do on average is 30-40MBps on these nic's with iSCSI. i DO see some bursts up to 700-800MBps however they are very short and then drop into the low three digit KBps range. Actual throughput testing with only two VM's on spinning disk or flash, generally average out to 30-60MBps. The same test on the same VM's, using 10gBaseT are getting 200MBps each for writes, and far beyond the interface speed for reads in the 3-6GBps range. Im guessing for reads to get this high it is heavily dependent on VAAI / and cacheing magic despite sourcing my test data from /dev/urandom to create the test files Im reading/writing to therefore not just files full of zeros. Iozone exibits the same behavior 20-40MBps on qsfp, 180-200MBps on 10gBaseT for writes and faster than interface reads.
Testing from vmware hosts with qsfp+ nics and iperf3, i reliably get 34-38gbps between vmware hosts and Truenas.
MPIO / RR to either a pair of 10gBaseT or a pair of QSFP's on the mellanox card is in place and working using this configuration:
esxcli storage nmp satp rule add -s VMW_SATP_ALUA -V "TrueNAS" -M "iSCSI Disk" -P VMW_PSP_RR -O iops=1 -e "TrueNAS SATP rule"
I have positively identified that multipathing is working great within VMWare. I can also see an even load of IO on the 10gBaseT nics when stress testing.
On the QSFP nics I often see crazy levels of latency when writing, upwards of 1100ms. Same workload, same testing, same vm's might see a maximum spike of 50ms on the 10gBaseT interfaces.
I thought I might have been having some strange QPI interface issues between the CPU's but that has been ruled out moving the SAS controller and the NIC's to various PCIE ports within the case/motherboard. I thought maybe there was an issue with the old HBA so i swapped it for the 9400. That had virtually no effect on the iSCSI traffic on the mellanox cards but added ~50-100MBps to the 10tb spinner pools and the SAS SSD pool when using the onboard 10gBase-T ports so I intend to keep the 9400 within the system.
local iozone results are fair given the circumstances of the lower CPU clock rate along with the lack of L2-ARC and ZIL/SLOG. I should have the new CPU's some time next week and I'm still searching for appropriate low profile brackets for L2arc and ZIL/SLOG.
I look forward to your suggestions and input. What do I need to do with the 40g interfaces to get them to perform with iSCSI?
TLDR summary, when using ConnectX-3 QSFP+ dual port NIC, iSCSI is nearly unusable. When using a pair of 10gBaseT copper interfaces everything is fine. What tunable's should i be trying to get the qsfp+ nics to work properly?
System Summary:
Super Micro X10DRi-T4+ mainboard
2x Xeon E5-2620v3. (2x e5-2667v3 in route to upgrade)
This has 4 on board 10gbaseT nics that work great
384g ddr4-2133 (384gig ordered for 768g total)
2x Intel DC series SATA SSD for OS in zfs mirror.
BPN-SAS3-846EL1 24-port 4U SAS3 12Gbps single-expander backplane (all spinners here)
BPN-SAS3-826EL1 12-port 2U SAS3 12Gbps single-expander backplane (flash & scratch drives here, two empty bays)
LSI-9400 16i IT mode, two physical connections, one to each expander backplane, two open/unused connections)
Mellanox ConnectX-3 CX354A MCX354A-FCBT Dual port QSFP+ nic This one causes me problems.
ZIL Optane 280g 900p and L2-Arc Samsung NVME currently offline/not installed due to bracket issues (need lo-pro brackets to install in the current rack mount case)
Pool1: qty 12, 10tb SAS3 4kn PMR HDD, 3 drive raid-z vdev's, 4 vdevs.
Pool2: qty 12, 10tb SAS3 4kn PMR HDD, 3 drive raid-z vdev's, 4 vdevs.
Pool3: qty 2, 4tb sata, PMR hdd, mirror (scratch drive for misc high fragmentation activities)
Pool4: qty 8, 3.84tb PM1633a, SAS3 SSD, 4 drive raidz vdev, 2 vdevs.
networking: HP 5930 4 Slot chassis switch, latest firmware, 2x 8 port qsfp+ cards, 1x SFP+, 1x 10gBaseT. Appropriate HP SR4 QSFP's in switch, appropriate Finisar QSFP's in Mellanox NIC's, MPO 12 fiber cables between, links in Up/Up state.
Local NTP source with ALL devices syncing to that source.
I use this NAS in my home lab for a lot of different tasks and use cases involving CIFS/Samba, NFS, iscsi, occasional local vm's, etc. Everything was working great except for often maxing out the 10g interfaces hence the desire to upgrade to the 40g nics which I installed a week ago.
I have been struggling to figure out how to get speed from the Mellanox NIC. It doesnt seem to matter if jumbo/MTU9000 is enabled or not, flow control, hard set 40g full duplex on both ends or auto negotiate, switching PCI slots (to change the cpu they are connected to) etc etc the best i have been able to do on average is 30-40MBps on these nic's with iSCSI. i DO see some bursts up to 700-800MBps however they are very short and then drop into the low three digit KBps range. Actual throughput testing with only two VM's on spinning disk or flash, generally average out to 30-60MBps. The same test on the same VM's, using 10gBaseT are getting 200MBps each for writes, and far beyond the interface speed for reads in the 3-6GBps range. Im guessing for reads to get this high it is heavily dependent on VAAI / and cacheing magic despite sourcing my test data from /dev/urandom to create the test files Im reading/writing to therefore not just files full of zeros. Iozone exibits the same behavior 20-40MBps on qsfp, 180-200MBps on 10gBaseT for writes and faster than interface reads.
Testing from vmware hosts with qsfp+ nics and iperf3, i reliably get 34-38gbps between vmware hosts and Truenas.
MPIO / RR to either a pair of 10gBaseT or a pair of QSFP's on the mellanox card is in place and working using this configuration:
esxcli storage nmp satp rule add -s VMW_SATP_ALUA -V "TrueNAS" -M "iSCSI Disk" -P VMW_PSP_RR -O iops=1 -e "TrueNAS SATP rule"
I have positively identified that multipathing is working great within VMWare. I can also see an even load of IO on the 10gBaseT nics when stress testing.
On the QSFP nics I often see crazy levels of latency when writing, upwards of 1100ms. Same workload, same testing, same vm's might see a maximum spike of 50ms on the 10gBaseT interfaces.
I thought I might have been having some strange QPI interface issues between the CPU's but that has been ruled out moving the SAS controller and the NIC's to various PCIE ports within the case/motherboard. I thought maybe there was an issue with the old HBA so i swapped it for the 9400. That had virtually no effect on the iSCSI traffic on the mellanox cards but added ~50-100MBps to the 10tb spinner pools and the SAS SSD pool when using the onboard 10gBase-T ports so I intend to keep the 9400 within the system.
local iozone results are fair given the circumstances of the lower CPU clock rate along with the lack of L2-ARC and ZIL/SLOG. I should have the new CPU's some time next week and I'm still searching for appropriate low profile brackets for L2arc and ZIL/SLOG.
I look forward to your suggestions and input. What do I need to do with the 40g interfaces to get them to perform with iSCSI?
