Help troubleshooting 10Gbe performance

[hakuna-matata]

Hi all,
Long-ish time TrueNAS user here, I think I started back when I created this account in 2014 and was running FreeNAS 9. Everything has been rock solid all of these years through numerous OS upgrades and I am still running my original server to this date. The drives are getting up there in age and the lack of a backup plan finally pushed me to building a backup server using an old machine was sitting in the closet and gathering dust. I decided to use this opportunity to learn about and experiment with 10Gbe to as consumer devices are now starting to become available with this as an option (e.g. M1/M2 Mac Mini). This is where I have run into some problems. More details below but first off here are the hardware specs:

Main Server (192.168.1.100)
TrueNAS-13.0-U3.1
Supermicro X9SCM-F-O
Intel Xeon E3-1230 V2
32GB DDR3 ECC RAM
6 x 3TB Toshiba DT01ACA300 in RAID-Z2
NIC: 10GTek X520-10G-1S-X8 installed in Slot 6 PCI-E 3.0/2.0 x8

Backup Server (192.168.1.110)
Lenovo Thinkserver TS140
TrueNAS-SCALE-22.12.0
Intel Core i3-4130
16GB DDR3 ECC RAM
2 x 18TB SATA3 Exos X18 ST18000NM000J in Mirror Pool
NIC: 10GTek X520-10G-1S-X8 installed in Slot 1 PCIe 3.0 x16

Both of these machines are connected to the SFP+ ports on Ubiquiti UDM SE (192.168.1.1) using 2 meter long DAC cables. (with the Ubiquiti compatability option)

I know that the TrueNAS forums recommend genuine Intel NICs or used SolarFlare cards, but these were not available in my country at reasonable prices so I decided to take abit of a gamble and go with these 10Gtek NICs as some people have successfully installed these in their systems.

Upon installating the cards, TrueNAS on both systems recognized them immediately as 10Gbe cards without any modifications on my part.

Main Server

Backup Server

Auto-negotiation is enabled on my UDM SE and a 10Gbe link speed is detected on both of the SFP+ ports (port 10 and port 11).

I thought everything was good up until this point! So I created a replication task to copy a 8TB dataset snapshot from my main server to a backup server overnight... and here is how that went.

(I am omitting charts for the HDD read/write speeds for brevity)

My next step was to try and understand where the bottleneck was coming from. Was it the HDDs or something with the network? Using my limited knowledge, I started to test if my network was performing as intended. I have only 2 devices in my network that are 10Gbe capable plus the UDM SE router.

I ran basic iperf3 tests on all three devices and here are the results.

So there certainly seems to be something going on with the network. Is it the NICs that are to blame? Perhaps bad DAC cables? Driver/compatibility issue with TrueNAS? Or maybe some setting that needs to be changed somewhere? Would appreciate some tips on how to troubleshoot further.

What I have tried so far without any success:
- Changing PCI-E slots
- Bumping up the MTU from 1500 to 9000
- Disabling Hardware Offloading on the NICs
- Enabling "Jumbo Frames" on the UDM-SE

Thanks in advance.

 

[john.o]

I use a UDM (non SE) as well and have a similar setup with a GTEK NIC in my TrueNAS CORE server. Have you tried turning on flow control to see if that improves performance?

 

[hakuna-matata]

I use a UDM (non SE) as well and have a similar setup with a GTEK NIC in my TrueNAS CORE server. Have you tried turning on flow control to see if that improves performance?

I purchased the Unifi USW 10gig aggregation switch and after plugging both servers into it, the network speed issue is resolved. So there seems to be a problem with the UDM-SE if both SFP+ ports are configured for LAN use.