Slow performance with 10GbE

[Tarantino]

Hey all, I've been scouring these forums for the past few weeks but I am not able to find a solution to my issue.

I am experiencing an issue where 10GBe performance is not at the level I would expect.

I am using TrueNAS-12.0-U8.

Here are my hardware specs:

• Motherboard: Supermicro X9SCL-F
• Processor: Xeon E3-1230 V2
• HBA: IBM M5110 8-Port 6Gbps PCI-e SAS/SATA IT MODE FW:P20 LSI 9207-8I SAS 2308
• 10GbE NIC: Mellanox HP 518001-001 10 GbE PCI-e
• RAM: 16GB UDIMM ECC
• Hard Drives:
  • 5x HGST 18TB 7200RPM 64MB Cache SATA 6Gb/s 3.5" Internal Hard Drive
  • 1x SSD 256GB Where I use for truenas and for jails as well

Workstation Specs:

• Intel i7 7700k
• 32GB DDR4 Ram
• Samsung 970 NVM
• Intel Mellanox ConnectX-3 MCX311A-XCAT EN 10G Ethernet 10GbE SFP+ PCI-E (connected with Cat6a)
• Tried on Windows 10, then upgraded to Windows 11, still same speeds

If I connect my PC with my NAS directly and setup an static IP Address I get awesome speeds, reading and writing from it, tested with iperf3 and windows(SMB).

BUT if I connect both NAS and PC to my switch (QNAP QSW-2104-2S 10GbE SFP+) then I get bad speeds on writing to NAS, and I did many experiments. Only getting 40~100MB/s when I could get at least 400MB/s

If I connect the NAS into the 10GB sfp port of switch (using nic adapters on nas and switch) and then connect from the port 2.5 ethernet of the switch into the PC but with an SFP adapter of 1gbe then I get this speeds, we can see that I get very solids 1gb speeds read and writing, and this is interesting... why when I capped the speed with an 1gbe adapter I get better speeds?

Then I did another test...
I connected the NAS to the Switch 2.5 ethernet port and then switch 2.5 ethernet to the PC (Using an NIC Adapter of 10gbE -> QSFPTEK 10GBASE-T SFP) and I got this speeds, how akward... 2.5GbE should support at least 250MB:

Did another test too, connecting from NAS (with nic adapter 10gbe) to 2.5 Switch eth port and then from the SFP 10gb port of the switch I connect the 2nd nic adapter there so I could connect into my motherboard 1gbe and this was the result, better but still not ideal:

And as final test I connected the SFPs cables directly between NAS to Switch sfp port and PC to Switch 2nd sfp port and this was the results:

Better results then using the NICs adapters, but still, very slow speeds...

Note: I am not using this SFP cables normally since the ones I got with the adapter were too short 0.5m lol, in any case my best results was when I connect the nas directly to the pc, both SFP ports but with nic adapters to ethernet and using an 1.5m cat6a cable...
I only have 2 nic 10gbe and 1 nic 1gbe. I recently purchase an Supermicro AOC-STG-i2T REV 1.0 Dual Port 10GbE Ethernet PCI-E but I didnt test it yet.

Anyone knows what is goin on here? xD

Is it the switch fault? Is there anything I can do to solve this?

 

[Tarantino]

Forgot to mention. The 5 HDD are in 1 pool, 1 vdev, raidz1.

 

[Tarantino]

So... I've news.

Almost all my previous tests were done without the sfp cables, just did 1 test with it and the speeds were the same and I dunno how, with sfp cables and after reboot the NAS, it started to work magically. Today I woke up, just test it again for no reason and I was already getting great speeds lool Dunno what I did before... I just remember to reset the network on truenas and pc. But maybe truenas should be reset so it could pick a different configuration, lets god know why xD

Bottom line... That means that all my first tests above with the NIC SFP to Eth QSFPTEK adapter for some reason is not compatible with something. I tried different mtu values and a lot of tunning configuration I found on this forum and in some linux blogs posts. And nothing worked.

I think the problem is that since the switch is in VPI instead of ETH, it doesnt perform as it should from NAS to PC. (Strange thing that it works fine PC to NAS) But after using sfp cables, and restart the machine and the networks I now got good speeds both ways:

So half the problem is solved I get finally 600~800MB/s from iperf, dunno why it doesnt go further, but its already great speeds for my specs.
If any of you think it is possible to get better by changing mtu, I'm using 1500 default one. Or any other tunning value, let me know! For what I read, the truenas version I use should not need any tunning since its already ready for10GbE, but maybe I misunderstood.


Only problem remaining...
My 2.5 ethernet ports of the router are still at 1Gb (110MB/s max via SMB) I have tested with an Supermicro AOC-STG-i2T REV 1.0 Dual Port 10GbE Ethernet on an 2nd computer. Need to do some iperf tests on that 2nd pc too and I'll probably put there another Intel Mellanox ConnectX-3 MCX311A-XCAT EN 10G Ethernet 10GbE SFP+ PCI-E with an NIC SFP adapter to eth so I can connect an cat6a between the 2nd pc to the 2.5 eth switch port and see how it goes... More news soon.

 

[Morris]

Welcome to the wonderful world of 10-Gb ethernet. Things are not as compatible as one would like, particularly with the new SOHO switches and third party SFP+. You need to be very carful about SFP+ and the NIC or switch they are used in. Double this caution when it comes to multi gig as it's so new. Also look out for flow control as it can cause the odd throughput you are seeing sometimes you will see bursts of speed when flow control is not matched correctly.

 

[Tarantino]

So, with the Supermicro AOC-STG-i2T on my NAS instead of the Mellanox Dual Port, I finnaly I was able to get full speed :)

I did change the MTU from default 1514 to 9014 (On TrueNAS and on the PCs). With default values I was getting 500~650MB/s both ways.

About the 2nd PC that I wanted to connect on 2.5GbE port, I was able to get 288MB/s from NAS to PC but only 30~80, it varies too much, from PC to NAS. If I want stability I need to put an NIC of 1GbE to get 114MB/s both ways.
I tried 3 different 10GbE NICs (HiFiber, 6COM and QSFPTek)

About the switch I ended up buying an QNAP QSW-2104-2T 10GbE (its a version only with ethernet instead of having sfp for 10gbe ports) and I bought too an MikroTik CRS305-1G-4S+IN but I got too many issues with it. (It worked perfectly with SFP Cables though, but if you need to use NICs to use ethernet cables, forget it... just troubles)

So I stick up with the qnap ethernet version and returned the sfp ones.

Someday I'll try to figure out how to solve the 2.5GbE port problem. Its not easy since I need to configure to work properly to both 2.5 and 10gb ports of that switch, but well...

The ideal for me were to get an QNAP with 4 10gbe ports... that would be perfect and solved all issues. Let's hope in the near future they can do more SOHO switches with 10GbE ports.

 

[Morris]

The MikroTik CRS305-1G-4S+IN works quite nicely with 10-Gb Copper using there SFP+ and some third party SFP+. For 2.5 Gb I recommend using the MikroTik SFP+

I use the ipolex SFP+ for 10-Gb
MikroTik MikroTik S+RJ10 for 2.5 Gb which will also work fine for 10-Gb. The first is much less expensive

 

[Tarantino]

Do I really need cat6a cables or I'm fine with cat6? For what I read, cat6 should be more then enough for 3m cables max. in my case.

 

[Morris]

Most don't know that cat 5E is good to 40 feet. Cat 6 will be fine