10gbe performance differences

[foureight84]

I was performing connectivity with iperf3 from my Windows 11 box to my TrueNAS installed and I'm noticing that on the return path from NAS to Client speed seems to be maxing out at around 5-6Gbit/s.

10.0.0.1 is the TrueNAS server over directly connected Mellanox MCX311A-XCAT

$> iperf3 -c 10.0.0.1
Connecting to host 10.0.0.1, port 5201
[ 5] local 172.24.225.110 port 59852 connected to 10.0.0.1 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 936 MBytes 7.85 Gbits/sec 0 3.15 MBytes
[ 5] 1.00-2.00 sec 1.07 GBytes 9.15 Gbits/sec 0 3.15 MBytes
[ 5] 2.00-3.00 sec 1.05 GBytes 9.02 Gbits/sec 0 3.15 MBytes
[ 5] 3.00-4.00 sec 1.06 GBytes 9.08 Gbits/sec 0 3.15 MBytes
[ 5] 4.00-5.00 sec 1.06 GBytes 9.09 Gbits/sec 0 3.15 MBytes
[ 5] 5.00-6.00 sec 1.06 GBytes 9.12 Gbits/sec 0 3.15 MBytes
[ 5] 6.00-7.00 sec 1.07 GBytes 9.18 Gbits/sec 0 3.15 MBytes
[ 5] 7.00-8.00 sec 1.08 GBytes 9.30 Gbits/sec 0 3.15 MBytes
[ 5] 8.00-9.00 sec 1.04 GBytes 8.90 Gbits/sec 0 3.15 MBytes
[ 5] 9.00-10.00 sec 1.06 GBytes 9.08 Gbits/sec 0 3.15 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 10.5 GBytes 8.98 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 10.4 GBytes 8.97 Gbits/sec receiver

iperf Done.

$> iperf3 -c 10.0.0.1 -R
Connecting to host 10.0.0.1, port 5201
Reverse mode, remote host 10.0.0.1 is sending
[ 5] local 172.24.225.110 port 59856 connected to 10.0.0.1 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 801 MBytes 6.72 Gbits/sec
[ 5] 1.00-2.00 sec 830 MBytes 6.96 Gbits/sec
[ 5] 2.00-3.00 sec 830 MBytes 6.96 Gbits/sec
[ 5] 3.00-4.00 sec 833 MBytes 6.99 Gbits/sec
[ 5] 4.00-5.00 sec 831 MBytes 6.97 Gbits/sec
[ 5] 5.00-6.00 sec 828 MBytes 6.94 Gbits/sec
[ 5] 6.00-7.00 sec 830 MBytes 6.96 Gbits/sec
[ 5] 7.00-8.00 sec 826 MBytes 6.93 Gbits/sec
[ 5] 8.00-9.00 sec 829 MBytes 6.96 Gbits/sec
[ 5] 9.00-10.00 sec 830 MBytes 6.96 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 8.08 GBytes 6.94 Gbits/sec 342 sender
[ 5] 0.00-10.00 sec 8.07 GBytes 6.94 Gbits/sec receiver

iperf Done.

I wonder if this is caused by CPU bottleneck on the NAS (which I don't think is the case). Here are my specs:

TrueNAS:
Asrock C236 WSI w/ i5-7500t
Onboard Intel i210 and I219-LM NICs (both connected to switch on LAGG and bonded on TrueNAS)
16GB DDR 2133
Mellanox MCX311A-XCAT

MTU: 9000

Windows 11:
Asus WS x299 Sage w/ i9-9920x
Onboard Intel i210 and I219-LM NICs
64GB DDR4 3200
Mellanox MCX311A-XCAT

Jumbo Packet: 9000

EDIT:
I thought it might have been because I ran iperf3 from WSL2 but that's not the case that now I've tested using the iperf3 windows binary. It's about the same speed going from the NAS to Windows 11. However, testing on an Ubuntu live boot, I'm seeing full speed both ways.

I am curious if this is a driver issue on Windows 11 since the MCX311a is currently using a Windows-provided driver instead of Mellanox's. According to Mellanox, ConnectX-3 will not receive Windows 11 support.

 

[NugentS]

Sounds like software to me

 

[foureight84]

Sounds like software to me

Yes, it's definitely software and it's quite odd!

I loaded a Windows 10 based Windows PE environment and ran iperf3 and I was able to get about 9.8Gbits/s in either direction. I then checked the driver version and that matches what's being used in Windows 11. Then I ran iperf3 again in Windows 11 and now I'm getting full speed in either direction.

It's really strange.