Performance issue at 2.5gb link speed

[repatch]

Hello all!

I've been trying to upgrade the performance critical portions of my home network to 2.5gbe, and of course, that includes my trusty TrueNAS boxes

I've been using the RT8125 based NICs, and after a bunch of searching and info from this forum got them working, at 2.5gb line rate on my TrueNAS box (using the tunables to enable the driver).

I haven't however been getting the speed boost I expected, after a bunch of searching I've narrowed the issue down to my TrueNAS box using iPerf.

For some reason I only get 1.37Gbps between either of my two other machines and the TrueNAS box:
iperf3.exe -V -c 192.168.1.21
iperf 3.1.3
CYGWIN_NT-10.0 BasementPC 2.5.1(0.297/5/3) 2016-04-21 22:14 x86_64
Time: Sun, 10 Jul 2022 00:45:58 GMT
Connecting to host 192.168.1.21, port 5201
Cookie: BasementPC.1657413958.278030.2b01512
TCP MSS: 0 (default)
[ 4] local 192.168.1.66 port 50097 connected to 192.168.1.21 port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 141 MBytes 1.18 Gbits/sec
[ 4] 1.00-2.00 sec 164 MBytes 1.38 Gbits/sec
[ 4] 2.00-3.00 sec 165 MBytes 1.39 Gbits/sec
[ 4] 3.00-4.00 sec 164 MBytes 1.38 Gbits/sec
[ 4] 4.00-5.00 sec 161 MBytes 1.35 Gbits/sec
[ 4] 5.00-6.00 sec 157 MBytes 1.32 Gbits/sec
[ 4] 6.00-7.00 sec 164 MBytes 1.37 Gbits/sec
[ 4] 7.00-8.00 sec 165 MBytes 1.39 Gbits/sec
[ 4] 8.00-9.00 sec 163 MBytes 1.37 Gbits/sec
[ 4] 9.00-10.00 sec 164 MBytes 1.38 Gbits/sec

Between those two other machines I get close to the 2.5Gbe line rate:

iperf3.exe -V -c 192.168.1.15
iperf 3.1.3
CYGWIN_NT-10.0 BasementPC 2.5.1(0.297/5/3) 2016-04-21 22:14 x86_64
Time: Sun, 10 Jul 2022 00:55:57 GMT
Connecting to host 192.168.1.15, port 5201
Cookie: BasementPC.1657414557.436893.68a1a2b
TCP MSS: 0 (default)
[ 4] local 192.168.1.66 port 50233 connected to 192.168.1.15 port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 173 MBytes 1.45 Gbits/sec
[ 4] 1.00-2.00 sec 280 MBytes 2.35 Gbits/sec
[ 4] 2.00-3.00 sec 280 MBytes 2.35 Gbits/sec
[ 4] 3.00-4.00 sec 273 MBytes 2.29 Gbits/sec
[ 4] 4.00-5.00 sec 204 MBytes 1.71 Gbits/sec
[ 4] 5.00-6.00 sec 258 MBytes 2.17 Gbits/sec
[ 4] 6.00-7.00 sec 280 MBytes 2.35 Gbits/sec
[ 4] 7.00-8.00 sec 276 MBytes 2.32 Gbits/sec
[ 4] 8.00-9.00 sec 272 MBytes 2.28 Gbits/sec
[ 4] 9.00-10.00 sec 275 MBytes 2.30 Gbits/sec

Now, the interesting bit: the two machines that hit the 2.5gbps rate are a Win10 machine, and a TrueNAS VM instance running on Proxmox.

The TrueNAS instance that's having trouble is running on bare metal.

I'm running TrueNAS Core Version: TrueNAS-12.0-U8.1

Anybody know what I should be looking at next?

Thanks for any help or suggestions!

 

[elvisimprsntr]

Did you enable NIC jumbo frames on the bare metal TrueNAS bare metal instance?

MTU=9000

You will need to enable jumbo frames on all devices in the path
1. TrueNAS
2. 2.5GB managed switch
3. Second clients

 

[repatch]

Did you enable NIC jumbo frames on the bare metal TrueNAS bare metal instance?

MTU=9000

You will need to enable jumbo frames on all devices in the path
1. TrueNAS
2. 2.5GB managed switch
3. Second clients

I haven't since everything I've read points to issues with devices on the network not having jumbo frames enabled.

I will note, I haven't enabled jumbo frames on the Win10 machine, nor on the TrueNAS VM instance running on Proxmox, and yet they hit 2.5Gbps without issue.