Only 155MB/s On 2.5Gb LAN with 10Gb NIC

[XenoMorpHx]

I am using Truenas for quite some years now and all worked just fine on a 1Gb LAN setup, max speed of 110-115MB/s.
Recently I upgraded my Internet connection to 2Gb, so to also upgrade my LAN I replaced some of my 1Gb hardware with 2.5Gb and 10Gb capable switches ad replaced cabels with CAT6 ones.

I bought an X550-T1 10Gb (Genuine one) NIC for the NAS and for my pc I would still use the 2.5NIC on the mainboard.
NAS setup (I know, it's a few years old, but still doing it's job):
Asrock AM1H-ITX
AMD Athlon 5350
8GB DDR3 RAM
Bootdisc: Kingston 60GB SSD
Pool1: 2x 4TB WD Red: WD-WCC7K5SJA717 / WD-WX52D21CSZCV (mirror)
Pool 2: 1x 4TB WD Red: WD-WCC7K3KZ84YN
Pool 3: 1x USB 3.1 (up to 5Gb/s), a 4TB Crucial P3 NVMe SSD in an USB enclosure.
The X550-T1 is in a 1 x PCI Express 2.0 x16 Slot (PCIE1 @ x4 mode). Max speed at 2 GB/s.

When I transfer a file to either of the drives (even the USB-SSD) the max speed is 150-156MB/s on a 2.5Gb connection. On a 1Gb connection I get about 110MB/s. So there is some gain you'd say.
On iPerf I get:

Code:

Server = Truenas
Client = PC
------------------------------------------------------------
Client connecting to 192.168.10.12, TCP port 5201
Sending 131072 byte packets
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[5480] local 192.168.10.67 port 57196 connected with 192.168.10.12 port 5201
[ID]    Interval    Transfer    Bandwidth
[5480] 0.0-2.0 sec   309 MBytes  1.30 Gbits/sec
[5480] 2.0-4.0 sec   310 MBytes  1.30 Gbits/sec
[5480] 4.0-6.0 sec   306 MBytes  1.28 Gbits/sec
[5480] 6.0-8.0 sec   307 MBytes  1.29 Gbits/sec
[5480] 8.0-10.0 sec   304 MBytes  1.27 Gbits/sec
[5480] 0.0-10.9 sec  1.64 GBytes  1.29 Gbits/sec

Code:

Server = PC
Client = Truenas
------------------------------------------------------------
Server listening on TCP port 5201
Receiving 131072 byte packets
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[5712] local ::ffff:192.168.10.67 port 5201 connected with ::ffff:192.168.10.12 port 28943
[5712] 0.0-2.0 sec   255 MBytes  1.07 Gbits/sec
[5712] 2.0-4.0 sec   254 MBytes  1.07 Gbits/sec
[5712] 4.0-6.0 sec   254 MBytes  1.07 Gbits/sec
[5712] 6.0-8.0 sec   251 MBytes  1.05 Gbits/sec
[5712] 8.0-10.0 sec   252 MBytes  1.06 Gbits/sec
[5712] 10.0-12.0 sec   250 MBytes  1.05 Gbits/sec

Searching the forum I've read that disabling encryption would help, so I tried that on the USB SSD, with no result, still 150MB/s.

I have no idea why the transfers are so low, I hope anyone could point me in the right direction.

 

[Tony-1971]

Hello,
When iperf is running, your CPU's are running at 100%? The bandwith seems very low.
Best Regards,
Antonio

 

[XenoMorpHx]

Hi, I just checked. Core #0 is maxing out @ 100%, the other 3 cores are @ 25-30%.

 

[Tony-1971]

If you run iperf with multiple streams you probably see the full bandwith.
But for one transfer you have reached the CPU limit: can you check the CPU usage when performing transfer?
You can try with higher MTU on both ethernet side (and switch).

 

[Etorix]

Your results are probably as good as your old NAS with only 8 GB RAM can achieve.
On the read side, 300 MB/s is in the right ballpark for a mirror of two old HDDs (ca. 150 MB/s).
On the write side, you're throttled by RAM, acting as write cache up to two transaction groups. The SSD is crippled by the USB link anyway.

 

[Ericloewe]

More than RAM, the processor itself is rather puny.

 

[XenoMorpHx]

I have set the MTU in Treunas to 9000 (it says 10Gase-T) and in windows to 9000. But no change.
When doing a transfer of a 8GB file, 1 core gets to 100% and the other 3 to 40-50ish.

What Etorix says about the RAM, I checked it during a file transfer and the Free RAM gets used by the ZFS cache, leaving 0.2GiB. Maybe it is the RAM bottle-necking the throughput?
As for the USB disc, I know it's slower cuz of the USB port, but still, it's rated 3.1 5Gb/s and should do 200-250MB/s. I am using it to test the connection speed HDD vs the SSD/USB.

I also changed the cable (what supposed to be a CAT6) with a different 5e cable, and I gained almost 8-10MB/s. But still not the 250-300MB on the USBdrive.

 

[XenoMorpHx]

Ok, this is weird...
After switching some cables, I ended up rebooting the server.

After rebooting it and plugging back in the original cable, I did another iperf test.

Code:

Client connecting to 192.168.10.12, TCP port 5201
Sending 131072 byte packets
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[5916] local 192.168.10.67 port 61207 connected with 192.168.10.12 port 5201
[5916] 0.0-2.0 sec   590 MBytes  2.47 Gbits/sec
[5916] 2.0-4.0 sec   588 MBytes  2.46 Gbits/sec
[5916] 4.0-6.0 sec   591 MBytes  2.48 Gbits/sec
[5916] 6.0-8.0 sec   591 MBytes  2.48 Gbits/sec
[5916] 8.0-10.0 sec   591 MBytes  2.48 Gbits/sec
[5916] 10.0-12.0 sec   584 MBytes  2.45 Gbits/sec
[5916] 12.0-14.0 sec   591 MBytes  2.48 Gbits/sec
[5916] 14.0-16.0 sec   591 MBytes  2.48 Gbits/sec
[5916] 16.0-18.0 sec   587 MBytes  2.46 Gbits/sec
[5916] 18.0-20.0 sec   591 MBytes  2.48 Gbits/sec
[5916] 20.0-22.0 sec   591 MBytes  2.48 Gbits/sec
[5916] 22.0-24.0 sec   591 MBytes  2.48 Gbits/sec
[5916] 24.0-26.0 sec   591 MBytes  2.48 Gbits/sec
[5916] 26.0-28.0 sec   591 MBytes  2.48 Gbits/sec
[5916] 28.0-30.0 sec   591 MBytes  2.48 Gbits/sec
[5916] 30.0-32.0 sec   591 MBytes  2.48 Gbits/sec
[5916] 32.0-34.0 sec   588 MBytes  2.47 Gbits/sec
[5916] 34.0-36.0 sec   591 MBytes  2.48 Gbits/sec
[5916] 36.0-38.0 sec   591 MBytes  2.48 Gbits/sec
[5916] 38.0-40.0 sec   589 MBytes  2.47 Gbits/sec
[5916] 40.0-42.0 sec   590 MBytes  2.48 Gbits/sec
[5916] 42.0-44.0 sec   591 MBytes  2.48 Gbits/sec
[5916] 44.0-46.0 sec   590 MBytes  2.48 Gbits/sec
[5916] 46.0-48.0 sec   591 MBytes  2.48 Gbits/sec
[5916] 48.0-50.0 sec   591 MBytes  2.48 Gbits/sec
[5916] 50.0-52.0 sec   591 MBytes  2.48 Gbits/sec
[5916] 52.0-54.0 sec   591 MBytes  2.48 Gbits/sec
[5916] 54.0-56.0 sec   591 MBytes  2.48 Gbits/sec
[5916] 56.0-58.0 sec   591 MBytes  2.48 Gbits/sec
[5916] 58.0-60.0 sec   591 MBytes  2.48 Gbits/sec
[5916] 0.0-60.7 sec  17.5 GBytes  2.48 Gbits/sec

A solid 2.5Gb connection. Doing 280MB/s on the SSD/USB (after the RAM is full it drops back to 170MB/s). And
I think the MTU setting needed a server reboot? That did the trick I think.

Thanks for the input all!

BTW, there is a NAS serverboard with an N100 on its way, already bought a 32GB DDR5 SODIMM, so I think that will do the trick much better.

 

[Etorix]

Please report on performance with the new server.

 

[Starrbuck]

You also may be limited by the Gen 2 of the motherboard's PCIe slot.

 

[XenoMorpHx]

No, that's not it. I thought that at first, but the theoretical bandwidth is 2000MB/s.
10Gb speed is approx. 1250MB/s.

To bad I don't have a second 10Gb NIC(these thing cost an arm and a leg) to test that out.

 

[Etorix]

Solarflare SFx122F or Chelsio T520 NICs can be found on eBay for $50 or less. Keep your limbs!