2.5gbe I225-V slow

[warriorcookie]

Setting up a new backup server and getting really slow iperf3 speeds from my 2.5gbe link.

Terramaster F4-223. Celeron N4505, 16gb DDR4, I225-V 2.5GBE
TrueNAS-SCALE-23.10.2 baremetal

Getting slow transfer speeds.
dmesg shows igc link is Up 2500 Mbps Full Duplex.
Cubic flow control was enabled by default, tried dctcp with little change.
Added postinit modprobe tcp_dctcp

and loaded the following sysctl (saw ~30% improvement here):
net.core.netdev_max_backlog = 300000
net.core.optmem_max = 268435456
net.core.rmem_default = 212992
net.core.rmem_max = 134217728
net.core.somaxconn = 8192
net.core.wmem_default = 212992
net.core.wmem_max = 134217728
net.ipv4.tcp_congestion_control = dctcp
net.ipv4.tcp_rmem = 4096 87380 134217728
net.ipv4.tcp_sack = 0
net.ipv4.tcp_wmem = 4096 65536 134217728

So now iperf3 -P 10 shows 738 Mbps between the new backup server and my main server. iperf3 shows > 9Gbps from 2 other machines to the main server on same switch.

I've read some comments that the I225-V is garbage, but I've also read others getting full bandwidth. Any pointers in where I can look would be appreciated.

 

[warriorcookie]

Upgraded to 24.04-RC.1 and no change.

 

[ChrisRJ]

How are the machines connected?

 

[winnielinnie]

Not sure what to say. On my TrueNAS Core system, with the same NIC chip (I225-V), directly connected to a Linux client with the same 2.5-GbE (nothing in between), I get sustained 2.48 Gpbs on iperf tests.

The only things that stand out in my situation:

• Direct connection (a single Cat6A cable from server to client)
• Server NIC and client NIC are set to 9000 MTU.

 

[warriorcookie]

How are the machines connected?

Mainserver connected to D-Link switch via DAC
Second machine used for testing connected to same D-Link switch via DAC

Problematic machine connected via 1 meter cat6 patch cable to same D-Link switch. I'm using a sfp+ to NBASE-T copper transceiver from FS.com. I've tested it with my QNAP QNA-UC5G1T and a windows machine to verify the transceiver will connect at 2.5G speed.

 

[warriorcookie]

Not sure what to say. On my TrueNAS Core system, with the same NIC chip (I225-V), directly connected to a Linux client with the same 2.5-GbE (nothing in between), I get sustained 2.48 Gpbs on iperf tests.

The only things that stand out in my situation:

• Direct connection (a single Cat6A cable from server to client)
• Server NIC and client NIC are set to 9000 MTU.

This evening I'll test direct connection without the switch and report back.

I tried jumbo frames on both clients and switch with no improvement. For now I've set back to 1500 MTU.

 

[warriorcookie]

I have a supermicro X11SDV-8C-TP8F with 2x 10g SFP+ ports and 2x 10g ethernet ports I've used for the following testing:

Terramaster 2.5g <-> Cat6 <-> Supermicro 10g ethernet connects at 1,000Mbps (iperf3 ~700Mbps)
Terramaster 2.5g <-> Cat6 <-> FS.com SFP-10G-T Transceiver <-> Supermicro 10g SFP+ connects at 2,500Mbps (iperf3 ~700Mbps)

Windoze <-> USB QNA-UC5G1T <-> Cat6 <-> Terramaster 2.5g connects at 2,500Mbps (iperf3 ~2.3Gbps)
Windoze <-> USB QNA-UC5G1T <-> Cat6 <-> FS.com SFP-10G-T Transceiver <-> Supermicro 10g SFP+ (iperf3 ~4.5Gbps)
Windoze <-> USB QNA-UC5G1T (force 2.5g) <-> Cat6 <-> FS.com SFP-10G-T Transceiver <-> Supermicro 10g SFP+ (iperf3 ~2.3Gbps)

Now after all this testing, I plug the Terramaster back into the transceiver and back into the switch and run iperf3 for a quick sanity check, and boom! 2.3Gbps!

wtf???

 

[NickF]

The Celeron N4505 may be the culprit here. IPerf3 is a single threaded process and the single threaded performance/IPC of this chip is probably the limiting factor.

Can you run htop while running your iPerf test to see if it’s hittting 100% usage?

 

[warriorcookie]

The Celeron N4505 may be the culprit here. IPerf3 is a single threaded process and the single threaded performance/IPC of this chip is probably the limiting factor.

Can you run htop while running your iPerf test to see if it’s hittting 100% usage?

I've been getting consistent 2.5gb iperf3 speeds now since yesterday. Each core bounces around 30% during transfer.

 

[NickF]

There maybe was a process running on the Terramaster eating some CPU cycles before? Did you start any services or apps or VMs previously that are not started now?

 

[warriorcookie]

There maybe was a process running on the Terramaster eating some CPU cycles before? Did you start any services or apps or VMs previously that are not started now?

It's a fresh install, with no VM's. I hadn't even enabled SMB or NFS yet!
I monitored CPU through the web portal and both cores always stayed well below 50%, usually closer to 30%.

I suspect it was a handshake issue with the SFP+ to ethernet transceiver. Maybe it wasn't negotiating 2.5g correctly so was limited to 1g.

Either way, it's still working fine today. Overall I'm pretty impressed with this little machine. Certainly performs well for a backup box.