25G Performance is worse than 10G

[robinyoublind]

I've used a 10G setup for TrueNas and have been able to consistently get 1GB/s speeds transferring between 2 NVMEs. I upgraded my network to support 25G hoping to get around 2GB/s, but the speeds are even slower now and I'm not sure what's wrong. Right now I am capping out at around 500MB/s file transfer speeds. Running 5 threads of iperf3 shows about 13gbit/sec which is not great, but doesn't explain why it would be so slow. So far on my desktop I have run TCP optimizer which had a bit of improvement (before iperf was around 9gbit/sec) and I added the tunables to the server from High Speed Networking Tuning to maximize your 10G, 25G, 40G networks.

My hope is to get 2GB/s speeds on this, and eventually upgrade the server to something more modern with 100G to allow multiple 25G connections to myself and a video editor.

Server:
Dell R720
2x E5-2667 v2
64GB Ram
2TB 970 EVO
Mellanox MCX4121A-ACAT CX4121A Dual-Port ConnectX-4 Lx 25GbE

PC:
Ryzen 9 3900XT
64GB Ram
4TB Sabrent Rocket
Mellanox MCX4121A-ACAT CX4121A Dual-Port ConnectX-4 Lx 25GbE

Network:
MIKROTIK CRS510-8XS-2XQ-IN
FS.com generic 25G fiber transceivers

 

[Ericloewe]

The Mellanoxes are kinda janky, but those results are pretty sucky. Are the 10 GbE results using the same switch (and switch config)?