100 gbe mellanox maxes out at 22gb/s

[erdas]

The 455A works on both the windows client and the truenas server.
They are connected via one twinax (DAC). Fiber is the same speed.

iperf3.exe -c 111.111.111.222 -t 60 -P 8

22gb/s is far from what I expected from a 100gb/s even with ETH and SMB overhead.
Both machines have beefy cpu where one core barely breaks 30%, both cards run firmware 3.0.25668 in ETH mode.
I don't know which card or OS is at fault.
On windows device manager/mellanox/info shows link speed at 100gbe/full duplex. truenas shows this

Can someone tell me how to get closer to the full 100gb/s or at least identify where the problem is?
Thanks.

Hardware: client: windows 10 pro 13600k z790-taichi with the mellanox 455A-ECAT slotted in the top cpu-pcie x8 (max speed is about 70gb/s), the client 455A is set to ETH, running firmware v3 and directly connected to the server via a DAC cable rated at 100gb/s. The server is a 1920x on a x399-taichi and the server's 455A-ECAT is slotted in a full 16x lanes pcie gen4, there is 16gb ddr4, the main volume is a stripeed 4x 1TB m.2 rated at 3GB/s which, when the machine was running windows, was benchmarked at 12GB/s sequential RW with 8 threads in crystaldiskmark. They're slotted in a 4x m.2 pcie card slotted in a pcie gen4 set to x4x4x4x4 bifurcation.

 

[Constantin]

Apologies for my ignorance but are those 4 separate but aggregated links or four sets of twinax needed to satisfy one Interface? I’ve never worked with something that fast.

But whenever I hear multiple physical cables being needed to make connections between equipment it suggests some form of LAGG aggregation, which will max out at the top capacity of one physical connection per user stream, really shining when there are a lot of parallel streams going to and from the server.

I look forward to learning about this technology though I doubt I will experience it anytime soon.

 

[erdas]

Apologies for my ignorance but are those 4 separate but aggregated links or four sets of twinax needed to satisfy one Interface? I’ve never worked with something that fast.

But whenever I hear multiple physical cables being needed to make connections between equipment it suggests some form of LAGG aggregation, which will max out at the top capacity of one physical connection per user stream, really shining when there are a lot of parallel streams going to and from the server.

I look forward to learning about this technology though I doubt I will experience it anytime soon.

It's 1 physical DAC cable and 1 network card on each end. It's just that 100Gbe is internally made of 4x25gbe links and the 22gb/s look like only 1 of these links is active. It's confusing how I worded it so I changed the description.

 

[artlessknave]

not my area of great knowledge, but I would think this needs aggregation, on both sides. windows clients don't do aggregation, iirc, only windows server, and probably only with propriety drives.
22Gb/s sounds like to be the max for the one channel that is connected, which is exactly what I would expect for no aggregation.
100GB is typically used for switch/switch backend, or switch/server backend.
I do not think this will work as you have it configured.

you will also struggle to fill a 22Gb/s link beyond iperf. as you didn't follow the forum rules and put your hardware, I'm going to assume you arent running a TrueNAS all SSD/NVME array, as that is what you would likely need to even begin to saturate a 25Gb/s link, not to mention 100Gb/s.
if you are running platters? yea, no; not unless you have like 100 15k mirrors or something. maybe.

 

[erdas]

not my area of great knowledge, but I would think this needs aggregation, on both sides. windows clients don't do aggregation, iirc, only windows server, and probably only with propriety drives.
22Gb/s sounds like to be the max for the one channel that is connected, which is exactly what I would expect for no aggregation.
100GB is typically used for switch/switch backend, or switch/server backend.
I do not think this will work as you have it configured.

you will also struggle to fill a 22Gb/s link beyond iperf. as you didn't follow the forum rules and put your hardware, I'm going to assume you arent running a TrueNAS all SSD/NVME array, as that is what you would likely need to even begin to saturate a 25Gb/s link, not to mention 100Gb/s.
if you are running platters? yea, no; not unless you have like 100 15k mirrors or something. maybe.

It's not an aggregation, it's only 1x 100gbe cable.
I update the first post with all hardware info, thanks.

 

[artlessknave]

I am quite sure that the 100Gbe cables (QSFP) ARE 4 links, not 1. this is why breakout cables to 4x10/25Gbe exist.
they are aggregating the 4 links together to make 100Gbe, and if either side cannot do this, you will get only 1 link.

 

[erdas]

I am quite sure that the 100Gbe cables (QSFP) ARE 4 links, not 1. this is why breakout cables to 4x10/25Gbe exist.
they are aggregating the 4 links together to make 100Gbe, and if either side cannot do this, you will get only 1 link.

I thought they were like pcie, you know, as many lanes with the system dealing with bifurcation but yeah if they're physically 4 cables in the fat DAC that would make sense why aggregation is needed. I'll look into this and I think aggregation is ok on windows 10 pro, at least the funky SMB multi link. Thanks.