Help understanding the bottleneck 100Gb/s

[jarble]

It seems that iperf3 exactly matches the file transfer behavior 10 down 20 up regardless of the P8 flag

Code:

C:\Users\Jarble\Downloads\iperf-3.1.3-win64\iperf-3.1.3-win64>iperf3.exe -c 10.40.0.68 -w1M -R
Connecting to host 10.40.0.68, port 5201
Reverse mode, remote host 10.40.0.68 is sending
[  4] local 10.40.0.133 port 56193 connected to 10.40.0.68 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  1.12 GBytes  9.65 Gbits/sec
[  4]   1.00-2.00   sec  1.17 GBytes  10.1 Gbits/sec
[  4]   2.00-3.00   sec  1.20 GBytes  10.3 Gbits/sec
[  4]   3.00-4.00   sec  1.19 GBytes  10.2 Gbits/sec
[  4]   4.00-5.00   sec  1.18 GBytes  10.1 Gbits/sec
[  4]   5.00-6.00   sec  1.18 GBytes  10.2 Gbits/sec
[  4]   6.00-7.00   sec  1.19 GBytes  10.2 Gbits/sec
[  4]   7.00-8.00   sec  1.18 GBytes  10.1 Gbits/sec
[  4]   8.00-9.00   sec  1.19 GBytes  10.2 Gbits/sec
[  4]   9.00-10.00  sec  1.18 GBytes  10.1 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  11.8 GBytes  10.1 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  11.8 GBytes  10.1 Gbits/sec                  receiver

iperf Done.

C:\Users\Jarble\Downloads\iperf-3.1.3-win64\iperf-3.1.3-win64>iperf3.exe -c 10.40.0.68 -w1M
Connecting to host 10.40.0.68, port 5201
[  4] local 10.40.0.133 port 56273 connected to 10.40.0.68 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  2.27 GBytes  19.5 Gbits/sec
[  4]   1.00-2.00   sec  2.08 GBytes  17.8 Gbits/sec
[  4]   2.00-3.00   sec  2.06 GBytes  17.7 Gbits/sec
[  4]   3.00-4.00   sec  2.09 GBytes  17.9 Gbits/sec
[  4]   4.00-5.00   sec  2.07 GBytes  17.8 Gbits/sec
[  4]   5.00-6.00   sec  2.07 GBytes  17.8 Gbits/sec
[  4]   6.00-7.00   sec  2.09 GBytes  18.0 Gbits/sec
[  4]   7.00-8.00   sec  2.09 GBytes  18.0 Gbits/sec
[  4]   8.00-9.00   sec  2.10 GBytes  18.0 Gbits/sec
[  4]   9.00-10.00  sec  2.10 GBytes  18.0 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  21.0 GBytes  18.1 Gbits/sec                  sender
[  4]   0.00-10.00  sec  21.0 GBytes  18.1 Gbits/sec                  receiver

iperf Done.

 

[NickF]

Let's just say there's a reason why AMD bought Pensando

AMD Pensando - Wire speed 100gbe with no CPU overhead is here today

Some crazy tech for you crazy techies! AMD brings their game when it comes to newer and faster server tech!*********************************Check us out onli...

www.youtube.com

I'm still on the side of the fact it's an IPC problem. The solution is to "work around it" by engineering a solution to parallelize your operations.

You're going to have to try and parallelize your operations with some VLANs. The Mellanox adapter natively supports/accelerates VLAN offloading (q-tags), so there's not a lot of extra cost. That way, even though you have a single layer 1 interface connecting the two - you will have multiple logical ones at layer 3. I would make 4 for what you are trying to do, you can imagine it kinda like a virtual 100 Gb to 4X 25 Gb DAC cable.

ESPCommunity

enterprise-support.nvidia.com

Setting Up a Network VLAN

Provides instructions on setting up a network VLAN interface on TrueNAS CORE.

www.truenas.com

Using SMB multi-channel or MPIO with iSCSI (all higher-level protocols, layers 4-7) you will be able to leverage those lower layer changes we made. In other words, you are leveraging parallelization at the application layer (7), in your testing the -P flag in FIO is a similar concept. More importantly though, you are now able to leverage parallelization at the network stack (2,3,4) layer.

@jgreco 's article is designed to help you increase the performance of a single adapter by modifying values in the network stack. His approach combined with parallelization, should get you closer. https://www.truenas.com/community/r...ng-to-maximize-your-10g-25g-40g-networks.207/

 

[jarble]

An update here I bit the bullet and decided to test out scale and off the bat I am getting "better" performance breaking the 10 gbits read but my writes have dropped a bit. This seems to confirm my suspicion that it is not a ipc bottleneck but a combination of other factors. I am not ruling out ipc as I am sure it plays in here but not at the weird 10g limit.

 

[indy]

Regarding your iperf3 results: I was having bad performance due to dropped packets on my 25G setup, which was fixed by enabling flow control.
Your results do not show dropped packets though.

Some other topics you could look into: interrupt moderation and maybe jumbo packets.

Can you post a CrystalDiskMark screenshot
- with "Test Data" set to zeros
- writing to an lz4 compressed dataset via smb

 

[jarble]

Regarding your iperf3 results: I was having bad performance due to dropped packets on my 25G setup, which was fixed by enabling flow control.
Your results do not show dropped packets though.

Some other topics you could look into: interrupt moderation and maybe jumbo packets.

Can you post a CrystalDiskMark screenshot
- with "Test Data" set to zeros
- writing to an lz4 compressed dataset via smb

 

[NickF]

Your performance looks really good for a single client. I'd bet if you hit it hard like that from 2 clients at the same time, you'll see theres still gas in the tank and both running concurrently doesn't just halve you single client performance.

Scale out your test environment now.

 

[buffer_overrun]

I have dual 25g working over smb multichannel and I also had problems capping 10g reads because of windows defender. Reboot into windows safe mode with networking and rerun iperf3 tests. I can do 24gb/sec both ways like this but even with defender disabled I see more like 17-20 only. I could see 6-10gb/sec with it enabled. I see some people suggesting to boot to a Linux bootable thumbdrive and testing Linux to Linux.