SOLVED TrueNAS SCALE using 10Gbe card limited to 300 Mb/s upload

[joeldc13]

Hi everyone,

First of all sorry for my english, not my native language.
I have an issue with my newly setup Truenas Scale 21.08-BETA.2 setup. It's a E3-1240 V2 system with 32 Gb of Ram and I am using a dual SFP+ 10 Gbe network card from dell that shows like that in lspci

Code:

01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries BCM57412 NetXtreme-E 10Gb RDMA Ethernet Controller
01:00.1 Ethernet controller: Broadcom Inc. and subsidiaries BCM57412 NetXtreme-E 10Gb RDMA Ethernet Controller (rev 01)

Card is this one : Dell 10Gbe Card

I have 2 others linux systems each using a 2.5 Gbe network card and when I test using iperf the link between the 2 2.5 Gbe network card, I have 2.3 Gb in both ways.

But when I try between my truenas server and one of this 2.5 Gbe server I have 2.3 Gb in download but only 300 Mb in upload when I host the iperf server on the truenas server.

proxmox is my 2.5Gb linux server, 192.168.1.15 is my truenas server

Code:

root@proxmox:~# iperf3 -c 192.168.1.15
Connecting to host 192.168.1.15, port 5201
[  5] local 192.168.1.6 port 58254 connected to 192.168.1.15 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   282 MBytes  2.37 Gbits/sec    0   1.02 MBytes
[  5]   1.00-2.00   sec   279 MBytes  2.34 Gbits/sec    0   1.38 MBytes
[  5]   2.00-3.00   sec   281 MBytes  2.36 Gbits/sec    0   1.38 MBytes
[  5]   3.00-4.00   sec   280 MBytes  2.35 Gbits/sec    0   1.49 MBytes
[  5]   4.00-5.00   sec   281 MBytes  2.36 Gbits/sec    0   1.49 MBytes
[  5]   5.00-6.00   sec   280 MBytes  2.35 Gbits/sec    0   1.49 MBytes
[  5]   6.00-7.00   sec   281 MBytes  2.36 Gbits/sec    0   1.49 MBytes
[  5]   7.00-8.00   sec   280 MBytes  2.35 Gbits/sec    0   1.49 MBytes
[  5]   8.00-9.00   sec   281 MBytes  2.36 Gbits/sec    0   1.49 MBytes
[  5]   9.00-10.00  sec   280 MBytes  2.35 Gbits/sec    0   1.49 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.74 GBytes  2.35 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  2.74 GBytes  2.35 Gbits/sec                  receiver

iperf Done.

And on the other way

Code:

root@proxmox:~# iperf3 -c 192.168.1.15 -R
Connecting to host 192.168.1.15, port 5201
Reverse mode, remote host 192.168.1.15 is sending
[  5] local 192.168.1.6 port 58258 connected to 192.168.1.15 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  42.9 MBytes   360 Mbits/sec
[  5]   1.00-2.00   sec  41.6 MBytes   349 Mbits/sec
[  5]   2.00-3.00   sec  43.2 MBytes   362 Mbits/sec
[  5]   3.00-4.00   sec  41.7 MBytes   350 Mbits/sec
[  5]   4.00-5.00   sec  42.3 MBytes   354 Mbits/sec
[  5]   5.00-6.00   sec  43.1 MBytes   362 Mbits/sec
[  5]   6.00-7.00   sec  42.8 MBytes   359 Mbits/sec
[  5]   7.00-8.00   sec  42.7 MBytes   358 Mbits/sec
[  5]   8.00-9.00   sec  42.7 MBytes   358 Mbits/sec
[  5]   9.00-10.00  sec  42.0 MBytes   352 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   426 MBytes   357 Mbits/sec  2581             sender
[  5]   0.00-10.00  sec   425 MBytes   356 Mbits/sec                  receiver

iperf Done.

I have tried with 2 other 10Gbe cards and I have the exact same results, tried with differents SFP+ cables (DAC cables) and always the same

here is the result of the ethtool on the nic

Code:

sudo ethtool enp1s0f1np1
Settings for enp1s0f1np1:
        Supported ports: [ FIBRE ]
        Supported link modes:   1000baseT/Full
                                10000baseT/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Supported FEC modes: RS  BASER
        Advertised link modes:  1000baseT/Full
                                10000baseT/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: RS         BASER
        Speed: 10000Mb/s
        Duplex: Full
        Auto-negotiation: on
        Port: Direct Attach Copper
        PHYAD: 1
        Transceiver: internal
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00002001 (8193)
                               drv hw
        Link detected: yes

This thing is driving me crazy, I don't understand what's going on, I have a new proxmox server that I am building with the same 10Gbe network card
If someone have any idea.

Thanks

 

[morganL]

Can you verify by running iperf from the truenas sever.
Is it only iperf3 that appears to be slow or is it also storage operations?

 

[joeldc13]

Hi, yes I have also verified that, same issue

Code:

jdacosta@truenas:~$ iperf3 -c 192.168.1.6
Connecting to host 192.168.1.6, port 5201
[  5] local 192.168.1.15 port 52796 connected to 192.168.1.6 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  44.5 MBytes   374 Mbits/sec  268    113 KBytes
[  5]   1.00-2.00   sec  42.5 MBytes   357 Mbits/sec  280    106 KBytes
[  5]   2.00-3.00   sec  42.9 MBytes   360 Mbits/sec  299   82.0 KBytes
[  5]   3.00-4.00   sec  43.1 MBytes   361 Mbits/sec  348   80.6 KBytes
[  5]   4.00-5.00   sec  42.7 MBytes   358 Mbits/sec  308   70.7 KBytes
[  5]   5.00-6.00   sec  42.6 MBytes   357 Mbits/sec  314   70.7 KBytes
[  5]   6.00-7.00   sec  40.3 MBytes   338 Mbits/sec  333   60.8 KBytes
[  5]   7.00-8.00   sec  42.8 MBytes   359 Mbits/sec  351   74.9 KBytes
[  5]   8.00-9.00   sec  42.1 MBytes   353 Mbits/sec  329   65.0 KBytes
[  5]   9.00-10.00  sec  42.0 MBytes   352 Mbits/sec  297   79.2 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   425 MBytes   357 Mbits/sec  3127             sender
[  5]   0.00-10.00  sec   425 MBytes   356 Mbits/sec                  receiver

iperf Done.

Unfortunately it's not only iperf3 that is impacted, it's also the storage operations, i ran into doing iperf3 tests because I wasn't having so much speed on the storage operations....

 

[joeldc13]

And one last information, the driver being used for the card is bnxt_en

 

[jgreco]

Well, on the FreeBSD side of things, the Broadcom cards are known to suck badly. I was hoping maybe they worked better under Linux.

 

[joeldc13]

Update on this topic, I have just installed my new proxmox server, with the same dell 10Gbe card. And now I have full 10Gb between them, on both sides (up and down).....

It seems that my issue is not on the truenas side, that's very wierd, maybe it's the switch (unifi aggregation switch), or the Mikrotik SFP+ RJ45 Modules (that are downgrading from 10Gbe to 2.5 Gbe), the thing that makes me confused is that between the two 2.5Gb servers I have full 2.5Gb on both sides, up and down.

Will try to dig into, but I think it's not on the truenas side.

 

[shadofall]

Likely due to Iperf's default tcp window size. it can vAry by OS, and the different cards could result in different windows sizes if it tried to adjust automatically in general its best to specify a tcp window size and or parallel streams when testing with iperf

 

[joeldc13]

In fact, I found the culprit, it's the 2.5Gb USB network sticks I was using, they are realtek RTL8156B based.
They are connected to a Unifi Aggregation 10Gb SFP+ switch using a Microtik SFP+ 10Gbe module.

When 2 of them are talking to each other, there is no issue, when mixing with 10 Gb or even 1Gb network card, they screw up on one side.

I have tried them on my laptop connected to my 1Gb Switch and I am having 100 Mb upload.... switched back to my 1gb ethernet adapater on my laptop and have 800Mb upload back.... And it's the same if I am connected to the SFP module on the 10Gb switch, on the 1Gb switch or on the router backplane directly (UDM Pro).

It's a shame because i was planning to use these adapters on all my laptops to give them 2.5Gb connectivity....

 

[pixelwave]

Hi,

I just stumbled across your post. I think I have similar issue / setup:

I installed one Mellanox ConnectX-4 (MCX4121A-ACAT) in my Workstation (W11Pro) and one into my Server (Proxmox with TrueNAS Scale as VM).

Both are connected to a 10G Managed Switch (USW-Aggregation) with SFP+ DAC cables.

If I upload to my server from the workstation I get the expected 10G speeds:

If I download from the server it seems oddly capped at 300 MB/s:

Download from / uploading to the same NVMe SMB share on the TureNAS ... ?!? Both systems NVMe only - no spinning rust ...

iperf3 results:

Workstation in CLIENT-MODE:

Server in CLIENT-MODE: