Slow iperf performance in Scale compared to Core

roysen

roysen

Dabbler
Joined
Dec 11, 2021
Messages
15
I'm getting poor performance in iperf3 compared to what I got when running TrueNAS Core on the same system/hardware. I have a 10GBe network and is having an Intel X520-DA adapter on my TrueNas Scale. Scale is not running the latest ixgbe driver from Intel but I guess it is tuned for its use. Do I need some tweaking of the driver?
My speed on Truenas Core was always close to 10GiB but now it is more like 6-7GiB. Any advise?

Some Intel X520 information from my system:
Code:
sudo ethtool -i enp2s0f1
driver: ixgbe
version: 5.10.70+truenas
firmware-version: 0x0001546c
expansion-rom-version:
bus-info: 0000:02:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

sudo ethtool enp2s0f1
Settings for enp2s0f1:
        Supported ports: [ FIBRE ]
        Supported link modes:   10000baseT/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: No
        Supported FEC modes: Not reported
        Advertised link modes:  10000baseT/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: No
        Advertised FEC modes: Not reported
        Speed: 10000Mb/s
        Duplex: Full
        Auto-negotiation: off
        Port: Direct Attach Copper
        PHYAD: 0
        Transceiver: internal
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes


My iperf3 test:
Code:
roysen@truenas:~$ iperf3 -c 192.168.68.100 -R
Connecting to host 192.168.68.100, port 5201
Reverse mode, remote host 192.168.68.100 is sending
[  5] local 192.168.68.5 port 55834 connected to 192.168.68.100 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   720 MBytes  6.04 Gbits/sec                 
[  5]   1.00-2.00   sec   800 MBytes  6.71 Gbits/sec                 
[  5]   2.00-3.00   sec   779 MBytes  6.53 Gbits/sec                 
[  5]   3.00-4.00   sec   757 MBytes  6.35 Gbits/sec                 
[  5]   4.00-5.00   sec   708 MBytes  5.94 Gbits/sec                 
[  5]   5.00-6.00   sec   797 MBytes  6.68 Gbits/sec                 
[  5]   6.00-7.00   sec   834 MBytes  6.99 Gbits/sec                 
[  5]   7.00-8.00   sec   750 MBytes  6.29 Gbits/sec                 
[  5]   8.00-9.00   sec   718 MBytes  6.02 Gbits/sec                 
[  5]   9.00-10.00  sec   705 MBytes  5.91 Gbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  7.39 GBytes  6.35 Gbits/sec                  sender
[  5]   0.00-10.00  sec  7.39 GBytes  6.35 Gbits/sec                  receiver

iperf Done.
roysen@truenas:~$ iperf3 -c 192.168.68.100
Connecting to host 192.168.68.100, port 5201
[  5] local 192.168.68.5 port 55902 connected to 192.168.68.100 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   720 MBytes  6.04 Gbits/sec    0    419 KBytes       
[  5]   1.00-2.00   sec   727 MBytes  6.10 Gbits/sec    0    419 KBytes       
[  5]   2.00-3.00   sec   730 MBytes  6.12 Gbits/sec    0    419 KBytes       
[  5]   3.00-4.00   sec   731 MBytes  6.13 Gbits/sec    0    419 KBytes       
[  5]   4.00-5.00   sec   718 MBytes  6.02 Gbits/sec    0    419 KBytes       
[  5]   5.00-6.00   sec   709 MBytes  5.95 Gbits/sec    0    419 KBytes       
[  5]   6.00-7.00   sec   710 MBytes  5.95 Gbits/sec    0    419 KBytes       
[  5]   7.00-8.00   sec   704 MBytes  5.90 Gbits/sec    0    419 KBytes       
[  5]   8.00-9.00   sec   720 MBytes  6.04 Gbits/sec    0    419 KBytes       
[  5]   9.00-10.00  sec   618 MBytes  5.18 Gbits/sec    0    419 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  6.92 GBytes  5.94 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  6.92 GBytes  5.94 Gbits/sec                  receiver

iperf Done.
 
Kris Moore

Kris Moore

SVP of Engineering
Administrator
Moderator
iXsystems
Joined
Nov 12, 2015
Messages
1,471
We've not done really much auto-tuning of specific things in SCALE yet... Be sure to check the MTU setting though. On my 10GBE card I had to set MTU of 9000 then I was able to get expected performance back.
 
roysen

roysen

Dabbler
Joined
Dec 11, 2021
Messages
15
Kris Moore said:
We've not done really much auto-tuning of specific things in SCALE yet... Be sure to check the MTU setting though. On my 10GBE card I had to set MTU of 9000 then I was able to get expected performance back.
Click to expand...
My MTU is at 9000. Maybe I need to go back to Core even if I struggle so bad with FreeBSD. Linux is much more familiar to me. :grin:
 
M

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,694
roysen said:
My MTU is at 9000. Maybe I need to go back to Core even if I struggle so bad with FreeBSD. Linux is much more familiar to me. :grin:
Click to expand...
You might need some patience. After RC2 is completed (next week), we'll be starting to more systematically test, measure and adddress performance issues. Nightlies will get improvements and then the February release and updates.

FreeBSD has had years of this focus.
 
roysen

roysen

Dabbler
Joined
Dec 11, 2021
Messages
15
morganL said:
You might need some patience. After RC2 is completed (next week), we'll be starting to more systematically test, measure and adddress performance issues. Nightlies will get improvements and then the February release and updates.
Click to expand...
I will not give up so easily and will test some tweaking of my network to see if I can get it better. My SMB performance is even worse so there will be some testing.
 
Kris Moore

Kris Moore

SVP of Engineering
Administrator
Moderator
iXsystems
Joined
Nov 12, 2015
Messages
1,471
roysen said:
I will not give up so easily and will test some tweaking of my network to see if I can get it better. My SMB performance is even worse so there will be some testing.
Click to expand...
Thats the spirit :)

Let us know how we can help. If you want to troubleshoot more in real-time, I'd suggest hopping on our Discord Server as well.
 
roysen

roysen

Dabbler
Joined
Dec 11, 2021
Messages
15
I tried to lower my MTU on my desktop and the Scale server and it seems like MTU 2040 is giving me the best speed so far.

My latest run of iperf3 Bad result in reverse mode:
Code:
PS C:\Users\royar> iperf3 -c 192.168.68.5
Connecting to host 192.168.68.5, port 5201
[  4] local 192.168.68.100 port 61110 connected to 192.168.68.5 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  1012 MBytes  8.49 Gbits/sec
[  4]   1.00-2.00   sec  1.03 GBytes  8.87 Gbits/sec
[  4]   2.00-3.00   sec   983 MBytes  8.25 Gbits/sec
[  4]   3.00-4.00   sec   964 MBytes  8.09 Gbits/sec
[  4]   4.00-5.00   sec   970 MBytes  8.14 Gbits/sec
[  4]   5.00-6.00   sec   981 MBytes  8.23 Gbits/sec
[  4]   6.00-7.00   sec   982 MBytes  8.24 Gbits/sec
[  4]   7.00-8.00   sec   970 MBytes  8.14 Gbits/sec
[  4]   8.00-9.00   sec   996 MBytes  8.35 Gbits/sec
[  4]   9.00-10.00  sec   932 MBytes  7.82 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  9.62 GBytes  8.26 Gbits/sec                  sender
[  4]   0.00-10.00  sec  9.62 GBytes  8.26 Gbits/sec                  receiver

iperf Done.
PS C:\Users\royar> iperf3 -c 192.168.68.5 -R
Connecting to host 192.168.68.5, port 5201
Reverse mode, remote host 192.168.68.5 is sending
[  4] local 192.168.68.100 port 61132 connected to 192.168.68.5 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   113 MBytes   945 Mbits/sec
[  4]   1.00-2.00   sec   112 MBytes   942 Mbits/sec
[  4]   2.00-3.00   sec   112 MBytes   942 Mbits/sec
[  4]   3.00-4.00   sec   110 MBytes   920 Mbits/sec
[  4]   4.00-5.00   sec   110 MBytes   925 Mbits/sec
[  4]   5.00-6.00   sec  83.5 MBytes   701 Mbits/sec
[  4]   6.00-7.00   sec   112 MBytes   942 Mbits/sec
[  4]   7.00-8.00   sec   112 MBytes   941 Mbits/sec
[  4]   8.00-9.00   sec   112 MBytes   942 Mbits/sec
[  4]   9.00-10.00  sec   110 MBytes   919 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  1.06 GBytes   913 Mbits/sec    0             sender
[  4]   0.00-10.00  sec  1.06 GBytes   912 Mbits/sec                  receiver

iperf Done.
 
roysen

roysen

Dabbler
Joined
Dec 11, 2021
Messages
15
I had a 1Gb nic that I used as a bridge for my VM and it used that one for my iperf3 even if I used the Bind options in iperf. So I disabled my bridge and my reverse speed is now like this:
Code:
PS C:\Users\royar> iperf3 -c 192.168.68.5 -R
Connecting to host 192.168.68.5, port 5201
Reverse mode, remote host 192.168.68.5 is sending
[  4] local 192.168.68.100 port 61643 connected to 192.168.68.5 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   948 MBytes  7.95 Gbits/sec
[  4]   1.00-2.00   sec   887 MBytes  7.44 Gbits/sec
[  4]   2.00-3.00   sec   846 MBytes  7.10 Gbits/sec
[  4]   3.00-4.00   sec   846 MBytes  7.09 Gbits/sec
[  4]   4.00-5.00   sec   917 MBytes  7.69 Gbits/sec
[  4]   5.00-6.00   sec   916 MBytes  7.69 Gbits/sec
[  4]   6.00-7.00   sec   928 MBytes  7.78 Gbits/sec
[  4]   7.00-8.00   sec   891 MBytes  7.47 Gbits/sec
[  4]   8.00-9.00   sec   848 MBytes  7.12 Gbits/sec
[  4]   9.00-10.00  sec   925 MBytes  7.76 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  8.74 GBytes  7.51 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  8.74 GBytes  7.51 Gbits/sec                  receiver

iperf Done.
 
You must log in or register to reply here.

Similar threads

W
TrueNas Mini XL not getting 10 gbit/s, only 3
Replies
0
Views
837
WaughTek
W
F
10gbe performance differences
Replies
2
Views
2K
foureight84
F
J
SOLVED TrueNAS SCALE using 10Gbe card limited to 300 Mb/s upload
Replies
8
Views
5K
pixelwave
pixelwave
C
SOLVED Speed drops off with SMB over 10Gb.
Replies
12
Views
4K
NugentS
NugentS
J
Slow SMB Transfer Speeds Any Suggestion?
Replies
2
Views
676
Junior007
J
Top