Poor 10GbE through with TrueNAS 12.0-RC1

D

dirtyfreebooter

Explorer
Joined
Oct 3, 2020
Messages
72
My setup

Dell R7515 with AMD 7402p running Proxmox 6.2 with Intel SFP+ dual-port NIC (82599ES 10-Gigabit SFI/SFP+ Network Connection).
TrueNAS is running in Proxmox with 32 threads, 224GB of memory, PCI passthrough of Dell HBA and PCI passthrough with 1 of the SFP+ ports.

Other machine is a Supermicro with Intel D-2141NT also running Proxmox 6.2. Everything is connected with OM3 fiber and UniFi XG-16 switch.

If I iperf3 between the hypervisors I get

Code:
Accepted connection from 192.168.42.11, port 57416
[  5] local 192.168.42.10 port 5201 connected to 192.168.42.11 port 57418
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  1.09 GBytes  9.40 Gbits/sec
[  5]   1.00-2.00   sec  1.10 GBytes  9.41 Gbits/sec
[  5]   2.00-3.00   sec  1.10 GBytes  9.41 Gbits/sec
[  5]   3.00-4.00   sec  1.10 GBytes  9.41 Gbits/sec
[  5]   4.00-5.00   sec  1.10 GBytes  9.42 Gbits/sec
[  5]   5.00-6.00   sec  1.10 GBytes  9.41 Gbits/sec
[  5]   6.00-7.00   sec  1.10 GBytes  9.41 Gbits/sec
[  5]   7.00-8.00   sec  1.10 GBytes  9.41 Gbits/sec
[  5]   8.00-9.00   sec  1.10 GBytes  9.41 Gbits/sec
[  5]   9.00-10.00  sec  1.10 GBytes  9.41 Gbits/sec
[  5]  10.00-10.00  sec  3.11 MBytes  9.24 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  11.0 GBytes  9.41 Gbits/sec


If I iperf3 between the the supermicro and TrueNAS, I get
Code:
Accepted connection from 192.168.42.2, port 22812
[ 5] local 192.168.42.10 port 5201 connected to 192.168.42.2 port 17033
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 463 MBytes 3.88 Gbits/sec
[ 5] 1.00-2.00 sec 537 MBytes 4.50 Gbits/sec
[ 5] 2.00-3.00 sec 529 MBytes 4.44 Gbits/sec
[ 5] 3.00-4.00 sec 541 MBytes 4.54 Gbits/sec
[ 5] 4.00-5.00 sec 528 MBytes 4.43 Gbits/sec
[ 5] 5.00-6.00 sec 547 MBytes 4.59 Gbits/sec
[ 5] 6.00-7.00 sec 515 MBytes 4.32 Gbits/sec
[ 5] 7.00-8.00 sec 524 MBytes 4.39 Gbits/sec
[ 5] 8.00-9.00 sec 539 MBytes 4.52 Gbits/sec
[ 5] 9.00-10.00 sec 533 MBytes 4.47 Gbits/sec
[ 5] 10.00-10.08 sec 34.4 MBytes 3.49 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[  5]   0.00-10.08  sec  5.17 GBytes  4.40 Gbits/sec


Linux guests using VirtIO network interface are also able to hit 9.41Gbits/s. Linux guests with VirtIO on both Proxmox servers also can achieve 9.41Gbit/s to each other.

Is this just poor FreeBSD drivers? Is there a better 10G NIC for use with FreeBSD? I these are X520 based, would the X710 NICs be any better?
 
J

jenksdrummer

Patron
Joined
Jun 7, 2011
Messages
250
PCI slot operating at half-bandwidth? IE, it might look like an x8, but it might actually be an x4?
 
D

dirtyfreebooter

Explorer
Joined
Oct 3, 2020
Messages
72
nah man, like i said its a two port NIC, one port is bound to hypervisor, the other port is PCIe passthrough to TrueNAS guest, and the hypervisor is the first result, 9.41Gbps, its only the passthrough port on FreeBSD
 
D

dirtyfreebooter

Explorer
Joined
Oct 3, 2020
Messages
72
also all the PCI ports on this Dell are 16x or 8x.

Specs:
2 x Gen3 slots (1 x8; 1 x16)
2 x Gen4 slots (2 x16)
 
D

dirtyfreebooter

Explorer
Joined
Oct 3, 2020
Messages
72
Lol, i went through the effort of backing up, installing TrueNAS on new drives, restoring, so TrueNAS is on baremetal and I got the same performance from the Intel X520-DA2 NICs, booted into a Linux Live CD and same NICs immediately get 10Gb/s .. I guess its just FreeBSD, kernel and IO stack is really poor compared to Linux but the feature set of FreeNAS/TrueNAS is too nice :)
 
K

Kcaj

Contributor
Joined
Jan 2, 2020
Messages
100
What happens if you Disable Hardware Offloading on the interface in the TrueNAS GUI?
 
D

dirtyfreebooter

Explorer
Joined
Oct 3, 2020
Messages
72
Kcaj said:
What happens if you Disable Hardware Offloading on the interface in the TrueNAS GUI?
Click to expand...

numbers go down slightly...

Code:
[  5] local 192.168.42.1 port 27199 connected to 192.168.42.2 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   347 MBytes  2.91 Gbits/sec    0    882 KBytes
[  5]   1.00-2.00   sec   345 MBytes  2.89 Gbits/sec    0    896 KBytes
[  5]   2.00-3.00   sec   345 MBytes  2.90 Gbits/sec    0    896 KBytes
[  5]   3.00-4.00   sec   344 MBytes  2.88 Gbits/sec    0    899 KBytes
[  5]   4.00-5.00   sec   330 MBytes  2.77 Gbits/sec    0    961 KBytes
[  5]   5.00-6.00   sec   302 MBytes  2.54 Gbits/sec    0    995 KBytes
[  5]   6.00-7.00   sec   318 MBytes  2.67 Gbits/sec    0   1.02 MBytes
[  5]   7.00-8.00   sec   324 MBytes  2.72 Gbits/sec    0   1.05 MBytes
[  5]   8.00-9.00   sec   322 MBytes  2.70 Gbits/sec    0   1.05 MBytes
[  5]   9.00-10.00  sec   322 MBytes  2.70 Gbits/sec    0   1.05 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  3.22 GBytes  2.77 Gbits/sec    0             sender
[  5]   0.00-10.06  sec  3.22 GBytes  2.75 Gbits/sec                  receiver
 
K

Kcaj

Contributor
Joined
Jan 2, 2020
Messages
100
Can you try in both directions?
Also whats the CPU doing while iPerf is running?
 
D

dirtyfreebooter

Explorer
Joined
Oct 3, 2020
Messages
72
The other direction

Code:
[  5] local 192.168.42.2 port 10793 connected to 192.168.42.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   297 MBytes  2.49 Gbits/sec    0   1.19 MBytes
[  5]   1.00-2.00   sec   300 MBytes  2.52 Gbits/sec    0   1.19 MBytes
[  5]   2.00-3.00   sec   311 MBytes  2.61 Gbits/sec    0   1.22 MBytes
[  5]   3.00-4.00   sec   302 MBytes  2.53 Gbits/sec    0   1.22 MBytes
[  5]   4.00-5.00   sec   299 MBytes  2.50 Gbits/sec    0   1.22 MBytes
[  5]   5.00-6.00   sec   301 MBytes  2.52 Gbits/sec    0   1.22 MBytes
[  5]   6.00-7.00   sec   299 MBytes  2.51 Gbits/sec    0   1.22 MBytes
[  5]   7.00-8.00   sec   299 MBytes  2.51 Gbits/sec    0   1.22 MBytes
[  5]   8.00-9.00   sec   299 MBytes  2.51 Gbits/sec    0   1.22 MBytes
[  5]   9.00-10.00  sec   299 MBytes  2.51 Gbits/sec    0   1.22 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.94 GBytes  2.52 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  2.94 GBytes  2.52 Gbits/sec                  receiver


I am going to try a different Intel NIC with X710 in a few days. If I boot the same machine with USB drive with Linux, instantly get 9.41Gbits/s

Both machines are idle for the test, TrueNAS box is AMD Rome 7402p with 256GB of memory, and Linux machine is a Proxmox 6.2 (debian 10) running kernel 5.4.x with Xeon 2278G CPU and 64GB of memory
 
F

FosCo

Dabbler
Joined
Sep 20, 2020
Messages
23
This might but does not need to be related:

Best feedback channel?

Hi, as I'm still fiddling with some hardware optimization I could not resist anymore to test the SCALE project instead of the stable CORE system. Partly because of the hassle bringing up Resilio Sync without a supported plugin (I *love* licensing issues) and want to do it with docker instead...
www.ixsystems.com www.ixsystems.com

10GbE performance issue in one direction

After having installed numerous kernels from https://kernel.ubuntu.com/~kernel-ppa/mainline/ I have now concluded that the regression on the Linux side happened between kernel versions 4.19.6 and 4.19.7. With that said, I will stop writing here and move to a Linux-centric forum / bug-reporter...
www.ixsystems.com www.ixsystems.com

Had the same issues while testing TrueNas Scale with 5.x Kernel against a Windows machine and the other thread describes a problem in Linux Mint with a similar behaviour.
 
D

dirtyfreebooter

Explorer
Joined
Oct 3, 2020
Messages
72
I switched out the X520-DA2 card for a X710-DA2 (some off brand Vogzone from Amazon) and...

Code:
Connecting to host 192.168.42.2, port 5201
[  5] local 192.168.42.10 port 46154 connected to 192.168.42.2 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.09 GBytes  9.39 Gbits/sec    0   1.41 MBytes
[  5]   1.00-2.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.41 MBytes
[  5]   2.00-3.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.41 MBytes
[  5]   3.00-4.00   sec  1.10 GBytes  9.41 Gbits/sec    0   1.41 MBytes
[  5]   4.00-5.00   sec  1.09 GBytes  9.41 Gbits/sec    0   1.41 MBytes
[  5]   5.00-6.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.41 MBytes
[  5]   6.00-7.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.41 MBytes
[  5]   7.00-8.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.41 MBytes
[  5]   8.00-9.00   sec  1.09 GBytes  9.41 Gbits/sec    0   1.41 MBytes
[  5]   9.00-10.00  sec  1.10 GBytes  9.42 Gbits/sec    0   1.41 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  11.0 GBytes  9.41 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  11.0 GBytes  9.41 Gbits/sec                  receiver


Was similar results in other direction. For whatever reason under FreeBSD/TrueNAS X520-DA2 performed like crap. Note that booting that same setup with Linux USB stick, X520-DA2 was able to get 9.41Gbit/s

oh well, problem solved by hardware. good enough for now.
 
Brezlord

Brezlord

Contributor
Joined
Jan 7, 2017
Messages
189
I had the same problems with Intel X520-DA2 cards. Changed to Cehlsio T520-CR and much better performance. I battled the Intel card for years and could never get it to preform correctly.
Post with my issues
How to replace NIC in lagg
 
D

dirtyfreebooter

Explorer
Joined
Oct 3, 2020
Messages
72
Ok, so I experimented some more with a Mellanox ConnectX-3 (single port PCIe x4). Now mind you all of this adapters, X520-DA2, X710-DA2, ConnectX-3 when booted into Linux (5.x kernel) got 9.41Gbit/s out of the box, no configuration necessary.

After more testing, I was not able to replicate the 9.41Gbit/s with the X710-DA2 on TrueNAS Core. After a reboot things went bad again. I started reading on 10GbE and FreeBSD and came across this: https://calomel.org/freebsd_network_tuning.html

I settled on these tuneables:
Code:
# loader
cc_htcp_load="YES"
kern.random.fortuna.minpoolsize=256
net.inet.tcp.hostcache.cachelimit=0
net.inet.tcp.soreceive_stream=1
net.isr.maxthreads=-1
net.isr.bindthreads=1
net.pf.source_nodes_hashsize=1048576

# sysctl
net.inet.tcp.abc_l_var=44
net.inet.tcp.cc.abe=1
net.inet.tcp.cc.algorithm=htcp
net.inet.tcp.cc.htcp.adaptive_backoff=1
net.inet.tcp.cc.htcp.rtt_scaling=1
net.inet.tcp.initcwnd_segments=44
net.inet.tcp.minmss=536
net.inet.tcp.mssdflt=1460
net.inet.tcp.rfc6675_pipe=1
net.inet.tcp.syncache.rexmtlimit=0
net.inet.tcp.syncookies=0


Now in repeated tests with all 3 cards (X520-DA2, X710-DA2, ConnectX-3) on this Dell R7515 with TrueNAS Core 12.0-RC1, 10GbE is achieved in both directions, as a client and as a server.

Code:
# iperf3 --client 192.168.42.3 --no-delay --zerocopy
Connecting to host 192.168.42.3, port 5201
[  5] local 192.168.42.2 port 61057 connected to 192.168.42.3 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.02 GBytes  8.78 Gbits/sec    0   1.79 MBytes
[  5]   1.00-2.00   sec  1.10 GBytes  9.41 Gbits/sec    0   1.79 MBytes
[  5]   2.00-3.00   sec  1.10 GBytes  9.41 Gbits/sec    0   1.79 MBytes
[  5]   3.00-4.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.79 MBytes
[  5]   4.00-5.00   sec  1.10 GBytes  9.41 Gbits/sec    0   1.79 MBytes
[  5]   5.00-6.00   sec  1.10 GBytes  9.41 Gbits/sec    0   1.79 MBytes
[  5]   6.00-7.00   sec  1.10 GBytes  9.41 Gbits/sec    0   1.79 MBytes
[  5]   7.00-8.00   sec  1.10 GBytes  9.41 Gbits/sec    0   1.79 MBytes
[  5]   8.00-9.00   sec  1.10 GBytes  9.41 Gbits/sec    0   1.79 MBytes
[  5]   9.00-10.00  sec  1.10 GBytes  9.41 Gbits/sec    0   1.79 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  10.9 GBytes  9.35 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  10.9 GBytes  9.35 Gbits/sec                  receiver

iperf Done.
# iperf3 --client 192.168.42.10 --no-delay --zerocopy
Connecting to host 192.168.42.10, port 5201
[  5] local 192.168.42.2 port 63821 connected to 192.168.42.10 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.07 GBytes  9.22 Gbits/sec   55    281 KBytes
[  5]   1.00-2.00   sec  1.08 GBytes  9.28 Gbits/sec   48    276 KBytes
[  5]   2.00-3.00   sec  1.07 GBytes  9.23 Gbits/sec   44    310 KBytes
[  5]   3.00-4.00   sec  1.08 GBytes  9.30 Gbits/sec   40    276 KBytes
[  5]   4.00-5.00   sec  1.08 GBytes  9.28 Gbits/sec   36    504 KBytes
[  5]   5.00-6.00   sec  1.08 GBytes  9.27 Gbits/sec   53    265 KBytes
[  5]   6.00-7.00   sec  1.07 GBytes  9.21 Gbits/sec   61    318 KBytes
[  5]   7.00-8.00   sec  1.08 GBytes  9.30 Gbits/sec   41    308 KBytes
[  5]   8.00-9.00   sec  1.08 GBytes  9.28 Gbits/sec   44    305 KBytes
[  5]   9.00-10.00  sec  1.08 GBytes  9.31 Gbits/sec   36    348 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  10.8 GBytes  9.27 Gbits/sec  458             sender
[  5]   0.00-10.08  sec  10.8 GBytes  9.19 Gbits/sec                  receiver

iperf Done.


192.168.42.3 is another TrueNAS Core machine running on PC hardware with the Mellanox ConnectX-3. 192.168.42.10 is a Supermicro 1U Proxmox (Debian 10 Linux with Xeon 2278G) machine with Intel X520-DA2. Interesting enough, very repeatable, the TrueNAS to TrueNAS iperf3 results are slightly better than the TrueNAS to Proxmox.

All machines involved are idle and grossly overpowered CPU and memory-wise.
 
D

dirtyfreebooter

Explorer
Joined
Oct 3, 2020
Messages
72
lol, FreeBSD is something else... so if you have jails and use vnet, once the bridge is created it will also tank your 10GbE performance.

www.youtube.com

Freenas Plex Jail Issue and 10gb Speed Issue Fixed!

These issues were related! Also it fixed a problem I didn't even know I had, and I pretty much created a new best practice that not even the Freenas folks kn...
www.youtube.com www.youtube.com

I put my jails on one of the Broadcom 1GbE NICs that are on the motherboard and 10GbE speed was back! The Broadcom NIC doesn't even need a cable, as it will be routed internally and finally out my default route on the 10G NIC. Geez. This is real janky setup. I look forward to TrueNAS SCALE and having a modern kernel and networking features.
 
Brezlord

Brezlord

Contributor
Joined
Jan 7, 2017
Messages
189
I have used info from that page in the past for FreeBSD 10 and 11, not sure how relevent they are for FreeBSD 12. The settings you have below did not have any positive effect on my current system, they seem to of made performance slightly worse. The defult settings for the Chelso T520-CR seem to work fine for me.

dirtyfreebooter said:
After more testing, I was not able to replicate the 9.41Gbit/s with the X710-DA2 on TrueNAS Core. After a reboot things went bad again. I started reading on 10GbE and FreeBSD and came across this: https://calomel.org/freebsd_network_tuning.html

I settled on these tuneables:
Code:
# loader
cc_htcp_load="YES"
kern.random.fortuna.minpoolsize=256
net.inet.tcp.hostcache.cachelimit=0
net.inet.tcp.soreceive_stream=1
net.isr.maxthreads=-1
net.isr.bindthreads=1
net.pf.source_nodes_hashsize=1048576

# sysctl
net.inet.tcp.abc_l_var=44
net.inet.tcp.cc.abe=1
net.inet.tcp.cc.algorithm=htcp
net.inet.tcp.cc.htcp.adaptive_backoff=1
net.inet.tcp.cc.htcp.rtt_scaling=1
net.inet.tcp.initcwnd_segments=44
net.inet.tcp.minmss=536
net.inet.tcp.mssdflt=1460
net.inet.tcp.rfc6675_pipe=1
net.inet.tcp.syncache.rexmtlimit=0
net.inet.tcp.syncookies=0
Click to expand...
 
D

Dan Tudora

Patron
Joined
Jul 6, 2017
Messages
276
hello
TrueNAS Scale is in Alpha stage
and is an Debian linux
what do you think about that
make a try with your NIC
never know
success
 
D

dirtyfreebooter

Explorer
Joined
Oct 3, 2020
Messages
72
yea, i tried 20.10 out, its still too alpha but looks super promising. FreeBSD is something. And I can confirm after much more testing, all of my issues are related to jails and vnet. The problem is, certain applications need vnet in jails, as they don't work in strange ways without vnet (ie full networking stack).

I just came over to TrueNAS Core from Ubuntu 18.04 with ZFS and I love TrueNAS ecosystem, but FreeBSD is kinda lame. If you didn't have 10GbE also, you would never see any issues, as 1 Gbps speeds, vnet isn't slow enough to be a problem.

And trying my workaround, putting the bridge0,vnet0 on a separate NIC port, kinda works, until it doesn't... Then randomly all my outgoing iperf3 results show EXACTLY 1 Gbps speeds.

Code:
# iperf3 --client 192.168.42.10 --no-delay --zerocopy
Connecting to host 192.168.42.10, port 5201
[  5] local 192.168.42.2 port 12838 connected to 192.168.42.10 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  86.2 MBytes   724 Mbits/sec  333    211 KBytes
[  5]   1.00-2.00   sec   111 MBytes   933 Mbits/sec  344    272 KBytes
[  5]   2.00-3.00   sec   112 MBytes   944 Mbits/sec  302    311 KBytes
[  5]   3.00-4.00   sec   111 MBytes   933 Mbits/sec  333    353 KBytes
[  5]   4.00-5.00   sec   111 MBytes   933 Mbits/sec  440    220 KBytes
[  5]   5.00-6.00   sec   111 MBytes   933 Mbits/sec  324    274 KBytes
[  5]   6.00-7.00   sec   111 MBytes   933 Mbits/sec  314    314 KBytes
[  5]   7.00-8.00   sec   112 MBytes   944 Mbits/sec  316    355 KBytes
[  5]   8.00-9.00   sec   111 MBytes   933 Mbits/sec  407    227 KBytes
[  5]   9.00-10.00  sec   111 MBytes   933 Mbits/sec  320    277 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.06 GBytes   914 Mbits/sec  3433             sender
[  5]   0.00-10.08  sec  1.06 GBytes   905 Mbits/sec                  receiver


I do it from the opposite direction Linux -> TrueNAS and 10GBit/s!

Code:
# iperf3 --client 192.168.42.2 --no-delay --zerocopy
Connecting to host 192.168.42.2, port 5201
[  5] local 192.168.42.10 port 45570 connected to 192.168.42.2 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.10 GBytes  9.44 Gbits/sec    0   1.56 MBytes
[  5]   1.00-2.00   sec  1.09 GBytes  9.41 Gbits/sec    0   1.56 MBytes
[  5]   2.00-3.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.56 MBytes
[  5]   3.00-4.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.56 MBytes
[  5]   4.00-5.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.56 MBytes
[  5]   5.00-6.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.56 MBytes
[  5]   6.00-7.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.56 MBytes
[  5]   7.00-8.00   sec  1.09 GBytes  9.41 Gbits/sec    0   1.56 MBytes
[  5]   8.00-9.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.56 MBytes
[  5]   9.00-10.00  sec  1.10 GBytes  9.42 Gbits/sec    0   1.56 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  11.0 GBytes  9.42 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  11.0 GBytes  9.41 Gbits/sec                  receiver


If I stop/start all my jails...

Code:
# iperf3 --client 192.168.42.10 --no-delay --zerocopy
Connecting to host 192.168.42.10, port 5201
[  5] local 192.168.42.2 port 11981 connected to 192.168.42.10 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.09 GBytes  9.41 Gbits/sec   16    566 KBytes
[  5]   1.00-2.00   sec  1.09 GBytes  9.41 Gbits/sec    8    846 KBytes
[  5]   2.00-3.01   sec   920 MBytes  7.65 Gbits/sec    5    563 KBytes
[  5]   3.01-4.00   sec   864 MBytes  7.31 Gbits/sec    7    720 KBytes
[  5]   4.00-5.00   sec  1.09 GBytes  9.41 Gbits/sec    4    783 KBytes
[  5]   5.00-6.00   sec  1.09 GBytes  9.40 Gbits/sec    6    480 KBytes
[  5]   6.00-7.00   sec  1.09 GBytes  9.32 Gbits/sec   11    609 KBytes
[  5]   7.00-8.00   sec  1.09 GBytes  9.40 Gbits/sec   24    532 KBytes
[  5]   8.00-9.00   sec  1.09 GBytes  9.41 Gbits/sec    7    548 KBytes
[  5]   9.00-10.00  sec  1.09 GBytes  9.41 Gbits/sec    6    354 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  10.5 GBytes  9.01 Gbits/sec   94             sender
[  5]   0.00-10.08  sec  10.5 GBytes  8.93 Gbits/sec                  receiver


Speeds are back. Argh. This is disappointing. Probably could survive until TrueNAS SCALE is in better shape. Looked at 20.10, still too many missing things, but its looking great IMHO. Can't wait!
 
D

dirtyfreebooter

Explorer
Joined
Oct 3, 2020
Messages
72
ok new behavior. after moving all the jails to another NIC, if i run iperf long enough, eventually it magically cap itself at 1 Gbps.. with very high retransmit count...

The switch is a UniFi 16XG, essentially idle.

Code:
# iperf3 --client 192.168.42.10 --no-delay --zerocopy --time 1800 --interval 20
Connecting to host 192.168.42.10, port 5201
[  5] local 192.168.42.2 port 50863 connected to 192.168.42.10 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-20.00  sec  21.9 GBytes  9.40 Gbits/sec  169    683 KBytes
[  5]  20.00-40.00  sec  21.9 GBytes  9.40 Gbits/sec  135    935 KBytes
[  5]  40.00-60.00  sec  21.9 GBytes  9.40 Gbits/sec  168    396 KBytes
... # remove repeated rows
[  5] 620.00-640.00 sec  21.9 GBytes  9.40 Gbits/sec  165    664 KBytes
[  5] 640.00-660.00 sec  21.9 GBytes  9.40 Gbits/sec  157    667 KBytes
[  5] 660.00-680.00 sec  21.9 GBytes  9.40 Gbits/sec  146    726 KBytes
[  5] 680.00-700.00 sec  21.9 GBytes  9.40 Gbits/sec  160    621 KBytes
[  5] 700.00-720.00 sec  21.9 GBytes  9.40 Gbits/sec  155    577 KBytes
[  5] 720.00-740.00 sec  21.9 GBytes  9.40 Gbits/sec  129    982 KBytes
[  5] 740.00-760.00 sec  13.9 GBytes  5.97 Gbits/sec  3262    342 KBytes
[  5] 760.00-780.00 sec  2.18 GBytes   936 Mbits/sec  6596    281 KBytes
[  5] 780.00-800.00 sec  2.18 GBytes   935 Mbits/sec  6710    259 KBytes
[  5] 800.00-820.00 sec  2.18 GBytes   935 Mbits/sec  6531    245 KBytes
[  5] 820.00-840.00 sec  2.18 GBytes   935 Mbits/sec  6343    356 KBytes
[  5] 840.00-860.00 sec  2.18 GBytes   935 Mbits/sec  6487    309 KBytes
[  5] 860.00-880.00 sec  2.18 GBytes   935 Mbits/sec  6400    275 KBytes
[  5] 880.00-900.00 sec  2.18 GBytes   935 Mbits/sec  6646    315 KBytes
[  5] 900.00-920.00 sec  2.18 GBytes   935 Mbits/sec  6640    346 KBytes
... # remove repeated rows
[  5] 1720.00-1740.00 sec  2.18 GBytes   935 Mbits/sec  5921    325 KBytes
[  5] 1740.00-1760.00 sec  2.18 GBytes   935 Mbits/sec  6503    264 KBytes
[  5] 1760.00-1780.00 sec  2.18 GBytes   935 Mbits/sec  6120    269 KBytes
[  5] 1780.00-1800.00 sec  2.18 GBytes   935 Mbits/sec  6009    265 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-1800.00 sec   937 GBytes  4.47 Gbits/sec  337390             sender
[  5]   0.00-1800.09 sec   937 GBytes  4.47 Gbits/sec                  receiver


Note this is one-way, if i --reverse, speeds are fine

Code:
# iperf3 --client 192.168.42.10 --no-delay --zerocopy --time 120 --interval 10 --reverse
Connecting to host 192.168.42.10, port 5201
Reverse mode, remote host 192.168.42.10 is sending
[  5] local 192.168.42.2 port 54176 connected to 192.168.42.10 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  11.0 GBytes  9.42 Gbits/sec
[  5]  10.00-20.00  sec  11.0 GBytes  9.41 Gbits/sec
[  5]  20.00-30.00  sec  11.0 GBytes  9.41 Gbits/sec
[  5]  30.00-40.00  sec  11.0 GBytes  9.41 Gbits/sec
[  5]  40.00-50.00  sec  11.0 GBytes  9.41 Gbits/sec
[  5]  50.00-60.00  sec  11.0 GBytes  9.41 Gbits/sec
[  5]  60.00-70.00  sec  11.0 GBytes  9.41 Gbits/sec
[  5]  70.00-80.00  sec  11.0 GBytes  9.41 Gbits/sec
[  5]  80.00-90.00  sec  11.0 GBytes  9.41 Gbits/sec
[  5]  90.00-100.00 sec  11.0 GBytes  9.41 Gbits/sec
[  5] 100.00-110.00 sec  11.0 GBytes  9.41 Gbits/sec
[  5] 110.00-120.00 sec  11.0 GBytes  9.41 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-120.08 sec   132 GBytes  9.41 Gbits/sec    0             sender
[  5]   0.00-120.00 sec   132 GBytes  9.41 Gbits/sec                  receiver



Now I re-ran the test and then stopped all jails (using the other NIC) while running the test...

Code:
# iperf3 --client 192.168.42.10 --no-delay --zerocopy --time 120 --interval 10
Connecting to host 192.168.42.10, port 5201
[  5] local 192.168.42.2 port 25786 connected to 192.168.42.10 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-10.00  sec  1.06 GBytes   914 Mbits/sec  3413    288 KBytes
[  5]  10.00-20.00  sec  1.09 GBytes   934 Mbits/sec  3452    245 KBytes
[  5]  20.00-30.00  sec  1.09 GBytes   935 Mbits/sec  3357    364 KBytes
[  5]  30.00-40.00  sec  1.09 GBytes   934 Mbits/sec  3594    349 KBytes
[  5]  40.00-50.00  sec  1.09 GBytes   934 Mbits/sec  3532    316 KBytes
[  5]  50.00-60.00  sec  1.09 GBytes   935 Mbits/sec  3608    261 KBytes
[  5]  60.00-70.00  sec  7.47 GBytes  6.42 Gbits/sec  1761    388 KBytes
[  5]  70.00-80.00  sec  10.7 GBytes  9.18 Gbits/sec  640    345 KBytes
[  5]  80.00-90.00  sec  10.7 GBytes  9.21 Gbits/sec  627    212 KBytes
[  5]  90.00-100.00 sec  10.7 GBytes  9.21 Gbits/sec  578    313 KBytes
[  5] 100.00-110.00 sec  10.7 GBytes  9.18 Gbits/sec  603    302 KBytes
[  5] 110.00-120.00 sec  10.7 GBytes  9.21 Gbits/sec  582    470 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-120.00 sec  67.5 GBytes  4.83 Gbits/sec  25747             sender
[  5]   0.00-120.08 sec  67.5 GBytes  4.83 Gbits/sec                  receiver


Right in the middle of the test the speeds come back!

Even when the jails are stopped, there is still high retr counts, when i boot this same machine into Linux and run the same tests or reverse the direction (--reverse) the retr counts are mostly zero, or just a few occasionally.
 
K

kspare

Guru
Joined
Feb 19, 2015
Messages
508
Im playing around with this as well....and having similar issues with 40gb T580-LP-CR network cards to a cisco nexus switch. one server can barely do 4-6gb on a 40gb adapter one way, but the other way does closer to 25-30gb.

I'm going to reinstall 11.3 and test between the two versions to see if there is any difference.
 
Elliot Dierksen

Elliot Dierksen

Guru
Joined
Dec 29, 2014
Messages
1,135
I had a very similar issue with a T580 card. I hate to be the bearer of bad news, but it was the card. I replaced the adapter, and then I was getting 39G+ with iperf. Maybe there are knockoff or bad Chelsio cards being sold on eBay. That was the source of my bad one. It would be easier if they failed completely. :-(
 
You must log in or register to reply here.

Similar threads

J
L2ARC working but not for file share.
Replies
4
Views
1K
James Gardiner
J
B
TrueNas 12 - Slow - Onto Permissions :/
Replies
5
Views
877
Bobsnet
B
M
Network 10Gbps
Replies
0
Views
1K
matt998
M
K
Truenas Core. Bhyve. Jails. Скорость виртуальной сети.
Replies
1
Views
642
kixx
K
bferrell
10G Speeds wildly up and down?
Replies
8
Views
3K
bferrell
bferrell
Top