10gbe performance issue with LRO after bootup

Silverstar24

Dabbler
Joined
Jan 22, 2020
Messages
11
After a few years i invest some money to upgrade my freenas/homelab environment. An Cisco UCS Server
wit ESX and alot of flash storage(~40TB). I decide to install freenas in a VM and passingtrought the
SAS HBA Controller into the VM. The freenas vm itself is stored on al local SSD raid esx datastore.

Now i have a strange problem with the performance of the 10gbe vmxnet3 interface on the freenas vm(Testet with 11.2 u5 and u7 same behavior on both versions). Windwos vm's on the same esx host and network config running fine without any performance issue.

When i try to write some data via smb to the NAS i get arround 350-400MB/s on one of my data pool (config see below).
But when i read datas i can reach the full bandwiche of 10gbe (~1.1GB/s). The pool delivers around 3GB/s with dd on the freenas directly.

So i was confused an tried with iperf3 and i get the results that you can see below. Than i ask google for some help and I had readed alot about 10gbe tuning in freebsd. I testet many tunables but nonthing was helping and the iperf3 results was allways around the same speed(see below). Also i disable VMWare DirectI/O.

Today i found a workaround. I switched on and off the LRO on the vmxnet3 adapter of the freenas vm. Not that i had testet to switch off TSO / LRO but i never tried to switch back on.

BAMM... now iperf3 can send around 30Gb/s in both way between Freenas and the Windows VM. Also now i can move data over smb around 1.4GB/s. After a reboot of the freenas vm the old speeds are back. Now i tried again and i switched only LRO off and on again. I get again 30Gb/s in both ways ...

Also my Macbook Pro with 10GBASE-T TB3 send and receive now 9.5Gb/s iperf3 speed with the freenas vm.

Any idea why this behavior comes up? Wiche logs can i provide to see more around this behavior? Note that i can't find any errors or warning in the logs, but maybe i look into the wrong logs.

Code:
System:

ESX HOST:
    
    Cisco UCS C240 M4SX V2 Rack Server
    Dual Socket System with 14 Core Intel Xenon E5-2690 v4 @ 2.6GHz 35MB L3 Cache
    8x 32GB DDR4 PC 2400 Registered ECC
    Fusion-MPT 12GSAS SAS3108 HBA Controller (In passtrought mode)
    Dual Port I350 Gigabit Network adapter (Management Interface only)
    Dual Port UCS VIC 1227 10gbe Network adapter (VM communication interface)
    Dual PSU 1200W
    
    ESX 6.7 Update 3
    
    -> Last Cisco Firmware level installed

FREENAS VM:

    8x vCPU E5-2690v4
    64GB RAM
    vmx0 10Gb/s vmxnet3 DirectI/O on for Clients
    vmx1 10Gb/s vmxnet3 DirectI/O on for ESX NFS Store
    Fusion-MPT 12GSAS SAS3108 HBA Controller (In passtrought mode)
    2x 800GB 12Gb SAS SSD -> zRAID1 Pool for VM's
    10x 3.8TB 6Gb SATA SSD -> Two zRAID1 Pool each with 5 disk

Tunables:
    
    net.isr.maxthreads="-1" -> loader
    net.isr.bindthreads="0" -> loader
    kern.ipc.maxsockbuf=157286400 -> sysctl
    net.inet.tcp.recvbuf_inc=65536 -> sysctl
    net.inet.tcp.recvbuf_max=4194304 -> sysctl
    net.inet.tcp.recvspace=65536 -> sysctl
    net.inet.tcp.sendbuf_inc=65536 -> sysctl
    net.inet.tcp.sendbuf_max=4194304 -> sysctl
    net.inet.tcp.sendspace=65536 -> sysctl

WINDOWS 2016 SERVER:

    4x vCPU E5-2690v4
    16GB RAM
    60GB DISK
    10Gb/s vmxnet3 DirectI/O on

    -> out of the Box (no changes or custom software like AV or somthing)

MACBOCK PRO
    
    MacOS Catalina 10.15.2
    I7 2.9GHz
    16GB RAM LPDDR3
    1TB NVMe SSD   

    -> SONNET Solo 10Gb/s Thunderbolt 3 adapter with last firmware

-------------------------------------------------
        Default with tunings
-------------------------------------------------
client -> freenas
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec   980 MBytes   822 Mbits/sec                  sender
[  4]   0.00-10.00  sec   980 MBytes   822 Mbits/sec                  receiver
[  6]   0.00-10.00  sec   982 MBytes   824 Mbits/sec                  sender
[  6]   0.00-10.00  sec   982 MBytes   824 Mbits/sec                  receiver
[  8]   0.00-10.00  sec   971 MBytes   815 Mbits/sec                  sender
[  8]   0.00-10.00  sec   971 MBytes   815 Mbits/sec                  receiver
[ 10]   0.00-10.00  sec   991 MBytes   831 Mbits/sec                  sender
[ 10]   0.00-10.00  sec   991 MBytes   831 Mbits/sec                  receiver
[ 12]   0.00-10.00  sec   982 MBytes   824 Mbits/sec                  sender
[ 12]   0.00-10.00  sec   982 MBytes   824 Mbits/sec                  receiver
[ 14]   0.00-10.00  sec   952 MBytes   799 Mbits/sec                  sender
[ 14]   0.00-10.00  sec   952 MBytes   799 Mbits/sec                  receiver
[ 16]   0.00-10.00  sec   984 MBytes   826 Mbits/sec                  sender
[ 16]   0.00-10.00  sec   984 MBytes   826 Mbits/sec                  receiver
[ 18]   0.00-10.00  sec   978 MBytes   820 Mbits/sec                  sender
[ 18]   0.00-10.00  sec   978 MBytes   820 Mbits/sec                  receiver
[SUM]   0.00-10.00  sec  7.64 GBytes  6.56 Gbits/sec                  sender
[SUM]   0.00-10.00  sec  7.64 GBytes  6.56 Gbits/sec                  receiver

freenas -> client
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  2.77 GBytes  2.38 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  2.77 GBytes  2.38 Gbits/sec                  receiver
[  6]   0.00-10.00  sec  2.77 GBytes  2.38 Gbits/sec    0             sender
[  6]   0.00-10.00  sec  2.77 GBytes  2.38 Gbits/sec                  receiver
[  8]   0.00-10.00  sec  2.77 GBytes  2.38 Gbits/sec    0             sender
[  8]   0.00-10.00  sec  2.77 GBytes  2.38 Gbits/sec                  receiver
[ 10]   0.00-10.00  sec  2.77 GBytes  2.38 Gbits/sec    0             sender
[ 10]   0.00-10.00  sec  2.77 GBytes  2.38 Gbits/sec                  receiver
[ 12]   0.00-10.00  sec  2.77 GBytes  2.38 Gbits/sec    0             sender
[ 12]   0.00-10.00  sec  2.77 GBytes  2.38 Gbits/sec                  receiver
[ 14]   0.00-10.00  sec  2.77 GBytes  2.38 Gbits/sec    0             sender
[ 14]   0.00-10.00  sec  2.77 GBytes  2.38 Gbits/sec                  receiver
[ 16]   0.00-10.00  sec  2.77 GBytes  2.38 Gbits/sec    0             sender
[ 16]   0.00-10.00  sec  2.77 GBytes  2.38 Gbits/sec                  receiver
[ 18]   0.00-10.00  sec  2.77 GBytes  2.38 Gbits/sec    0             sender
[ 18]   0.00-10.00  sec  2.77 GBytes  2.38 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  22.2 GBytes  19.0 Gbits/sec    0             sender
[SUM]   0.00-10.00  sec  22.2 GBytes  19.0 Gbits/sec                  receiver


-------------------------------------------------
        After lro reset

ifconfig vmx0 -lro & ifconfig vmx0 lro
-------------------------------------------------


client -> freenas
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  5.39 GBytes  4.63 Gbits/sec                  sender
[  4]   0.00-10.00  sec  5.39 GBytes  4.63 Gbits/sec                  receiver
[  6]   0.00-10.00  sec  4.84 GBytes  4.16 Gbits/sec                  sender
[  6]   0.00-10.00  sec  4.84 GBytes  4.16 Gbits/sec                  receiver
[  8]   0.00-10.00  sec  5.05 GBytes  4.34 Gbits/sec                  sender
[  8]   0.00-10.00  sec  5.05 GBytes  4.34 Gbits/sec                  receiver
[ 10]   0.00-10.00  sec  4.55 GBytes  3.91 Gbits/sec                  sender
[ 10]   0.00-10.00  sec  4.55 GBytes  3.91 Gbits/sec                  receiver
[ 12]   0.00-10.00  sec  4.91 GBytes  4.22 Gbits/sec                  sender
[ 12]   0.00-10.00  sec  4.91 GBytes  4.22 Gbits/sec                  receiver
[ 14]   0.00-10.00  sec  5.36 GBytes  4.61 Gbits/sec                  sender
[ 14]   0.00-10.00  sec  5.36 GBytes  4.61 Gbits/sec                  receiver
[ 16]   0.00-10.00  sec  4.79 GBytes  4.11 Gbits/sec                  sender
[ 16]   0.00-10.00  sec  4.79 GBytes  4.11 Gbits/sec                  receiver
[ 18]   0.00-10.00  sec  4.40 GBytes  3.78 Gbits/sec                  sender
[ 18]   0.00-10.00  sec  4.40 GBytes  3.78 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  39.3 GBytes  33.7 Gbits/sec                  sender
[SUM]   0.00-10.00  sec  39.3 GBytes  33.7 Gbits/sec                  receiver

freenas -> client
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  4.14 GBytes  3.55 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  4.14 GBytes  3.55 Gbits/sec                  receiver
[  6]   0.00-10.00  sec  4.03 GBytes  3.46 Gbits/sec    0             sender
[  6]   0.00-10.00  sec  4.03 GBytes  3.46 Gbits/sec                  receiver
[  8]   0.00-10.00  sec  4.08 GBytes  3.51 Gbits/sec    0             sender
[  8]   0.00-10.00  sec  4.08 GBytes  3.51 Gbits/sec                  receiver
[ 10]   0.00-10.00  sec  4.06 GBytes  3.49 Gbits/sec    0             sender
[ 10]   0.00-10.00  sec  4.06 GBytes  3.49 Gbits/sec                  receiver
[ 12]   0.00-10.00  sec  4.01 GBytes  3.45 Gbits/sec    0             sender
[ 12]   0.00-10.00  sec  4.01 GBytes  3.45 Gbits/sec                  receiver
[ 14]   0.00-10.00  sec  4.03 GBytes  3.46 Gbits/sec    0             sender
[ 14]   0.00-10.00  sec  4.03 GBytes  3.46 Gbits/sec                  receiver
[ 16]   0.00-10.00  sec  4.00 GBytes  3.43 Gbits/sec    0             sender
[ 16]   0.00-10.00  sec  4.00 GBytes  3.43 Gbits/sec                  receiver
[ 18]   0.00-10.00  sec  3.97 GBytes  3.41 Gbits/sec    0             sender
[ 18]   0.00-10.00  sec  3.97 GBytes  3.41 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  32.3 GBytes  27.8 Gbits/sec    0             sender
[SUM]   0.00-10.00  sec  32.3 GBytes  27.8 Gbits/sec                  receiver
 
Top