Silverstar24
Dabbler
- Joined
- Jan 22, 2020
- Messages
- 11
After a few years i invest some money to upgrade my freenas/homelab environment. An Cisco UCS Server
wit ESX and alot of flash storage(~40TB). I decide to install freenas in a VM and passingtrought the
SAS HBA Controller into the VM. The freenas vm itself is stored on al local SSD raid esx datastore.
Now i have a strange problem with the performance of the 10gbe vmxnet3 interface on the freenas vm(Testet with 11.2 u5 and u7 same behavior on both versions). Windwos vm's on the same esx host and network config running fine without any performance issue.
When i try to write some data via smb to the NAS i get arround 350-400MB/s on one of my data pool (config see below).
But when i read datas i can reach the full bandwiche of 10gbe (~1.1GB/s). The pool delivers around 3GB/s with dd on the freenas directly.
So i was confused an tried with iperf3 and i get the results that you can see below. Than i ask google for some help and I had readed alot about 10gbe tuning in freebsd. I testet many tunables but nonthing was helping and the iperf3 results was allways around the same speed(see below). Also i disable VMWare DirectI/O.
Today i found a workaround. I switched on and off the LRO on the vmxnet3 adapter of the freenas vm. Not that i had testet to switch off TSO / LRO but i never tried to switch back on.
BAMM... now iperf3 can send around 30Gb/s in both way between Freenas and the Windows VM. Also now i can move data over smb around 1.4GB/s. After a reboot of the freenas vm the old speeds are back. Now i tried again and i switched only LRO off and on again. I get again 30Gb/s in both ways ...
Also my Macbook Pro with 10GBASE-T TB3 send and receive now 9.5Gb/s iperf3 speed with the freenas vm.
Any idea why this behavior comes up? Wiche logs can i provide to see more around this behavior? Note that i can't find any errors or warning in the logs, but maybe i look into the wrong logs.
wit ESX and alot of flash storage(~40TB). I decide to install freenas in a VM and passingtrought the
SAS HBA Controller into the VM. The freenas vm itself is stored on al local SSD raid esx datastore.
Now i have a strange problem with the performance of the 10gbe vmxnet3 interface on the freenas vm(Testet with 11.2 u5 and u7 same behavior on both versions). Windwos vm's on the same esx host and network config running fine without any performance issue.
When i try to write some data via smb to the NAS i get arround 350-400MB/s on one of my data pool (config see below).
But when i read datas i can reach the full bandwiche of 10gbe (~1.1GB/s). The pool delivers around 3GB/s with dd on the freenas directly.
So i was confused an tried with iperf3 and i get the results that you can see below. Than i ask google for some help and I had readed alot about 10gbe tuning in freebsd. I testet many tunables but nonthing was helping and the iperf3 results was allways around the same speed(see below). Also i disable VMWare DirectI/O.
Today i found a workaround. I switched on and off the LRO on the vmxnet3 adapter of the freenas vm. Not that i had testet to switch off TSO / LRO but i never tried to switch back on.
BAMM... now iperf3 can send around 30Gb/s in both way between Freenas and the Windows VM. Also now i can move data over smb around 1.4GB/s. After a reboot of the freenas vm the old speeds are back. Now i tried again and i switched only LRO off and on again. I get again 30Gb/s in both ways ...
Also my Macbook Pro with 10GBASE-T TB3 send and receive now 9.5Gb/s iperf3 speed with the freenas vm.
Any idea why this behavior comes up? Wiche logs can i provide to see more around this behavior? Note that i can't find any errors or warning in the logs, but maybe i look into the wrong logs.
Code:
System: ESX HOST: Cisco UCS C240 M4SX V2 Rack Server Dual Socket System with 14 Core Intel Xenon E5-2690 v4 @ 2.6GHz 35MB L3 Cache 8x 32GB DDR4 PC 2400 Registered ECC Fusion-MPT 12GSAS SAS3108 HBA Controller (In passtrought mode) Dual Port I350 Gigabit Network adapter (Management Interface only) Dual Port UCS VIC 1227 10gbe Network adapter (VM communication interface) Dual PSU 1200W ESX 6.7 Update 3 -> Last Cisco Firmware level installed FREENAS VM: 8x vCPU E5-2690v4 64GB RAM vmx0 10Gb/s vmxnet3 DirectI/O on for Clients vmx1 10Gb/s vmxnet3 DirectI/O on for ESX NFS Store Fusion-MPT 12GSAS SAS3108 HBA Controller (In passtrought mode) 2x 800GB 12Gb SAS SSD -> zRAID1 Pool for VM's 10x 3.8TB 6Gb SATA SSD -> Two zRAID1 Pool each with 5 disk Tunables: net.isr.maxthreads="-1" -> loader net.isr.bindthreads="0" -> loader kern.ipc.maxsockbuf=157286400 -> sysctl net.inet.tcp.recvbuf_inc=65536 -> sysctl net.inet.tcp.recvbuf_max=4194304 -> sysctl net.inet.tcp.recvspace=65536 -> sysctl net.inet.tcp.sendbuf_inc=65536 -> sysctl net.inet.tcp.sendbuf_max=4194304 -> sysctl net.inet.tcp.sendspace=65536 -> sysctl WINDOWS 2016 SERVER: 4x vCPU E5-2690v4 16GB RAM 60GB DISK 10Gb/s vmxnet3 DirectI/O on -> out of the Box (no changes or custom software like AV or somthing) MACBOCK PRO MacOS Catalina 10.15.2 I7 2.9GHz 16GB RAM LPDDR3 1TB NVMe SSD -> SONNET Solo 10Gb/s Thunderbolt 3 adapter with last firmware ------------------------------------------------- Default with tunings ------------------------------------------------- client -> freenas - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 4] 0.00-10.00 sec 980 MBytes 822 Mbits/sec sender [ 4] 0.00-10.00 sec 980 MBytes 822 Mbits/sec receiver [ 6] 0.00-10.00 sec 982 MBytes 824 Mbits/sec sender [ 6] 0.00-10.00 sec 982 MBytes 824 Mbits/sec receiver [ 8] 0.00-10.00 sec 971 MBytes 815 Mbits/sec sender [ 8] 0.00-10.00 sec 971 MBytes 815 Mbits/sec receiver [ 10] 0.00-10.00 sec 991 MBytes 831 Mbits/sec sender [ 10] 0.00-10.00 sec 991 MBytes 831 Mbits/sec receiver [ 12] 0.00-10.00 sec 982 MBytes 824 Mbits/sec sender [ 12] 0.00-10.00 sec 982 MBytes 824 Mbits/sec receiver [ 14] 0.00-10.00 sec 952 MBytes 799 Mbits/sec sender [ 14] 0.00-10.00 sec 952 MBytes 799 Mbits/sec receiver [ 16] 0.00-10.00 sec 984 MBytes 826 Mbits/sec sender [ 16] 0.00-10.00 sec 984 MBytes 826 Mbits/sec receiver [ 18] 0.00-10.00 sec 978 MBytes 820 Mbits/sec sender [ 18] 0.00-10.00 sec 978 MBytes 820 Mbits/sec receiver [SUM] 0.00-10.00 sec 7.64 GBytes 6.56 Gbits/sec sender [SUM] 0.00-10.00 sec 7.64 GBytes 6.56 Gbits/sec receiver freenas -> client - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 2.77 GBytes 2.38 Gbits/sec 0 sender [ 4] 0.00-10.00 sec 2.77 GBytes 2.38 Gbits/sec receiver [ 6] 0.00-10.00 sec 2.77 GBytes 2.38 Gbits/sec 0 sender [ 6] 0.00-10.00 sec 2.77 GBytes 2.38 Gbits/sec receiver [ 8] 0.00-10.00 sec 2.77 GBytes 2.38 Gbits/sec 0 sender [ 8] 0.00-10.00 sec 2.77 GBytes 2.38 Gbits/sec receiver [ 10] 0.00-10.00 sec 2.77 GBytes 2.38 Gbits/sec 0 sender [ 10] 0.00-10.00 sec 2.77 GBytes 2.38 Gbits/sec receiver [ 12] 0.00-10.00 sec 2.77 GBytes 2.38 Gbits/sec 0 sender [ 12] 0.00-10.00 sec 2.77 GBytes 2.38 Gbits/sec receiver [ 14] 0.00-10.00 sec 2.77 GBytes 2.38 Gbits/sec 0 sender [ 14] 0.00-10.00 sec 2.77 GBytes 2.38 Gbits/sec receiver [ 16] 0.00-10.00 sec 2.77 GBytes 2.38 Gbits/sec 0 sender [ 16] 0.00-10.00 sec 2.77 GBytes 2.38 Gbits/sec receiver [ 18] 0.00-10.00 sec 2.77 GBytes 2.38 Gbits/sec 0 sender [ 18] 0.00-10.00 sec 2.77 GBytes 2.38 Gbits/sec receiver [SUM] 0.00-10.00 sec 22.2 GBytes 19.0 Gbits/sec 0 sender [SUM] 0.00-10.00 sec 22.2 GBytes 19.0 Gbits/sec receiver ------------------------------------------------- After lro reset ifconfig vmx0 -lro & ifconfig vmx0 lro ------------------------------------------------- client -> freenas - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 4] 0.00-10.00 sec 5.39 GBytes 4.63 Gbits/sec sender [ 4] 0.00-10.00 sec 5.39 GBytes 4.63 Gbits/sec receiver [ 6] 0.00-10.00 sec 4.84 GBytes 4.16 Gbits/sec sender [ 6] 0.00-10.00 sec 4.84 GBytes 4.16 Gbits/sec receiver [ 8] 0.00-10.00 sec 5.05 GBytes 4.34 Gbits/sec sender [ 8] 0.00-10.00 sec 5.05 GBytes 4.34 Gbits/sec receiver [ 10] 0.00-10.00 sec 4.55 GBytes 3.91 Gbits/sec sender [ 10] 0.00-10.00 sec 4.55 GBytes 3.91 Gbits/sec receiver [ 12] 0.00-10.00 sec 4.91 GBytes 4.22 Gbits/sec sender [ 12] 0.00-10.00 sec 4.91 GBytes 4.22 Gbits/sec receiver [ 14] 0.00-10.00 sec 5.36 GBytes 4.61 Gbits/sec sender [ 14] 0.00-10.00 sec 5.36 GBytes 4.61 Gbits/sec receiver [ 16] 0.00-10.00 sec 4.79 GBytes 4.11 Gbits/sec sender [ 16] 0.00-10.00 sec 4.79 GBytes 4.11 Gbits/sec receiver [ 18] 0.00-10.00 sec 4.40 GBytes 3.78 Gbits/sec sender [ 18] 0.00-10.00 sec 4.40 GBytes 3.78 Gbits/sec receiver [SUM] 0.00-10.00 sec 39.3 GBytes 33.7 Gbits/sec sender [SUM] 0.00-10.00 sec 39.3 GBytes 33.7 Gbits/sec receiver freenas -> client - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 4.14 GBytes 3.55 Gbits/sec 0 sender [ 4] 0.00-10.00 sec 4.14 GBytes 3.55 Gbits/sec receiver [ 6] 0.00-10.00 sec 4.03 GBytes 3.46 Gbits/sec 0 sender [ 6] 0.00-10.00 sec 4.03 GBytes 3.46 Gbits/sec receiver [ 8] 0.00-10.00 sec 4.08 GBytes 3.51 Gbits/sec 0 sender [ 8] 0.00-10.00 sec 4.08 GBytes 3.51 Gbits/sec receiver [ 10] 0.00-10.00 sec 4.06 GBytes 3.49 Gbits/sec 0 sender [ 10] 0.00-10.00 sec 4.06 GBytes 3.49 Gbits/sec receiver [ 12] 0.00-10.00 sec 4.01 GBytes 3.45 Gbits/sec 0 sender [ 12] 0.00-10.00 sec 4.01 GBytes 3.45 Gbits/sec receiver [ 14] 0.00-10.00 sec 4.03 GBytes 3.46 Gbits/sec 0 sender [ 14] 0.00-10.00 sec 4.03 GBytes 3.46 Gbits/sec receiver [ 16] 0.00-10.00 sec 4.00 GBytes 3.43 Gbits/sec 0 sender [ 16] 0.00-10.00 sec 4.00 GBytes 3.43 Gbits/sec receiver [ 18] 0.00-10.00 sec 3.97 GBytes 3.41 Gbits/sec 0 sender [ 18] 0.00-10.00 sec 3.97 GBytes 3.41 Gbits/sec receiver [SUM] 0.00-10.00 sec 32.3 GBytes 27.8 Gbits/sec 0 sender [SUM] 0.00-10.00 sec 32.3 GBytes 27.8 Gbits/sec receiver