FreeNAS 12 VM and jail network performance is very very slow

KevDog

Patron
Joined
Nov 26, 2016
Messages
462
I'm getting terrible network performance within FreeNAS 12.0 particularly between clients and either FreeNAS jails or VMs. I'm not finding a problem when connecting to the main FreeNAS installation - only the VMs or jails. jails are utilizing the vnet implementation. In terms of hardware I have a SuperMicro X11SSL-CF motherboard. Something tells me its not hardware related since I my iperf3 tests (shown below) between a wireless client located on the same LAN are roughly equivalent between the main freenas installation and another VM not running on FreeNAS. The network speed drops precipitously with jails or VMs implemented by FreeNAS. For what its worth I have all traffic tagged entering the FreeNAS and a serious of bridges associated with each tagged network. Perhaps the introduction of tags is slowing performance. In my tests below the FreeNAS jail and VM were all tied to the same bridge.

Igb0---->>> VLAN1 ---->>> Bridge0 ---->>> Jails and VM (in this test tied to Bridge == vnet0:bridge0)
|
| ---> Main FreeNas Installation

---->>> VLAN20 ---->>> Bridge20
---->>> VLAN30 ---->>> Bridge30
---->>> VLAN40 ---->>> Bridge40

Additional Screenshots showing implementation

Screen Shot 2020-11-26 at 7.43.18 AM.png


Screen Shot 2020-11-26 at 7.44.05 AM.png

Screen Shot 2020-11-26 at 7.53.01 AM.png



iperf 3 results from client to main freenas installation:

Code:
iperf3 -c 10.0.1.197 -f K                                                                                                                                   
Connecting to host 10.0.1.197, port 5201
[  5] local 10.0.1.184 port 56885 connected to 10.0.1.197 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  3.19 MBytes  3263 KBytes/sec                 
[  5]   1.00-2.00   sec  5.97 MBytes  6105 KBytes/sec                 
[  5]   2.00-3.00   sec  6.73 MBytes  6901 KBytes/sec                 
[  5]   3.00-4.00   sec  6.27 MBytes  6398 KBytes/sec                 
[  5]   4.00-5.00   sec  7.11 MBytes  7303 KBytes/sec                 
[  5]   5.00-6.00   sec  7.42 MBytes  7600 KBytes/sec                 
[  5]   6.00-7.00   sec  6.87 MBytes  7005 KBytes/sec                 
[  5]   7.00-8.00   sec  7.23 MBytes  7431 KBytes/sec                 
[  5]   8.00-9.00   sec  6.51 MBytes  6661 KBytes/sec                 
[  5]   9.00-10.00  sec  7.18 MBytes  7347 KBytes/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  64.5 MBytes  6601 KBytes/sec                  sender
[  5]   0.00-10.04  sec  64.4 MBytes  6571 KBytes/sec                  receiver

iperf Done.


iperf3 from client to FreeNAS Arch Linux VM

Code:
iperf3 -c 10.0.1.86 -f K                                                                                                                                
Connecting to host 10.0.1.86, port 5201
[  5] local 10.0.1.184 port 53855 connected to 10.0.1.86 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   203 KBytes   202 KBytes/sec                 
[  5]   1.00-2.00   sec  45.2 KBytes  45.2 KBytes/sec                 
[  5]   2.00-3.00   sec  18.4 KBytes  18.5 KBytes/sec                 
[  5]   3.00-4.00   sec  12.7 KBytes  12.7 KBytes/sec                 
[  5]   4.00-5.00   sec  24.0 KBytes  24.0 KBytes/sec                 
[  5]   5.00-6.00   sec  58.0 KBytes  58.2 KBytes/sec                 
[  5]   6.00-7.00   sec  43.8 KBytes  43.7 KBytes/sec                 
[  5]   7.00-8.00   sec  43.8 KBytes  43.8 KBytes/sec                 
[  5]   8.00-9.00   sec  66.5 KBytes  66.5 KBytes/sec                 
[  5]   9.00-10.00  sec  38.2 KBytes  38.2 KBytes/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec   554 KBytes  55.3 KBytes/sec                  sender
[  5]   0.00-10.02  sec   427 KBytes  42.6 KBytes/sec                  receiver

iperf Done


iperf3 from client to FreeNAS jail with base jail of
Code:
iperf3 -c 10.0.1.158 -f K                                                                                                                                    
Connecting to host 10.0.1.158, port 5201
[  5] local 10.0.1.184 port 54231 connected to 10.0.1.158 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   158 KBytes   157 KBytes/sec                 
[  5]   1.00-2.00   sec  67.9 KBytes  68.1 KBytes/sec                 
[  5]   2.00-3.00   sec  26.9 KBytes  26.8 KBytes/sec                 
[  5]   3.00-4.00   sec  9.90 KBytes  9.94 KBytes/sec                 
[  5]   4.00-5.00   sec  42.4 KBytes  42.4 KBytes/sec                 
[  5]   5.00-6.00   sec  43.8 KBytes  43.9 KBytes/sec                 
[  5]   6.00-7.00   sec  65.0 KBytes  64.9 KBytes/sec                 
[  5]   7.00-8.00   sec  43.8 KBytes  43.9 KBytes/sec                 
[  5]   8.00-9.00   sec  55.1 KBytes  55.2 KBytes/sec                 
[  5]   9.00-10.00  sec  26.9 KBytes  26.9 KBytes/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec   539 KBytes  53.9 KBytes/sec                  sender
[  5]   0.00-10.02  sec   413 KBytes  41.2 KBytes/sec                  receiver

iperf Done.


iperf3 from client to another LAN VM running within xcp-ng hypervisor -- not associated with FreeNAS

Code:
perf3 -c 10.0.1.11 -f K                                                                                                                                    
Connecting to host 10.0.1.11, port 5201
[  5] local 10.0.1.184 port 56503 connected to 10.0.1.11 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  6.15 MBytes  6291 KBytes/sec                 
[  5]   1.00-2.00   sec  6.66 MBytes  6826 KBytes/sec                 
[  5]   2.00-3.00   sec  4.26 MBytes  4354 KBytes/sec                 
[  5]   3.00-4.00   sec  7.38 MBytes  7567 KBytes/sec                 
[  5]   4.00-5.00   sec  8.76 MBytes  8964 KBytes/sec                 
[  5]   5.00-6.00   sec  8.07 MBytes  8254 KBytes/sec                 
[  5]   6.00-7.00   sec  7.54 MBytes  7710 KBytes/sec                 
[  5]   7.00-8.00   sec  6.02 MBytes  6181 KBytes/sec                 
[  5]   8.00-9.00   sec  7.26 MBytes  7431 KBytes/sec                 
[  5]   9.00-10.00  sec  7.68 MBytes  7852 KBytes/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  69.8 MBytes  7143 KBytes/sec                  sender
[  5]   0.00-10.01  sec  69.2 MBytes  7082 KBytes/sec                  receiver

iperf Done.


Here are my system tunable network related settings:

net.inet.tcp.recvbuf_max
16777216
SYSCTL
Generated by autotune


net.inet.tcp.recvbuf_inc
524288
SYSCTL
Generated by autotune


net.inet.tcp.reass.maxqueuelen
4096
SYSCTL


net.inet.tcp.mssdflt
1448
SYSCTL


net.inet.tcp.delayed_ack
0
SYSCTL


net.add_addr_allfibs
0
SYSCTL
 

KevDog

Patron
Joined
Nov 26, 2016
Messages
462
Top