Networking performance issue, can only get ~25% of available bandwidth

[Joshua.Weber]

I'm running a OpenVPN inside a jail on FreeNAS. The connection to the VPN seems to authenticate and connect correctly. Routing to other devices on the server subnet work. Proxy of internet requests via the server's public internet connection work. However performance is slow, and at times it seems the connection temporarily drops or lags out (video streams halt and report a lack of internet connection).

The Client is located in Japan with a server based in the USA. So response time on ping is reasonably high at ~135ms. Server bandwidth via speedtest-cli shows 75Mbps down / 35 Mbps up. Speedtest results at the client are 63 Mbps down and 154 Mbps up. However all bandwidth speed tests on the client through the VPN connection are very limited, ~5-7 Mbps down and ~3-5 Mbps up.

CPU reported by freenas reporting never exceeds 10%. It doesn't feel compute limited or resource limited.

I've tried to change MTU settings via -fragment and -mssfix. Currently set to --fragment 1390 --mssfix. But it doesn't seem to have an impact on performance.
I've setup an new jail. Installed a Wireguard server. Changed my client to a Wireguard client. Connection works, routing to all destinations works. But still limited in bandwidth to perhaps ~<10 Mbps during speedtesting.

I decided to do performance testing outside of a VPN. I utilized iperf3 tool to measure network performance from my client to FreeNAS server host directly via the public internet. Which was only able to achieve a 7.5 Mbps performance.

I'm currently confused as to what the problem could be. But it does seem like there is a significant performance bottleneck on my FreeNAS host. Extra confusing is that the FreeNAS host seems to be able to communicate directly with the public internet without any limitation, scoring high on the speedtest-cli results. Being able to download via wget at a fast full speed rate.

Any advice or next steps anyone could suggest would be greatly appreciated.

Josh

FreeNAS 11.1-U6. CPU Xeon E3-1220. 16GB Memory.

 

[dlavigne]

What's the full output of ifconfig ?

 

[Joshua.Weber]

What's the full output of ifconfig ?

ifconfig inside my OpenVPN jail

Code:

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384             
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>                 
        inet6 ::1 prefixlen 128                                               
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1                           
        inet 127.0.0.1 netmask 0xff000000                                     
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>                             
        groups: lo                                                           
epair0b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>                                                   
        ether 52:7d:28:30:cd:7d                                               
        inet 192.168.1.202 netmask 0xffffff00 broadcast 192.168.1.255         
        nd6 options=9<PERFORMNUD,IFDISABLED>                                 
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)                   
        status: active                                                       
        groups: epair                                                         
tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500         
        options=80000<LINKSTATE>                                             
        inet 10.8.0.1 --> 10.8.0.2  netmask 0xffffffff                       
        nd6 options=1<PERFORMNUD>                                             
        groups: tun                                                           
        Opened by PID 95690                                                   

And from my FreeNAS host.

Code:

igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500                      
        options=2400b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6>     
        ether 0c:c4:7a:c2:17:24                                                                         
        hwaddr 0c:c4:7a:c2:17:24                                                                        
        inet 192.168.1.200 netmask 0xffffff00 broadcast 192.168.1.255                                   
        nd6 options=9<PERFORMNUD,IFDISABLED>                                                            
        media: Ethernet autoselect (100baseTX <full-duplex>)                                            
        status: active                                                                                  
igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500                              
        options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,
V6>                                                                                                     
        ether 0c:c4:7a:c2:17:25                                                                         
        hwaddr 0c:c4:7a:c2:17:25                                                                        
        nd6 options=9<PERFORMNUD,IFDISABLED>                                                            
        media: Ethernet autoselect                                                                      
        status: no carrier                                                                              
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384                                       
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>                                           
        inet6 ::1 prefixlen 128                                                                         
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3                                                      
        inet 127.0.0.1 netmask 0xff000000                                                               
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>                                                       
        groups: lo                                                                                      
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500                           
        ether 02:45:d3:58:b6:00                                                                         
        nd6 options=9<PERFORMNUD,IFDISABLED>                                                            
        groups: bridge                                                                                  
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15                                     
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200                                        
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0                                        
        member: epair2a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>                                   
                ifmaxaddr 0 port 7 priority 128 path cost 2000                                          
        member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>                                   
                ifmaxaddr 0 port 5 priority 128 path cost 2000                                          
        member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>                                   
                ifmaxaddr 0 port 6 priority 128 path cost 2000                                          
        member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>                                      
                ifmaxaddr 0 port 1 priority 128 path cost 200000                                        
epair1a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500                   
        options=8<VLAN_MTU>                                                                             
        ether 02:cd:50:00:06:0a                                                                         
        hwaddr 02:cd:50:00:06:0a                                                                        
        nd6 options=1<PERFORMNUD>                                                                       
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)                                             
        status: active                                                                                  
        groups: epair                                                                                   
epair0a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500                   
        options=8<VLAN_MTU>                                                                             
        ether 02:cd:50:00:05:0a                                                                         
        hwaddr 02:cd:50:00:05:0a                                                                        
        nd6 options=1<PERFORMNUD>                                                                       
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)                                             
        status: active                                                                                  
        groups: epair                                                                                   
epair2a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500                   
        options=8<VLAN_MTU>                                                                             
        ether 02:cd:50:00:07:0a                                                                         
        hwaddr 02:cd:50:00:07:0a                                                                        
        nd6 options=1<PERFORMNUD>                                                                       
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)                                             
        status: active                                                                                  
        groups: epair                                                                                   

 

[dlavigne]

Were you able to resolve this?

 

[Joshua.Weber]

I have still been unable to resolve this. I'm at a loss for next steps. I'm thinking of doing an upgrade from 11.1 -> 11.2. And then recreating the jails. My understanding is there is potentially some difference between the warden and iocage jails types.

But I'm really just grasping at straws. I don't understand what the underlying issue could be.

 

[sretalla]

You're probably on the right track with the MTU and fragmentation.

A tunnel will add headers to the packets, so the tunnel needs to have a smaller MTU than the interface it relies on (and actually the entire route it will traverse).

What the number will need to be exactly can be found by using ping -D -s 1490 <IP ADDRESS> in the jail, where you keep reducing from 1490 until you hit a number where you see the ping respond correctly. Then you set the MTU to that number in the jail.

 

[Joshua.Weber]

Hmmmm. Mentioning the MTU made me realize I haven't perhaps been looking at the whole picture. And lack some background in network technology and how MTU fully works. A network packet traverses the following

Client PC -> WiFi Router (running OpenPVN client) -> ISP Modem -> ISP/Internet -> Host network router -> FreeNAS host -> FreeNAS jail -> OpenVPN server app -> FreeNAS host -> Host network router -> Internet

When I was changing or attempting to target MTU I did it at the OpenVPN layer. Trying to reduce the setting via --fragment and --msfix options. But I never looked at any of the other adapters.

If the WiFi router picks up the smaller MTU passed via the OpenVPN server, Will the client PC follow those rules? Does MTU propagate across connection settings for the network? Does MTU settings on the FreeNAS / FreeNAS jail adapters impact this?

If anyone with more network knowledge than I could share some insight I would be grateful.

 

[Elliot Dierksen]

If anyone with more network knowledge than I could share some insight I would be grateful.

I don't have a "basics of VPN tunnels" link handy, but here is the analogy I normally use to explain it to people. Imagine you regular data packet is a letter sized envelope. Your underlying transport is also a letter sized envelope as in exactly the same size packets. If you have a packet that fills the envelope, you have to fold it over to jam it into another envelope of the same size. When the encrypting device gets a packet that is going to be encrypted but won't fit in a single outgoing packet, it has two choices. First, it can just drop the packet. This is what will frequently happen, particularly if the sending system sets the DF bit (do not fragment). Windows loves to do this for no good reason. The other choice is for the encrypting device to break the single packet into multiple encrypted packets. This can sometimes cause problems if this annoys the application, plus adds additional work to the encrypting device. That is why it is a standard practice to artificially push the MTU down for traffic that will be encrypted to avoid this problem.

 

[sretalla]

When I was changing or attempting to target MTU I did it at the OpenVPN layer. Trying to reduce the setting via --fragment and --msfix options. But I never looked at any of the other adapters.

I think this may describe the source of the problem...

MTU describes the maximum number of bytes that can be transmitted in a single packet (including its headers).

When you tell OpenVPN to fragment at 1380, it will fragment all the packets bigger than that (probably almost all of them) and hence double the number of packets needing to be sent along the tunnel compared to the input.

If you are going to set that option, you should make it big enough to fit one entire packet from the source and final destination networks.

You then need to look at the source and destination MTU size (and make sure everything in between is OK too... although your "full speed" testing seems to show it is probably OK) and make sure that after the OpenVPN tunnel headers are added to the packet, it still fits within a single MTU-sized packet.

A quick bit of googling tells me that 18 bytes is perhaps the number you need to allow for, so an MTU of 1482 would probably be OK for OpenVPN for it to operate on a network chain where the MTU is 1500 without the tunnel.

This means all the clients on the OpenVPN network need to respect the MTU of 1482 also, or packets will need to be fragmented or will be dropped as mentioned by Elliot.