Super slow SMB speeds on TrueNAS Mini X+, Core 12

[630fireman]

So we recently purchased a high spec TrueNAS Mini X+, the version with 64GB RAM and the integral twin 10Gbps ports, running 4x Seagate Exos 16TB in RAIDZ2, running latest TrueNAS Core 12. Unfortunately, we've never been able to get this thing to really run properly on the network locally or over OpenVPN. The OpenVPN connection works great, and I'm getting the similar pathetic speeds I'm seeing locally. I've done some of this testing over the VPN too, but overall, the speeds are similar enough to local connections that I feel that I can say that service is set up properly, but I also did some testing locally with the OpenVPN services shut completely off.

iperf2 results over VPN (I'm no pro with it, so there are probably better parameters to have used)

[ ID] Interval Transfer Bandwidth [ 8] 0.0-30.1 sec 6.38 MBytes 1.78 Mbits/sec [ 6] 0.0-30.2 sec 8.38 MBytes 2.33 Mbits/sec [ 11] 0.0-30.2 sec 14.6 MBytes 4.06 Mbits/sec [ 4] 0.0-30.3 sec 7.88 MBytes 2.18 Mbits/sec [ 3] 0.0-30.3 sec 6.00 MBytes 1.66 Mbits/sec [ 5] 0.0-30.3 sec 7.50 MBytes 2.08 Mbits/sec [ 9] 0.0-30.3 sec 7.75 MBytes 2.14 Mbits/sec [ 10] 0.0-30.4 sec 7.00 MBytes 1.93 Mbits/sec [ ID] Interval Transfer Bandwidth [ 10] 0.0-10.1 sec 4.25 MBytes 3.54 Mbits/sec [ 3] 0.0-10.1 sec 2.88 MBytes 2.39 Mbits/sec [ 8] 0.0-10.1 sec 3.12 MBytes 2.59 Mbits/sec [ 7] 0.0-10.2 sec 4.50 MBytes 3.70 Mbits/sec [ 4] 0.0-10.2 sec 3.62 MBytes 2.98 Mbits/sec [ 6] 0.0-10.3 sec 2.25 MBytes 1.83 Mbits/sec [ 11] 0.0-10.4 sec 3.75 MBytes 3.03 Mbits/sec [ 5] 0.0-10.5 sec 2.88 MBytes 2.31 Mbits/sec

My theory is that it is somehow our network connection. Connecting to it locally through the wired LAN or wifi has given very poor speeds (~11MB/s and ~3MB/s respectively), but isolating the system from the main network and connecting directly via an unmanaged switch and static IPs (and manually setting the gateway on the PC to the NAS IP) gave speeds around 117MB/s. These aren't the only benchmarks I've run, but the results that I had saved at least. Have tried multiple PCs and Macs for testing and benchmarking with similar results.


CrystalDiskMark benchmark - Left locally wired through network, Right via isolated connection

This is making me conclude that the problem lies within the network, but I'm not sure what is wrong. The connection to the internet is fine as seen below and the router seems to not interfere with anything else (internet to and from as well as network printers/scanners and the web utilities for these devices and also TrueNAS web admin). I've checked MTUs (running at 1500). The router is a cheap gigabit router (Trendnet TEW-824DRU) and is on my list to upgrade, but I don't want to pull ahead a big invest in more professional gear for our organization (a non-profit historical association) without confirming that is 100% the root cause (and then not having it fix the problem...). We also have a 24 port 10/100/1000Mbps switch that some devices are plugged into. Both the LAN feeding the switch from the router and one of the TrueNAS 10Gbps ports are plugged directly into the 1000Mbps ports of this switch. Many of the other devices are using the 100Mbps ports (could explain the wired speed perhaps?), but the wireless speed should still be greater if that were the case. The other TrueNAS port is wired directly to the router. Benchmarking that one had similar speeds, wired and wireless.


Ookla Speedtest run through the TrueNAS shell

Does anyone have any ideas on the root cause? Perhaps it's not the router, but some settings I've missed in TrueNAS? I've done as much testing as I can find and have tried changing multiple things without success, but am open to more testing if necessary to diagnose and also to investing in Ubiquiti gear sooner than previously planned if necessary. I'll be happy to explain more to help dig into things.

 

[sretalla]

I've checked MTUs (running at 1500).

Is this in or out of the OpenVPN tunnel... if the tunnel is set at 1500 too, then all your packets will be getting fragmented. (just one example, but maybe it will help to understand and diagnose: https://www.thegeekpub.com/271035/openvpn-mtu-finding-the-correct-settings/)

 

[630fireman]

Is this in or out of the OpenVPN tunnel... if the tunnel is set at 1500 too, then all your packets will be getting fragmented. (just one example, but maybe it will help to understand and diagnose: https://www.thegeekpub.com/271035/openvpn-mtu-finding-the-correct-settings/)

Both. The router only supports up to 1500 MTU.

 

[Jessep]

Check switch port (and/or TrueNAS NIC) for 1Gb negotiation? 11MB is 100Mb speeds.

 

[630fireman]

Check switch port (and/or TrueNAS NIC) for 1Gb negotiation? 11MB is 100Mb speeds.

The TrueNAS is hooked up to the 1Gbps port on the switch and the line between the switch and router are on Gbps ports as well. I do know the line between that desktop pc and the switch is on a 100Mbps port so that speed does make sense.

But still, having 1-4Mbps over both WiFi and VPN does not make sense.

 

[sretalla]

Both. The router only supports up to 1500 MTU.

Did you visit the link I posted? if the tunnel MTU is 1500 and that's what you have on the NIC and routers too, you'll be adding your tunnel headers to that 1500 in every packet and it will need to be bifurcated... that will slow you down for sure... read the article I linked.

 

[Jessep]

The TrueNAS is hooked up to the 1Gbps port on the switch and the line between the switch and router are on Gbps ports as well. I do know the line between that desktop pc and the switch is on a 100Mbps port so that speed does make sense.

But still, having 1-4Mbps over both WiFi and VPN does not make sense.

What the port is capable of isn't necessarily what it is running at

I would suggest follow each cable/port in the full path and verify in the switch what the port is running at

You could have a misconfigured port, a bad cable, etc.

 

[630fireman]

What the port is capable of isn't necessarily what it is running at

I would suggest follow each cable/port in the full path and verify in the switch what the port is running at

You could have a misconfigured port, a bad cable, etc.

Today, I did a trace on all the ports, moved everything to a different gigabit switch and was successfully getting gigabit speeds through the switch and also through the router, testing both through iperf and CrystalDiskMark. So now I know I'm getting gigabit speeds to and through the router.

Did you visit the link I posted? if the tunnel MTU is 1500 and that's what you have on the NIC and routers too, you'll be adding your tunnel headers to that 1500 in every packet and it will need to be bifurcated... that will slow you down for sure... read the article I linked.

Reviewing the link. Tried the connection over the VPN both before I did the experimentation about and after. Speeds are marginally faster, but nothing blazing fast, at both 1470 and 1450 MTU, and also with mssfix 1410 (at MTU 1450). At this point it's leading me to a configuration issue with OpenVPN.

L - Speed over wired LAN, TN system on switch, PC on Switch.
M - Speed over wired LAN, TN system on switch, PC on router, router to switch.
R - Speed over VPN (from my house with Gigabit internet ~5 miles away from install site.

Current OpenVPN config in TrueNAS.
I did have it running UDP at one point weeks ago, but that for some reason started to never connect, and I also was getting poor speeds through it.

Code:

client
dev tun
proto tcp
port 1194
remote "redacted"
user nobody
group nobody
persist-key
persist-tun
tun-mtu 1450
mssfix 1410
**keys redacted**
verb 3
remote-cert-tls server
auth SHA256
cipher AES-256-CBC
**tls crypt redacted**

.ovpn file parameters

 

[630fireman]

Bumping to see if there are any thoughts on the root cause of the VPN speed. As mentioned in my previous post, I did resolve the issue with speed over the LAN.

 

[630fireman]

This problem with speed outside of the LAN is still plaguing me unfortunately.

I recently upgraded our network gear to Ubiquiti, but am still getting the same problems: Gbps on LAN (great!), but still only about 16Mbps over VPN (both OpenVPN on TrueNAS and IPSec on the UDM Pro). I also tried connecting via SMB directly via port forward to see if the VPN was the issue and still the same slow speeds. I called the ISP and they confirmed there are no port restrictions.

I've tried all the same tests as above, plus also tried jumbo frames, but no luck there either. Anyone have any thoughts?

 

[spuky]

in your first post you had a speedtest with arround 100 Megabit/s so divided by 8 brings us to about 12.5 Megabyte/s substract vpn and smb overhead we are not that far of... did your internet connection get any faster since your first post ? maybe post speedtest from both locations