Major performance rw problems over VPN

[Sawtaytoes]

I just built this NAS last week and have run into some HUGE performance issues with both reads and writes over VPN: 1/10th of my Internet connection speed.

1. I tried both the OpenVPN (build-in Service) and WireGuard (setup by following the docs page).
2. I wanna rule out my Internet connection because it's 1/1Gb Google Fiber.
3. I also want to rule out my NAS's hardware. I tried local Samba from my 2.5Gb PC and get over 200MB/s on both zpools I tried (HDD + Optane cache and all an SSD pool). `robocopy` maxed out my 2.5Gb Ethernet connection (~310MB/s) seen by looking at Task Manager. I have no 10Gb machines to test with, so I don't yet know the write and read limits.

My NAS:

• TrueNAS Core 13.0 (latest as of posting)
• ASRock Rack ROMED8-2T/BCM
• Eypc 7323p
• 128GB ECC RAM
• 3 x LSI 9305 24i
• 1 x ASUS Hyper M.2 PCI 4.0
• 8 x 2TB SSDs (separate zpool literally created to test VPN speeds)
• 2 x 160GB SSDs (boot)
• 2 x 60GB SSDs (system dataset, but there's nothing on these drives).
• 8 x Intel Optane 905p (cache, log, metadata)
• 37 x HGST HDDs (includes iocage/ right now, but I wanna move it to the system drive SSDs)
• HDDs setup in 2 pools of 10 mirrors each. The second pool doesn't have 10 drives as I'm waiting for replacements nor does it have anything but data drives.
• Connected via 1 of the onboard 10Gb NICs to an 8-port UniFi Aggregation Switch with a MikroTik SFP+ adapter running at 10Gb.

I did some `iperf3` tests using OpenVPN and WireGuard, and the numbers are correct for my 1Gb Internet.

TrueNAS Core to TrueNAS Core over OpenVPN:

TrueNAS Core to Windows 11 over WireGuard:

TrueNAS Core to Windows 11 over 2.5Gb LAN:

Notice those fast-slow dips when using WireGuard? I see those over Samba as well when Windows shows the transfer graph.

I'm able to get the full connection speed using the VPN itself, but as soon as I write or read a file from either the HDD + Optane cache zpool or the all-SSD zpool, it maxes out around 12MB/s using Samba and FTPS. It sometimes goes higher. I've gotten it to show 60MB/s for a bit, but then it drops down to 12MB/s. When a friend copied files to my NAS from the other side of the world, he got 0.25-3.00MB/s over Samba using either VPN solution when his connection is 900/900Mb/s.

I wanna know what I can look at to fix VPN performance since long-distance Samba is one of the reasons I built this NAS.

I don't like the idea of using Docker over Jails, but I've been thinking of switching to TrueNAS Scale. I'm more comfortable with Debian than FreeBSD, and Linux might offer improve VPN performance and more potential solutions.

 

[jgreco]

I wanna know what I can look at to fix VPN performance since long-distance Samba is one of the reasons I built this NAS.

TCP has long had problems with latency. Most of the TCP congestion control algorithms are oriented towards endpoints that are relatively nearby, maybe a few thousand miles at best. You are very unlikely to be able to get anywhere near 1Gbps going halfway around the globe. The ping times alone just don't really support it.

For TCP, in order to get better speeds, you have to cram more stuff in the pipe so that there is more in flight. This involves increasing the window size and buffer sizes as far as practical, eliminating any packet loss, and making sure you have the best congestion control algorithm for the job.

As for SMB, I am only vaguely familiar with it at a low level; I know it is an extremely chatty protocol, which is going to murder what little performance is possible on a high latency link, but there may be some tuning available. I'm just not the one who has those details. If there is a chance that you can move to a protocol such as HTTP that has MUCH less unnecessary chattiness, that's likely to be vastly preferable and there are many resources out on the 'net that discuss how to optimize TCP for high latency WAN.

 

[Sawtaytoes]

I saw something on WebDAV. Pretty sure I got slow speeds there too, but not 100% sure.

I also tried FTP and has higher consistency, but the same speeds.

 

[morganL]

I saw something on WebDAV. Pretty sure I got slow speeds there too, but not 100% sure.

I also tried FTP and has higher consistency, but the same speeds.

Can you be very clear about how you test the file transfer speeds... what software, what file sizes etc.

 

[Sawtaytoes]

I go into Windows, lookup the network share, and copy-paste a large video file or multiple (4GB+ each) from my striped SATA or PCIe 4.0 NVMe drives.

Or I go into FileZilla or WinSCP and transfer the same file over FTP.

 

[Volts]

Linux might offer improve VPN performance and more potential solutions

I wonder if the SCALE kernel includes TCP BBR.

On CORE you could load and test cubic. It anecdotally helps Plex.

As a troubleshooting step, rule out the VPN vs. Internet Latency. Does performance drop when connected over the VPN on the LAN?

 

[ChrisRJ]

Or I go into FileZilla or WinSCP and transfer the same file over FTP.

Not SFTP?

 

[Sawtaytoes]

Not SFTP?

Naw, specifically FTP + TLS. I only did it for testing. Samba would be ideal.

 

[Sawtaytoes]

While this may sound scorched-earth, I upgraded to TrueNAS SCALE.

I tried OpenVPN Connect, but getting access to the host wasn't possible over Samba. Probably because of the interface not being exposed or who knows what.

I didn't wanna bother and instead installed Tailscale from TrueCharts. Then I set that up in Windows, and it works at 88-103MB/s. That's fantastic! This is the kind of speed I'd expect.

Considering I have IoT devices constantly pushing data to the cloud, this was a pretty slick solution!

Sucks I had to abandon TrueNAS Core, but SCALE is way better. The UI, interface, everything is faster and easier to use, the OS is faster and easier to use, and the whole thing just works.

 

[Sawtaytoes]

As a troubleshooting step, rule out the VPN vs. Internet Latency. Does performance drop when connected over the VPN on the LAN?

My friend noticed hiccups in his network connection when connecting over OpenVPN. He wasn't even tunneling anything but requests to the NAS itself.

 

[Sawtaytoes]

My Ethernet connection is also more consistent. It might be TrueNAS SCALE or from swapping the SFP+ adapter on my switch. Not sure which happened in what order.

I did an iperf3 test with my friend in the Philippines, and the slow speeds are still there for him even with Tailscale.

iperf3 measured him at 8Mb/s. Not sure why since his connection is 900/900. Must be something to do with international links.

 

[jgreco]

iperf3 measured him at 8Mb/s. Not sure why since his connection is 900/900. Must be something to do with international links.

International links, sure, but also probably a misunderstanding of what you are buying. Your 900/900 isn't dedicated global bandwidth. It's bandwidth to your ISP. Your ISP does not buy 900 megabits of bandwidth to the rest of the Internet on your behalf, they rely on "oversubscription", where they may expect between maybe 50 and 1000 customers to each share 10Gbps of upstream bandwidth that they might have available. You MIGHT be able to get 900/900 to the nextdoor ISP on the island, as long as only a few people are trying to do it.

It is fairly cheap to set up a local ethernet peering with other local service providers, so usually that's the thing that there will be best bandwidth for. However, once you start talking traversing undersea fiber, those waves are expensive, and your ISP is probably going to have a detailed understanding of what "normal access patterns" look like, and will have purchased that amount of bandwidth plus a modest amount of extra capacity.