Very slow file transfers, can't figure out why

[cjb]

Hi all, I have a TrueNAS server in a data center that I access from my laptop. For a long time everything worked fine, I'd get reasonable data transfer speeds of ~200 Mbps. However one day a few weeks ago speeds suddenly dropped off, and ever since I'm only getting ~10-15 Mbps (very painful). Nothing noteworthy changed on my server or laptop. I tried restarting both my server and laptop, the problem still persists. I ran speedtest on both the server and laptop, and it shows a reasonable bandwidth of ~1 Gbps for both of them. I ran iperf3 and it confirms my experience of ~10-15 Mbps. I've tried it from different networks (on the laptop client side), in different geographical areas, with and without a VPN, but I always get those slow transfers. I called both my server host and my ISP, both insist they are not throttling me. I've exhausted everything I know how to do, but can't root cause or fix the issue. I'm hoping someone here has some insight? I don't know if this is a TrueNAS issue or something else, sorry if I'm in the wrong place. Thanks in advance for your help!

 

[Arwen]

First, full hardware details, including the disk make, model and what port is used to connect the disks.

Their are known problems with some hardware that is basically incompatible with ZFS. This sounds like one of them. And yes, things like disks can appear to be higher speed, but whence you use these disks, they get slower and slower.

 

[cjb]

Hi Arwen, thanks for the response. I have a SuperMicro SSG-5019D8-TR12P, it's a 1U 12-disk Xeon server. I have 6x Seagate 16TB Exos SATA disks. The disks are 65% full. I don't think the disks are the issue though, I've been using this server for years with no problems, this has only emerged recently. I haven't made any changes to server configuration recently (related to disks or otherwise) that might have triggered the problem. When I copy files locally on the server the disks run as fast as ever.

 

[Arwen]

How is the network wired?

According to the system board specifications, their are 4 x 1Gbit/ps, 2 x 10Gbit/ps over copper and 2 x 10Gbit/ps on SFP+. I don't remember how to check network speed, but that is one thing to look at.

I once had a copper run that should have been 1Gbits/ps, but it dropped to 100Mbits/ps. Turns out one of the patch panel jacks had a minor wiring problem, in such a way that it affected only Gigabit Ethernet. That is because 1Gbit/ps and above use all 4 pairs. But 10/100Mbit/ps only use 2 pairs of the Category 5 or above cable. So I got "lucky" and the path still worked, but at the lower speed. Easy fix.

So, if you are using either 1Gbit/ps or 10Gbit/ps over copper, perhaps a wire has failed. And yes, a single wire failing would cause a drop to 100Mbit/ps, if it was not one of the 4 wires used by 10/100Mbit/ps Ethernet. And again yes, they CAN fail without much effort if it was marginal before. For example, heat and vibration may make a wire that was working, wiggle it's way loose.


All that said, I have seen external network paths be the issue because of increased load. Or asymmetrical routing where one path is noticeably slower.

 

[cjb]

Interesting info, thanks! I don't have physical access to the server, but I'll have the network admin double check the cabling. Unfortunately I don't think we'll find the problem here though, because speedtest.net tells me I get almost 1 Gbps both up/down when connecting to a test server in the local area of my server.

This did lead me down a different path though, and I found something possibly unusual. When I connect to a speedtest server that is geographically further away, in the local vicinity of my client laptop (I'm about 1000 mi away from my server), the bandwidth drops to ~100 Mbps. I imagine it's normal for bandwidth to drop off with distance, but a 10x drop seems pretty big? And it still doesn't account for my actual data transfer performance, which is yet another 10x slower (more like 10 Mbps).

 

[Arwen]

Well, I am out of ideas then.

Perhaps someone else can jump in and assist further.

 

[joeschmuck]

I called both my server host and my ISP, both insist they are not throttling me.

Unfortunately it does sound exactly like I would expect throttling to look like.

When I connect to a speedtest server that is geographically further away, in the local vicinity of my client laptop (I'm about 1000 mi away from my server), the bandwidth drops to ~100 Mbps.

Could it be a network issue with some other internet entity? I don't know anything about how the physical internet actually works, how data makes it from point A to point B. Maybe your ISP can perform a speed test as you did to verify the 100Mb/s speed issue, then work their magic to figure out what is happening. With any luck it will just clear up in a few days. Maybe too many kids are playing games on their smartphones hogging the bandwidth

 

[cjb]

Thanks Arwen for trying, and your suggestions did lead me to this new discovery about the dropoff in bandwidth with distance.

Thanks Joe, I agree I'll probably have to work with the data center to figure out what's going on. The ISP shouldn't be the issue, I've tried from a few different ISPs in different geographical areas and have the same problem. And it's been very persistent over time, so unlikely to be a transient surge in traffic. Maybe my server provider is throttling me but doesn't realize it.

 

[Constantin]

Is your service symmetrical or assymetric like my residential ISP connection ? (Comcast). Xfinity claims 400/20Mbit service here and 1000/50 on the receiving end of a Wireguard tunnel but maximum upload throughput through the tunnel is a consistent 13 Mbit, regardless of local load.

Unless you have a commercial account with guaranteed throughput, I’d wager your connection is getting throttled somewhere or a cable / config has gone bad. If possible, I’d work with the ISP / data center to iperf connections at every step from home to the data center with random data to see where the slowdown occurs?

 

[cjb]

Thanks Constantin, I'll check with the data center to see if a config might be causing this. I'm supposed to have a symmetric connection, and I did have symmetric performance for years until I started hitting this issue :)