Remote Connections Limited to 10 mbps

[TheSunKing]

Hey everyone,

In brief: I have 20 mbps upload on my home network and the server is capable of transferring at 24 mbps over local wifi, but when downloading files remotely via SFTP, it seems to be limited to 10 mbps.

More detail:

Router: Cheap-o TP Link N600

Server Hardware:
MB: SUPERMICRO MBD-A1SRi-2758F
CPU: Embedded Intel Atom C2758
RAM: 2x Kingston 8 GB ECC RAM (KVR16LSE11/8)
Pool: 3x 2 TB WD Red HDDs, RAIDZ
Network Controller: Embedded C2000 SoC I354 Quad GbE Controller

Please see the picture above for the various connection speeds that I've tested.
Between running iozone and the 24 mbps local transfer from the server to my phone (over wifi, at that), I'm convinced the server is not being limited by IO.
Additionally, the transfer to my phone was via an SFTP connection, so I'm somewhat convinced the server is not being bound by encryption speeds. Also, I experimented with other ciphers (blowfish and arcfour) and they had no effect on the transfer speeds.
Running speedtests on my local network show that my upload speed is indeed ~20 mbps up, so I shouldn't be hitting my upload cap... All my ethernet cables are Cat6...
As noted in the picture, ifconfig reports that the LAN interface on the server has negotiated a 1 Gb connection.

So, despite all the pieces seeming to check out individually, SFTP transfers from the server to a remote PC are only 10 mbps. IOperf reports only 6 mbps, at that.
I'm stumped!

Does anyone have any ideas on what could be limiting the transfer speed to remote devices?
Many thanks in advance.

 

[kdragon75]

Sounds like it could be an ISP thing. Try setting your SSH port to something in the 10000 range and see if that makes any difference. It could also be the "cheap-o TPlink". Also how do your internet speed tests look?

 

[TheSunKing]

The SSH port is already in the 60,000 range, so it shouldn't be one of the ports that an ISP might throttle.
The TP Link is supposedly capable of gigabit connections, but I'm not sure I trust that claim. It should at least be capable of 100 mbps, though.
As noted in the picture, speedtests on the local network (over both LAN and wifi) show 100 mbps down and 20 mpbs up, as they should.

 

[kdragon75]

Your diagram does not show raw upload performance from LAN to WAN.

I have 20 mbps upload on my home network

This is the only claim to 20mb/s or 2.5MB/s WAN upload.

 

[TheSunKing]

The purple lines on the the diagram show the speedtests I've run on the local network. Is there a different test I should run?
Thanks!

 

[kdragon75]

speedtest.net ?

 

[TheSunKing]

Speedtest.net shows 110 mbps down and 22 up on the local network. The previous results were from the speed test built in to google.

 

[kdragon75]

Speedtest.net shows 110 mbps down and 22 up on the local network. The previous results were from the speed test built in to google.

Ok I understand now. You didn't say the 20mb/s was from a speed test. I sounded like you were simply stating you service is provisioned for 20mb/s.

Do you have jumbo frames enabled anywhere? This definitely sounds like a router/ISP issue.

 

[TheSunKing]

I had never heard of jumbo frames before - just had to look them up.
Currently the max transfer unit of my router is set to the default, 1500 bytes. I couldn't find any other related options in the router.
If it's possible to modify it in FreeNAS, I never have, so it would be set to the default.

 

[kdragon75]

Yeah don't mess with jumbo frames.

 

[Elliot Dierksen]

Yeah don't mess with jumbo frames.

I will second that. For jumbo frames to do you any good, they have to be supported along the entire path that the packets traverse. I can guarantee you that your ISP doesn't support them. The current wisdom seems to be that jumbo frames really aren't worth the trouble any more since the gains from they are pretty minimal. You also have to remember that you are only fast as the slowest link in the chain. You may be able to synthesize 20Mb of upstream traffic to the speed test site, but that doesn't mean that your hard drives spinning, file system organization, and a myriad of other things won't slow you down way below the theoretical limit. You can even have propagation delays in TCP traffic. It depends how much data the TCP stack will push when it hasn't received an acknowledgment. That can cause slow transmits over fast links just because of the time it takes the bits to reach their intended destination. Sorry, I hope that didn't muddy the waters too much.

 

[TheSunKing]

You may be able to synthesize 20Mb of upstream traffic to the speed test site, but that doesn't mean that your hard drives spinning, file system organization, and a myriad of other things won't slow you down way below the theoretical limit.

The server shouldn't be limited by io, given that it's capable of the desired speeds on the local network. (And iozone shows read/write speeds much higher that my upload limit)

You can even have propagation delays in TCP traffic. It depends how much data the TCP stack will push when it hasn't received an acknowledgment. That can cause slow transmits over fast links just because of the time it takes the bits to reach their intended destination.

That may be worth investigating! Any tests I can run to see if this is impacting me?

 

[Stux]

Some ISPs prioritize Speedtest traffic.

One reliable way to test is to put a large file (100-1000MB) on a web server facing the network, then time how long it takes you to download that from the network.

It should be circa 2MB/s.

Don’t muddy the waters until you’ve confirmed you can actually get 20mbps, and I don’t think speedtest.net is good enough for that. As I said, some ISPs prioritize it.

 

[TheSunKing]

Some ISPs prioritize Speedtest traffic.

Ooh, tricky. Alright.

One reliable way to test is to put a large file (100-1000MB) on a web server facing the network...

I won't lie to you, this kind of went over my head. I tried uploading a ~200 MB file to google drive and timing the upload. That uploaded at 2.1 MB/s. Is that test sufficient?

 

[Stux]

Ooh, tricky. all right.


I won't lie to you, this kind of went over my head. I tried uploading a ~200 MB file to google drive and timing the upload. That uploaded at 2.1 MB/s. Is that test sufficient?

Yes. Probably. Unless they’re priorizing google drive too ;)

 

[Elliot Dierksen]

I tried uploading a ~200 MB file to google drive and timing the upload. That uploaded at 2.1 MB/s. Is that test sufficient?

Another thing to keep in mind is you are mixing bits and bytes there. I honestly am not being patronizing here, or not intending to be that way. 20 megabits divided by 8 is 2.5 megabytes. With protocol overhead and various other things, 2.1 megabytes speed is actually close to the stated speed. My personal rule of thumb is that any time you get to 90% of the physical link speed, you are doing pretty well.

 

[Elliot Dierksen]

That may be worth investigating! Any tests I can run to see if this is impacting me?

I used to use an open source tool called iperf. If you are just trying to test the limits of the transport itself, the best way to do that is to use a program that generates the traffic. When you do file transfers, you bring a host of other elements into the picture. Those are all relevant, but you have to test things in a way that allows you to validate one piece at a time. Otherwise it is difficult determine what part is slowing you down.

 

[TheSunKing]

2.1 megabytes speed is actually close to the stated speed

Understood. I'm perfectly happy with 2.1 MB/s. The problem I'm having is the 1.2 MB/s (10 mbps) I'm getting on the remote PC.

I used to use an open source tool called iperf.

Yup, I used iperf/jperf before posting. I actually made a typo in my diagram - ioperf should be iperf. I got only 0.7 MB/s (6 mbps) in the iperf test.

 

[kdragon75]

Yup, I used iperf/jperf before posting. I actually made a typo in my diagram - ioperf should be iperf. I got only 0.7 MB/s (6 mbps) in the iperf test.

Can you provide the EXACT command that was used for iperf (both ends please)? Iperf has man options for testing different aspects of network performance and we need t ascertain whether you method was conducive to testing throughput.

 

[TheSunKing]

Can you provide the EXACT command that was used for iperf (both ends please)? Iperf has man options for testing different aspects of network performance and we need t ascertain whether you method was conducive to testing throughput.

Sure thing!

Server side:

iperf -sD


Client side, using jperf GUI:

iperf -c redacted.duckdns.org -P 1 -i 1 -p xxxxx -f M -t 120