Good iperf perfomance - poor scp performance

[vom]

Hello,

I'm noticing something that might have been an issue for a while, but other troubleshooting seems to have uncovered it for me. I have FreeNAS 9.3 connected via gig (wired), and my primary client is my Macbook Air connected via 802.11ac. Using iperf - I get very good numbers. Using scp (to /dev/null to rule out disk) - I get ~ 10% of the iperf numbers. Doing the same tests to a linux server however results in better more consistent numbers.

Here are the iperf numbers, FreeNAS first, linux server second:

Code:

[vom@freenas ~]$ iperf -s

------------------------------------------------------------

Server listening on TCP port 5001

TCP window size: 64.0 KByte (default)

------------------------------------------------------------

[  4] local 192.168.64.12 port 5001 connected with 192.168.64.21 port 56236

[ ID] Interval       Transfer     Bandwidth

[  4]  0.0-10.0 sec   634 MBytes   531 Mbits/sec

[vom@freenas ~]$ iperf -c 192.168.64.21

------------------------------------------------------------

Client connecting to 192.168.64.21, TCP port 5001

TCP window size: 32.5 KByte (default)

------------------------------------------------------------

[  3] local 192.168.64.12 port 30023 connected with 192.168.64.21 port 5001

[ ID] Interval       Transfer     Bandwidth

[  3]  0.0-10.0 sec   545 MBytes   457 Mbits/sec

---
vom@ice:~$ iperf -s

------------------------------------------------------------

Server listening on TCP port 5001

TCP window size: 85.3 KByte (default)

------------------------------------------------------------

[  4] local 192.168.64.1 port 5001 connected with 192.168.64.21 port 56353

[ ID] Interval       Transfer     Bandwidth

[  4]  0.0-10.0 sec   735 MBytes   615 Mbits/sec

vom@ice:~$ iperf -c 192.168.64.21

------------------------------------------------------------

Client connecting to 192.168.64.21, TCP port 5001

TCP window size: 23.5 KByte (default)

------------------------------------------------------------

[  3] local 192.168.64.1 port 49390 connected with 192.168.64.21 port 5001

[ ID] Interval       Transfer     Bandwidth

[  3]  0.0-10.0 sec   516 MBytes   432 Mbits/sec

Here's where I scratch my head - scp'ing an ISO to FreeNAS /dev/null gets 5.9MB/s - and the same to linux gets 47.0MB/s. These boxes are the same hardware, connected to the same switch.

Code:

onosendai:~   vom$ scp isos/ubuntu-14.04-server-amd64.iso vom@freenas:/dev/null

ubuntu-14.04-server-amd64.iso                                                                                                                   100%  564MB   5.9MB/s   01:36    

onosendai:~   vom$ scp -P 54321 isos/ubuntu-14.04-server-amd64.iso vom@ice:/dev/null

ubuntu-14.04-server-amd64.iso                                                                                                                   100%  564MB  47.0MB/s   00:12    

Scp'ing from the wired linux to wired FreeNAS results in pretty good numbers (>30MB/s). It almost feels like FreeBSD is more sensitive to latency on the wireless and Linux adapts ?

PS: Yes I know wireless sucks. I'm simply looking for anyone with any ideas regarding the above data points and apparent discrepancies. In other words - I know how to "solve" the problem (don't use wireless) - but I'm bothered that I can get numbers I expect out of linux but not FreeNAS.

Thanks in advance.

 

[anodos]

Post hardware specs

 

[vom]

Mobo MSI NF725M-P43
CPU AMD Phenom(tm) II X2 560
8G ram
NIC (Onboard) Realtek Semiconductor Co., Ltd. RTL8111/8168/8411
Drives - 4 x HDS725050KLA360
System boots/runs from Sandisk 8G Cruzer Fit

 

[gpsguy]

For starters, you've got a Realcrap NIC. An Intel Pro/1000 CT (~$30 USD) offers much better performance.

 

[anodos]

I suggest the following:
1) Test over wired connection
2) Retest using Intel NIC. I recently purchased a 4-port pci-e gigabit nic for about $35. There are some pretty good deals to be had.

 

[vom]

I suggest the following:
1) Test over wired connection
2) Retest using Intel NIC. I recently purchased a 4-port pci-e gigabit nic for about $35. There are some pretty good deals to be had.

1) I did do this - I get ~ 30MB/s from a linux host to freenas over wired.

2) I bought and installed an Intel 1000CT NIC as advised.

I'm getting the same results as my original post. :(

As an added measure - I did a fresh install of FreeNAS and even nuked and rebuilt my ZFS (I was wanting to do this anyway).

Any other ideas ? The way I look at it - the primary change that degrades the transfer speed is wireless. Again - iperf is fine - it's primarily scp. I'll try some AFP and maybe even FTP to see if they behave any different.

Thanks all for the information and ideas so far.

 

[DifferentStrokes]

I wonder what the hardware is on the Linux machine. SCP would have a higher CPU load.

Not real sure here what your network infrastructure is either. I know you said you tried it wired, but was this through a switch? Maybe try directly connected, with a known good patch cable, to rule out punch downs, cabling, networking equipment, etc.

These are just a few things that come to mind. It can take a little time sometimes to figure this stuff out. I had a similar problem recently. IPerf performance was great but everything else was bad. Turned out a crossover patch cable slipped into the network. The switch auto-negotiated it and flipped it so I never noticed. Swapped the cable out and speeds were where they should be.

Post back and we will help with what we can.

 

[vom]

I wonder what the hardware is on the Linux machine. SCP would have a higher CPU load.

Not real sure here what your network infrastructure is either. I know you said you tried it wired, but was this through a switch? Maybe try directly connected, with a known good patch cable, to rule out punch downs, cabling, networking equipment, etc.

These are just a few things that come to mind. It can take a little time sometimes to figure this stuff out. I had a similar problem recently. IPerf performance was great but everything else was bad. Turned out a crossover patch cable slipped into the network. The switch auto-negotiated it and flipped it so I never noticed. Swapped the cable out and speeds were where they should be.

Post back and we will help with what we can.

Hardware is identical in the linux machine (I bought all the components at the same time a few years ago).

Switch is a Cisco 3750G - it's layer2 (it's layer3 capable but these guys are all on the same vlan / subnet - dead simple).

FTP was faster:

Code:

226 Transfer complete
3320512512 bytes sent in 01:57 (26.89 MiB/s)

AFP was faster as well - but hard to quantify with the Mac's "friendly" copy dialog.

This seems very specific to scp (and thereby openssh server + fbsd IP stack).

I think for the time being if I have something to copy over wireless - I'll kick it old school and use FTP.

 

[depasseg]

Check out 'top' when running. I'm guessing the encryption involved with SCP is placing a heavy load on your CPU.



Sent from my Nexus 7 using Tapatalk

 

[vom]

Check out 'top' when running. I'm guessing the encryption involved with SCP is placing a heavy load on your CPU.



Sent from my Nexus 7 using Tapatalk

Yes I can certainly look at that - however scp from a box right beside it on gig wired runs at a decent speed. The only change that affects the speed is connecting via wireless.

 

[SweetAndLow]

your iperf results are pretty slow you have some kind of network or hardware problem. Don't use wifi to do perf testing because it will always be a problem and be caution using the thunderbolt to Ethernet adapter for your air. Next is that your cpu is slower and the encryption might be slowing you down. You might want to see if you can find someone else on the forums that has the same cpu and see what they have posted for their performance.

 

[ttsiodras]

Sorry to necrobump - but I've met the same problem and believe I've found a solution.

You can read all about it in a question I posted to UNIX StackExchange (question number 434825 - I don't post a link to avoid being misclassified as a spammer). Anyway, the executive summary is that in my MacBook Air, there was a huge difference of achieved bandwidth in my SSH copies if I switched my SSH config to use "ProxyCommand nc %h %p".

I know this doesn't have any right to impact speed - but it does, tremendously. To see it in action you can also see a recording I made in asciinema (video id: rMpxMHNXIBAaOiBvSjAISbLBU ).

root@mbair ~
# ifconfig wlan0 | grep packets
RX packets 34765 bytes 1917414 (1.8 MiB)
TX packets 97671 bytes 147536170 (140.7 MiB)

root@mbair ~
# dd if=/dev/urandom bs=1M count=40 of=/dev/shm/data ; pv /dev/shm/data | ssh -o "ProxyCommand nc %h %p" root@192.168.1.150 'cat >/dev/null'
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.180914 s, 232 MB/s
40.0MiB 0:00:16 [2.39MiB/s] [2.39MiB/s] [=================================================================================>] 100%

root@mbair ~
# ifconfig wlan0 | grep packets
RX packets 44096 bytes 2419055 (2.3 MiB)
TX packets 126701 bytes 191445527 (182.5 MiB)

root@mbair ~
# dd if=/dev/urandom bs=1M count=40 of=/dev/shm/data ; pv /dev/shm/data | ssh root@192.168.1.150 'cat >/dev/null'
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.182137 s, 230 MB/s
40.0MiB 0:01:12 [ 567KiB/s] [ 567KiB/s] [=================================================================================>] 100%

root@mbair ~
# ifconfig wlan0 | grep packets
RX packets 57787 bytes 3164340 (3.0 MiB)
TX packets 155796 bytes 235439254 (224.5 MiB)

As you can see, we go froom 567KBytes/sec to 2.39MBytes/sec.