Poor network performance between FreeNAS and QNAP using scp and rsync

[slayer99199]

FreeNAS config:
FreeNAS-11.0-U4 (54848d13b)
5x4TB HGST DeskStar NAS 7200 RPM
ASRock C236 WSI
16GB ECC memory

Old NAS:
QNAP TS-439

Networking:
FreeNAS - 1 x 1GB connection (I currently don't have a GB switch that supports LACP) on the FreeNAS.
QNAP - 2 x 1GB connections load-balanced (QNAP doesn't require LACP)
They're both connected directly to the switch. No Jumbo frames. Same subnet.

Boot: FreeNAS is currently installed on a 120GB SSD and backed up to a USB drive nightly through cron.

I'm stumped here. I've been battling this for a couple days and I feel like I'm missing something obvious. I'm in the process of copying my media from my QNAP to my FreeNAS. I'm experiencing very slow copies between the devices. I've tested both rsync and scp using the -rv commands and the speed is around 13-14mpbs.

CPU on both devices does not seem to be strained in any way.

I have a Windows network share to my QNAP and I set one up for FreeNAS...and ran robocopy on my windows machine between the 2 using the UNC paths and got 40mpbs. I know my QNAP is only capable of 60-70mbps through Windows (have tested previously).

I do have 2 jails configured. 1 for Plex and the other for Emby...but they're not being used (not configured in the house yet).

I have root SSH on both devices so I can provide whatever anyone needs to assist.

 

[SweetAndLow]

iperf is a great way to test network speed. rsync is going to be slow no matter what. SCP will max out a gigabit connection. You can also use -c none to set the cipher type to none so there is no encryption. I suspect it could be that your qnap system is just slow especially since it can only do 60mbps over smb.

Also don't ever plug two nic's from the same machine into the same network. You do not get more performance by using more nic's. It just makes it complicated and messed up and you will never know unless you know how to look.

 

[slayer99199]

iperf is a great way to test netowrk speed. rsync is going to be slow no matter what. SCP will max out a gigabit connection. You can also use -c none to set the cipher type to none so there is no encryption. I suspect it could be that your qnap system is just slow especially since it can only do 60mbps over smb.

Also don't ever plug two nic's from the same machine into the same network. You do not get more performance by using more nic's. It just makes it complicated and messed up and you will never know unless you know how to look.

Unfortunately, it doesn't look like I'm going to be able to get iperf installed on the QNAP. It's running Ubuntu 4.1.2...no apt-get.

SCP copy with the -c none made a negligible improvement.

I also broke the port trunk and killed 1 of the ports at the switch on the QNAP so it's running on one NIC. No improvement in speed.

Worst case, I run a robocopy to get all my data over to the FreeNAS. My concern is that it's a larger issue that will come back to haunt me.

 

[slayer99199]

SweetAndLow. You got me thinking. I haven't tested the NAS performance of my QNAP in years. I just replaced the drives in April (after 5 years one failed and another one had SMART errors...yeah, that was a scary time because it happened within a week). I have kept the QNAP software updated, but the CPU and memory on the QNAP aren't very robust (in addition to being 6 years old).

So, I ran some read/write NAS tests using the NAS Performance Tester (which really only works on Windows).

Here's FreeNAS:
NAS performance tester 1.7 http://www.808.dk/?nastester
Running warmup...
Running a 400MB file write on W: 5 times...
Iteration 1: 55.15 MB/sec
Iteration 2: 55.82 MB/sec
Iteration 3: 55.62 MB/sec
Iteration 4: 54.73 MB/sec
Iteration 5: 54.72 MB/sec
-----------------------------
Average (W): 55.20 MB/sec
-----------------------------
Running a 400MB file read on W: 5 times...
Iteration 1: 110.57 MB/sec
Iteration 2: 118.34 MB/sec
Iteration 3: 118.34 MB/sec
Iteration 4: 97.63 MB/sec
Iteration 5: 100.94 MB/sec
-----------------------------
Average (R): 109.16 MB/sec
-----------------------------

Here's the QNAP:
Running a 400MB file write on Y: 5 times...
Iteration 1: 39.46 MB/sec
Iteration 2: 40.00 MB/sec
Iteration 3: 40.11 MB/sec
Iteration 4: 39.55 MB/sec
Iteration 5: 38.54 MB/sec
-----------------------------
Average (W): 39.53 MB/sec
-----------------------------
Running a 400MB file read on Y: 5 times...
Iteration 1: 93.62 MB/sec
Iteration 2: 111.17 MB/sec
Iteration 3: 111.86 MB/sec
Iteration 4: 113.72 MB/sec
Iteration 5: 89.80 MB/sec
-----------------------------
Average (R): 104.03 MB/sec
-----------------------------

Looks like the best I can do is 40MB/sec on the QNAP over SMB.

So the question is...are my results "normal" for my FreeNAS build?

 

[SweetAndLow]

Your freenas write speeds are not good. You should easily be able to max out a 1gig connection when doing streaming writes. You should run a local dd test on the FreeNAS box to test write performance.

Write test:
------------
1. create a test dataset with compression disabled
2. dd if=/dev/zero of=/mnt/tank/dataset/100g.dat bs=1M count=100000

Read test:
-----------
2. dd of=/dev/null if=/mnt/tank/dataset/100g.dat bs=1M count=100000

Then test using iperf3 on windows and iperf on FreeNAS, run it both directions so your FreeNAS as a server then FreeNAS as a client.

these tests will eliminate the FreeNAS box from the bottleneck and then we can look at your qnap.

 

[slayer99199]

Your freenas write speeds are not good. You should easily be able to max out a 1gig connection when doing streaming writes. You should run a local dd test on the FreeNAS box to test write performance.

Write test:
------------
1. create a test dataset with compression disabled
2. dd if=/dev/zero of=/mnt/tank/dataset/100g.dat bs=1M count=100000

Read test:
-----------
2. dd of=/dev/null if=/mnt/tank/dataset/100g.dat bs=1M count=100000

Then test using iperf3 on windows and iperf on FreeNAS, run it both directions so your FreeNAS as a server then FreeNAS as a client.

these tests will eliminate the FreeNAS box from the bottleneck and then we can look at your qnap.

Write test:
100000+0 records in
100000+0 records out
104857600000 bytes transferred in 16.699394 secs (6279126243 bytes/sec)

Read Test:
100000+0 records in
100000+0 records out
104857600000 bytes transferred in 6.771651 secs (15484790164 bytes/sec)

Working on the iperf test.

 

[SweetAndLow]

You didn't follow directions. you have to use a dataset with compression turned off.

 

[slayer99199]

You didn't follow directions. you have to use a dataset with compression turned off.

I'm an idiot. Sorry.

Write:
100000+0 records in
100000+0 records out
104857600000 bytes transferred in 259.446360 secs (404159071 bytes/sec)

Read:
100000+0 records in
100000+0 records out
104857600000 bytes transferred in 229.774380 secs (456350268 bytes/sec)

 

[SweetAndLow]

I'm an idiot. Sorry.

Write:
100000+0 records in
100000+0 records out
104857600000 bytes transferred in 259.446360 secs (404159071 bytes/sec)

Read:
100000+0 records in
100000+0 records out
104857600000 bytes transferred in 229.774380 secs (456350268 bytes/sec)

That's better! those speeds look to be about right. I suspect iperf is also going to be ok so that means your qnap is just really slow when using ssh. Did you try having freenas run the scp command and compare that to qnap run the scp command? Might be faster one way?

 

[slayer99199]

That's better! those speeds look to be about right. I suspect iperf is also going to be ok so that means your qnap is just really slow when using ssh. Did you try having freenas run the scp command and compare that to qnap run the scp command? Might be faster one way?

That's a relief. Here's the iperf (windows in server mode, FreeNAS in client)
Client connecting to 10.0.100.99, TCP port 5201
TCP window size: 0.03 MByte (default)
------------------------------------------------------------
[ 3] local 10.0.100.40 port 31660 connected with 10.0.100.99 port 5201
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-79.6 sec 17592186044416 MBytes 1854669895757 Mbits/sec

Client connecting to 10.0.100.99, TCP port 5201
TCP window size: 0.13 MByte (WARNING: requested 0.12 MByte)
------------------------------------------------------------
[ 3] local 10.0.100.40 port 57371 connected with 10.0.100.99 port 5201
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 17592186044376 MBytes 14751136351780 Mbits/sec

------------------------------------------------------------
Client connecting to 10.0.100.99, TCP port 5201
TCP window size: 1.00 MByte (WARNING: requested 1.00 MByte)
------------------------------------------------------------
[ 3] local 10.0.100.40 port 47956 connected with 10.0.100.99 port 5201
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-79.6 sec 1.00 MBytes 0.11 Mbits/sec



Yeah, I ran scp both ways...it was still pretty slow (marginally faster from FreeNAS pulling from QNAP).

Honestly, my QNAP is perfectly fine as a mass storage device...but it's not what I call speedy. Nor can it do media transcoding (Plex and Emby)...hence the reason I built FreeNAS.

I plan on testing this for a good month or so before scrubbing the data off my QNAP and shutting it down. Thanks for the help!

 

[SweetAndLow]

Those iperf numbers are super strange. How did you run them?