Tips to make rsync faster

Status
Not open for further replies.

mvcad

Contributor
Joined
Feb 25, 2018
Messages
116
Hi All,
I've been trying to copy a 2TB dataset from a synology NAS to our new FreeNAS server and I've noticed very slow transfer speeds. If I copy a file from windows explorer from a folder in the synology to a folder in FreeNAS I get more than 700 Mbps
with rsync I am only getting about 50!.
upload_2018-9-5_14-31-6.png

My rsync command runs from an automatic task in synology that runs a shell file. the command looks like this:
/usr/syno/bin/rsync -avz --omit-dir-times --delete $SOURCE $USER@$SERVER:$TARGET >> $LOG 2>&1

Without -z parameter I am getting 100 Mbps. Also I tried using ssh transfer with -p but it didn't make much difference.

We bought a new FreeNAS mini XL whit these specs

upload_2018-9-5_14-38-3.png


The data volume is 4 off 4TB WD Red in RidZ2 configuration with mirrored Samsung SSD evos 256 GB as SLOG.
any tips to make it faster? I tried last weekend for 3 days and it did't finish.

Thanks in advance.
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,703
I saw that what you're doing is actually rsync over SSH (so encrypting comms) (due to the colon in the command).

Doing this:
/usr/syno/bin/rsync -avz --omit-dir-times --delete $SOURCE $USER@$SERVER$TARGET >> $LOG 2>&1

may then switch to rsync native (no encrypt of traffic in transit), so I would expect an improvement as a result... how much remains to be seen.
 
Joined
Jul 3, 2015
Messages
926

wblock

Documentation Engineer
Joined
Nov 14, 2014
Messages
1,506
Is this a one-time copy, or something that will happen periodically?

The processor in the Synology might be the bottleneck, having to compress and encrypt all the data. But then, remember that rsync is not aimed at speed but to minimize network traffic. Can you copy the files with Windows, then use rsync to copy any attributes that Windows does not get?

Says "The main limitation is it does not handle remote source or target directory, they must be locally accessible (local disk, nfs/cifs/other mountpoint)."
But that might be an idea: mount the FreeNAS directory on the Synology with whatever SMB client they have, then copy or rsync it as if it was local.
 
Last edited:

mvcad

Contributor
Joined
Feb 25, 2018
Messages
116
Is this a one-time copy, or something that will happen periodically?
Thanks for the suggestions, It will be a one-time only
 

mvcad

Contributor
Joined
Feb 25, 2018
Messages
116
I saw that what you're doing is actually rsync over SSH (so encrypting comms) (due to the colon in the command).

Doing this:
/usr/syno/bin/rsync -avz --omit-dir-times --delete $SOURCE $USER@$SERVER$TARGET >> $LOG 2>&1

may then switch to rsync native (no encrypt of traffic in transit), so I would expect an improvement as a result... how much remains to be seen.
I tried your suggestion but I am getting an error
Code:
ERROR: module is read only
rsync error: syntax or usage error (code 1) at main.c(1034) [Receiver=3.0.9]
rsync error: rsync service is no running (code 43) at io.c(687) [sender=3.0.9]


My script looks like this:
Code:
#!/bin/bash

USER="rsyncbackup"
SERVER="192.168.0.50"
SOURCE="/volume1/Contractors_shared/"
TARGET="/mnt/Vol8TB/test/backup/"
LOG="/volume1/homes/admin/backup.log"

/usr/syno/bin/rsync -av --omit-dir-times --delete $SOURCE $USER@$SERVER:$TARGET >> $LOG 2>&1

Am i doing something wrong?
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,703
The error seems to indicate that the rsync service isn't running on the end where we changed the script.

Do you have it configured on the synology?
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,703
It's also saying that the target is read only. You need to look at the rsync config on FreeNAS.
 

mvcad

Contributor
Joined
Feb 25, 2018
Messages
116
well, if I put the colon back in it works fine.
Do I need a space or a different character between $SERVER and $TARGET?. WITHOUT THE COLON IS THIS:
Code:
/usr/syno/bin/rsync -av --omit-dir-times --delete $SOURCE $USER@$SERVER$TARGET >> $LOG 2>&1
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,703
well, if I put the colon back in it works fine.
Do I need a space or a different character between $SERVER and $TARGET?. WITHOUT THE COLON IS THIS:
Code:
/usr/syno/bin/rsync -av --omit-dir-times --delete $SOURCE $USER@$SERVER$TARGET >> $LOG 2>&1
Putting the colon back returns to using ssh, so is encrypting traffic. If you work out the rsync config on both sides, you can use rsync natively and avoid the encryption.
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,703
You begin the path with a slash, so that's already enough to separate server from the path.
 

mvcad

Contributor
Joined
Feb 25, 2018
Messages
116
If you work out the rsync config on both sides, you can use rsync natively and avoid the encryption.
Thanks, I will have to learn how to do that, hope I can get it to work.
 
Status
Not open for further replies.
Top