Rsync speed gains when dropping SSH and using daemon. Can it be made even better?

[kroko]

Hi,

FreeNAS 9.10.2-U1

Currently I do (did) offsite backups from FreeNAS using rsync over SSH.

I have set it up via cron (to execute a script) not rsync tasks. This is because scripting gives stuff that is not available in rsync tasks GUI anyways, like remote deletion backups shuffling (old habit, pratically unneeded if local snapshotting exists), custom SSH private key just for this backup, mailing on error or success, a.o.

However, now I have a bit new setup both SW and HW.

On SW side I am preperaing that Corral upgrade and requestioning my own historical decisions globally in FreeNAS, restructurising, making changes needed (starting to rework flags, namely, --archive, because till now I had unix permissions everywhere, but on preparing soon-to-come-yet-try-again-migration-to-Corral-that-broke-my-smb-on-first-try:) I have already ACL'ed all needed datasets, set non-trivail stuff). Offt.

HW has changed:
- The offsite backup is now self managed Qnap TS-653A hidden in dark woods, not "existing service that we can only SSH into"
- This offsite box will be 24/7 OpenVPN'ed within the DMZ where FreeNAS is located

Thus I do not need rsync over SSH any more, as the traffic to the offsite Qnap is already in 24/7 VPN tunnel and in (V)LAN I do not need encryption. Daemon?

In FreeNAS I enabled rsync service, no tasks.
In Qnap Backup Station > Rsync Server enabled the service and set some test username offsitersyncduser and pw.

Creted 1000x1MB, 200x5MB, 50x100MB files. Tested in dummy 1GbE LAN w/o activity.

Rsync daemon (without SSH)

time /usr/local/bin/rsync \
--verbose --recursive --archive --compress --protect-args \
--exclude-from=/mnt/vol/offsiteconfigs/offsite-rsyncexclude-master \
--delete --delete-excluded --force --ignore-errors \
--backup --backup-dir="deletebackups-$(date +%Y-%m-%d)-$(date +%s)" \
--password-file=/mnt/vol/offsiteconfigs/.rsyncdpassoffsite \
/mnt/vol/offsitebackuptestfiles/ \
rsync://offsitersyncduser@192.168.48.160:/rsyncfromfreenastest

First sync
Destionation is empty

Code:

sent 7,344,283,608 bytes  received 23,865 bytes  34,724,858.03 bytes/sec
total size is 7,340,032,320  speedup is 1.00

real	3m30.385s
user	3m25.080s
sys	0m2.845s

Second sync
No data changed on Freenas

Code:

sending incremental file list

sent 28,767 bytes  received 21 bytes  57,576.00 bytes/sec
total size is 7,340,032,320  speedup is 254,968.47

real	0m0.207s
user	0m0.005s
sys	0m0.013s

Third sync
All thousand 1MB files on Freenas now have appended "bleh" bytes, thus all those files are modified
All 5MB files deleted (so --delete also kicks in)

Code:

sent 69,070 bytes  received 6,175,948 bytes  462,593.93 bytes/sec
total size is 6,291,461,320  speedup is 1,007.44

real	0m12.761s
user	0m11.310s
sys	0m0.195s

Good ol rsync over SSH

time /usr/local/bin/rsync \
--verbose --recursive --archive --compress --protect-args \
--exclude-from=/mnt/vol/offsiteconfigs/offsite-rsyncexclude-master \
--delete --delete-excluded --force --ignore-errors \
--backup --backup-dir="deletebackups-$(date +%Y-%m-%d)-$(date +%s)" \
-e "/usr/bin/ssh -p 22 -i /mnt/vol/offsiteconfigs/id_rsa-offsitebackup -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" \
/mnt/vol/offsitebackuptestfiles/ \
admin@192.168.48.160:/share/CACHEDEV1_DATA/rsyncfromfreenastest

First sync
Destionation is empty

Code:

sent 7,344,283,584 bytes  received 23,873 bytes  31,453,136.86 bytes/sec
total size is 7,340,032,320  speedup is 1.00

real	3m53.069s
user	4m3.190s
sys	0m9.821s

multiple kex protocol errors: type 7 seq ..., due to not using -x dlag to turn X forwarding off, whatever.

Second sync
No data changed on Freenas

Code:

sending incremental file list

sent 28,747 bytes  received 21 bytes  19,178.67 bytes/sec
total size is 7,340,032,320  speedup is 255,145.73

real	0m0.249s
user	0m0.013s
sys	0m0.014s

Third sync
All thousand 1MB files on Freenas now have appended "bleh" bytes, thus all those files are modified
All 5MB files deleted (so --delete also kicks in)

Code:

sent 70,682 bytes  received 6,175,948 bytes  499,730.40 bytes/sec
total size is 6,291,461,320  speedup is 1,007.18

real	0m12.217s
user	0m11.577s
sys	0m0.337s

The first sync gains considerbaly.
But third is dead on (SSH is even faster in bytes/sec, total execution was faster)

So I can go with daemon method.

But can it be made even better, so that "third case" would show benefits of dropping ssh overhead? I have used rsync many times over years for different purposes, but i never have used daemon so - any tips & tricks?
I kow of --skip-compress, --numeric-ds, but those do not apply to this test (would affect both SSH and daemon). SSH cipher hacks are also not relevant, as I am looking if there is way to make daemon faster, not SSH. Maybe I'm just hitting CPU max capability in both cases, so that's it (first daemon sync is ~35 MB/sec).

The interest is only in speed, not how much CPU is used during backup - he should do its job :)

Thanks!

 

[kroko]

Optimized SSH way for a test - it is only 1.04% slower on first sync.

Probably my expectations about daemon were too high.

It does not seem that qnap box max is hit, not even close.
Max 36% CPU load during transfer via SSH
Max 26% CPU load during transfer via daemon
Celeron N3160 @ 1.60GHz, 4GB RAM, currently for a test 4x4TB RAID5, will be rebuilt as 6x4TB RAID6.

Choosing daemon seems to be more enegry efficient and that's it :)