rsync from Synology DSM 6.2 to FN 11.1u5

[RegularJoe]

Hi All,

I hope this is the right spot for rsync questions.

I have a synology box that I want to rsync to FreeNAS. I think to make it as fast as possible on a 1gig network I need to not use encryption(ssh) and no compression. I am not finding much on the internet or on synology's web site on how to do this.

Do you know of a guide that shows the process to make rsync work with these non standard NASes?

Thanks,
Joe

 

[dlavigne]

Were you able to figure out how to do this?

 

[RegularJoe]

yes, those silly engineers at sunology have 3 ways to do this with hypervault, and two of them are NOT rsync but make a DB file and such. The "one time" rsync was what I wanted and I got it to work with a dedicated user and an rsync job on FreeNAS 11.2 rc1. It is quite fast and compression is at 24%, takes 3 hours to do all 9tb of data each night. the changed files are so small that they really do not show up on radar. the nightly scrub takes the same time 3 hours and that is with a crappy two set RAIDz2 volume. 12 - 3tb 7200 rpm SAS drives.

the synology's days are limited, I will use it ONLY for time machine backup of Mac computers and it might be that it can't even do that well and will have to be an iSCSI server for a Mac server that does the time machine backups.

the real killer on synology is they do not have a push button convert ext4/md to btrfs.... if they did it might not be trustworthy.

FreeNAS rocks, I can't wait for some managed services in the future!

 

[Spearfoot]

Hi All,

I hope this is the right spot for rsync questions.

I have a synology box that I want to rsync to FreeNAS. I think to make it as fast as possible on a 1gig network I need to not use encryption(ssh) and no compression. I am not finding much on the internet or on synology's web site on how to do this.

Do you know of a guide that shows the process to make rsync work with these non standard NASes?

Thanks,
Joe

I used to use a Synology diskstation and had the same desire, i.e., to disable compression and encryption. Never found a completely satisfactory solution... but this script I wrote (rsync-invoke.sh) gave usable results via ssh by disabling compressing and using a low-cost encryption algorithm (arcfour).

You're free to use it 'as is' or as a basis for further tweaking, just be sure to read and understand it beforehand. No warranty implied, use at your own risk, etc.

Search the forum for 'rsync' and my username and you should find some additional posts on the subject.

Good luck!

Code:

#!/bin/sh
#######################################################################
# A wrapper for invoking rsync to copy a source dataset to a destination
# dataset with options that are 'known to work' with FreeNAS/FreeBSD
# and Windows datasets. This means we have to avoid the '-p' (preserve
# permissions) option in all of its forms, per bug report 7713:
#
# https://bugs.freenas.org/issues/7713
#
# Command-line parameters:
#   1: Source dataset	   R_SRC	 (local system)  /mnt/tank/sharename/
#   2: Destination dataset  R_DEST	(local system)  /volume1/sharename
#   3: Log file			 R_LOGFILE (local system)  /mnt/tank/sysadmin/log/push-log.log
#
# R_SRC or R_DEST may include user ID and hostname specifier
#
# Example usage, where 'boomer' is a FreeNAS server and 'bertrand' is
# a Synology Diskstation. Note that you should include a trailing slash
# only on the source dataset specifier:
#
# rsync-invoke.sh root@bertrand:/volume1/devtools/ /mnt/tank/devtools /mnt/tank/sysadmin/log/pull-from-bertrand.log
# rsync-invoke.sh /mnt/tank/devtools/ root@bertrand:/volume1/devtools /mnt/tank/sysadmin/log/push-to-bertrand.log
#
# Assumes SSH has been enabled and configured between the source
# and destination hosts.
#
# Invokes rsync with these SSH options to optimize transfer speed:
#
#   "-e ssh -T -c arcfour -o Compression=no -x"
#
# You may need to remove this altogether, or modify the encryption
# specifier (-c arcfour) to use a scheme available on your system.
#
# !!! WARNING !!!
# This script deletes files on the destination that don't exist
# on the source! Edit R_OPTIONS below and remove '--delete-during'
# and '--inplace' if you don't want this behavior!'
#					'
# Tested with:
#   FreeNAS 9.3-STABLE
#   FreeNAS 9.10-STABLE
#   Synology DSM 5.x (as destination only)
#######################################################################

if [ $# -ne 3 ]
then
  echo "Error: not enough arguments!"
  echo "Usage is: $0 r_src r_dest r_logfile"
  exit 2
fi

R_SRC=$1
R_DEST=$2
R_LOGFILE=$3

# Options:
#   -r  recurse into directories
#   -l  copy symlinks as symlinks
#   -t  preserve modification times
#   -g  preserve group
#   -o  preserve owner
#   -D  preserve device and special files
#   -h  human readable progress
#   -v  increase verbosity

#   --delete-during   receiver deletes during the transfer
#   --inplace		 write updated data directly to destination file
#   --log-file		specify log file

R_OPTIONS="-rltgoDhv --delete-during --inplace --progress --log-file="${R_LOGFILE}

# Files to exclude:
#   .windows	  FreeNAS ACL settings (?)
#   vmware.log	VMware virtual machine log files
#   vmware-*.log
#   @eaDir/	   Synology extended attributes (?)
#   @eaDir
#   Thumbs.db	 Windows system files

# R_EXCLUDE="--exclude .windows --exclude vmware.log --exclude vmware-*.log --exclude @eaDir/ --exclude @eaDir --exclude Thumbs.db"
R_EXCLUDE="--exclude vmware.log --exclude vmware-*.log --exclude @eaDir/ --exclude @eaDir --exclude Thumbs.db"

echo "$(date) Copy" ${R_SRC} "to" ${R_DEST} >> ${R_LOGFILE}
rsync ${R_OPTIONS} ${R_EXCLUDE} -e "ssh -T -c none -o Compression=no -x" ${R_SRC} ${R_DEST}
echo "$(date) Copy completed" >> ${R_LOGFILE}
exit

 

[Sasquatch]

Hi. I've been trying to set Synology hyper backup to freenas 11.2 using rsync.
achieved so far:
unencrypted transfer using dedicated user.
encrypted transfer only using Freenas's root user and password.
logs on both ends show nothing useful.
what may I be doing wrong? my guess are permissions, but no idea which. Dataset permissions for backup user are set correctly.

rsyncd.conf needed to be copied to backup user home folder.
but it still fails, from daemon.log:

Code:

Jan  3 17:55:44 Sasquatchv rsyncd[56919]: connect from host-92-
Jan  3 17:55:44 Sasquatchv rsyncd[56919]: rsync allowed access on module DMS_backup from host-92-
Jan  3 17:55:44 Sasquatchv rsyncd[56919]: rsync: setgid 1000 failed: Operation not permitted (1)
.
.
Jan  3 18:01:25 Sasquatchv rsyncd[58224]: connect from host-92-
Jan  3 18:01:25 Sasquatchv rsyncd[58224]: rsync allowed access on module DMS_backup from host-92
Jan  3 18:01:25 Sasquatchv rsyncd[58224]: rsync: setgroups failed: Operation not permitted (1)

 

[RegularJoe]

Sasquatch,

I am working on it today. I have it working on the same subnet, synology > classic rsync with files and synology hyper vault DB > FreeNAS rsync. I have to see what exactly needs to be done as the last time I did this I may have made many un-necessary changes to FreeNAS to get this silly thing to work.

I am having issues with my Synology where a backup is taking so much cpu time and volume resource IO that AFP users running Adobe products are getting kicked off the shares. I just replaced 2 questionable drives so I am hoping this resolves the crashing so I can continue moving forward at replacing the Synology.

Thanks,
Joe

 

[RegularJoe]

Hi Sasquatch,

I have it working and figured out what the issue was. You need to use the New GUI to make the rsync module for Synology to connect to. The settings in the old UI are rubbish.

Thanks,
Jjoe

 

[Sasquatch]

Hi Sasquatch,

I have it working and figured out what the issue was. You need to use the New GUI to make the rsync module for Synology to connect to. The settings in the old UI are rubbish.

Thanks,
Jjoe

Thanks Joe.
I was using new gui from the begining. Never got it working with encryption and user other than root. Issue seems to be synology's greed for power, there is no need to change groups to perform backup, but synology/hyperbackup insist on doing it.
I did SSh to synology and invoked encrypted rsync manually with backup user, it worked.
hyperbackup backup won't work without root priviliges on freenas.
Gave up and now waiting for synology box to die so we can replace it with freenas.

 

[RegularJoe]

I am backing up to the Hypervault database file so that the other techs can do multiple file restores if they have to on the synology. The other backup will be a typical rsync file/folder backup without encryption as the synology comes to it's knees when backups are done and it is so bad a single Mac user on an AFP volume gets kicked off at times.

I too am moving off the synology, the pervious person did not use any directory so all the user accounts are local and file ownership is a mess as he used a LOT of different shares rather than learning how to use groups.

The consistency check on MDRAID6 takes 65 hours where the same data on a two stripe RAIDz2 scrub takes 2.25 hours.

I plan on putting some junk SATA refurbished 3-6TB SATA drives in the synology and using it ONLY for time machine backup and I am still thinking it will run out of steam and space doing stripes of mirrors.

 

[RegularJoe]

I think the groups change if you do not have the same users/groups on the destination and may have to have matching UID's as well.