Easy Rsync task creation?

[Tweebeenvis]

Hi Guys, I've been a long time user of FreeNas and finally decided to join the form today. Hello :D.

I was wondering if there is an easy way to add Rsync task entries. I have googled considerably but cannot seem to find anything except this https://forums.freenas.org/index.php?threads/rsync-config-file-resets-rsyncd-conf.13135/ (which indicates that manually editing the config file is a big no-no).

I am currently "rsyncing over ssh" a folder containing user shares to another FreeNas system. I am now tasked with syncing only certain user folders (sub-directories of that folder) to a third FreeNas system, problem being there are about 150 odd entries that will need to be set up. Would appreciate any suggestions on how to approach this.

Thanks to the AWESOME devs for this amazing product.

 

[depasseg]

Welcome! You could try the API:
http://api.freenas.org/resources/services.html#id18

 

[Spearfoot]

Hi Guys, I've been a long time user of FreeNas and finally decided to join the form today. Hello :D.

I was wondering if there is an easy way to add Rsync task entries. I have googled considerably but cannot seem to find anything except this https://forums.freenas.org/index.php?threads/rsync-config-file-resets-rsyncd-conf.13135/ (which indicates that manually editing the config file is a big no-no).

I am currently "rsyncing over ssh" a folder containing user shares to another FreeNas system. I am now tasked with syncing only certain user folders (sub-directories of that folder) to a third FreeNas system, problem being there are about 150 odd entries that will need to be set up. Would appreciate any suggestions on how to approach this.

Thanks to the AWESOME devs for this amazing product.

Hello, and welcome to the forum!

You don't want 150+ rsync tasks; that would be a management nightmare.

Here's what I do to accomplish much the same thing you're wanting to do. Write three scripts:

1. A basic rsync-invocation script
2. A 'copy to target host' script which runs the first script
3. A top-level script which runs the second script to synchronize the user datasets

Once you tweak the scripts and get everything working, create a Cron Job to run the third script at whatever interval seems appropriate. To add or delete user datasets for synchronization you'd only need to edit the third script instead of slogging through 150+ Rsync Task entries in the GUI.

I have written an rsync-invocation script which avoids certain permissions problems when used on Windows datasets. It's available here on pastebin. Be sure to read it carefully before use, as you may want to modify some of the options. In particular, I use arcfour SSH encryption for increased speed and you will have to enable this encryption scheme on your target server, or just change the encryption scheme as documented in the script. Also, my script uses the --delete-during option, which deletes any files on the target dataset that don't exist on the source; this may or may not be your desired behavior. I name this script rsync-invoke.sh and store it (and all of my other administrative scripts) in a dataset created in /mnt/tank/sysadmin, which you will need to create on your system.

Your 'copy to target host' script is just a wrapper for the rsync invocation script and requires simply 2 parameters: the source and target dataset names. A third, optional parameter lets you specify a log filename if desired, otherwise the script will write log information to a log file based on the target host name and located in directory /mnt/tank/sysadmin/log (which you must also create). The script will look something like this, for user root on a target host named foohost:

Code:

#!/bin/bash

if [ $# -lt 2 ]
then
  echo "Error: not enough arguments!"
  echo "Usage is: $0 source target logfile"
  exit 2
fi

if [ $# -gt 3 ]
then
  echo "Error: too many arguments!"
  echo "Usage is: $0 source target logfile"
  exit 2
fi

R_USERID=root
R_HOST=foohost
R_SRC=/mnt/tank/$1/
R_DEST=/mnt/tank/$2

if [ $# -gt 2 ]
then
  R_LOGFILE="/mnt/tank/sysadmin/log/"$3
else
  R_LOGFILE="/mnt/tank/sysadmin/log/rsync-push-to-$R_HOST.log"
fi

/mnt/tank/sysadmin/rsync-invoke.sh $R_SRC $R_USERID@$R_HOST:$R_DEST $R_LOGFILE

exit

We will name this script rsync-push-to-foohost.sh.

Finally(!), your third and simplest script, the one you will invoke as a Chron Job, will run the 'copy to target host' script ( rsync-push-to-foohost.sh) for each user dataset. For demonstration purposes, I assume your user directories are in dataset /mnt/tank/users and are named 'john', 'jane', 'bill', 'betty', etc., on both the source and target systems. In this case, the script would look like this:

Code:

#!/bin/bash

cd /mnt/tank/sysadmin

# Push user datasets to server 'foohost':

./rsync-push-to-foohost.sh users/john users/john
./rsync-push-to-foohost.sh users/jane users/jane
./rsync-push-to-foohost.sh users/bill users/bill
./rsync-push-to-foohost.sh users/betty users/betty

Hope this helps... good luck!

 

[Tweebeenvis]

Thanks guys. Haha was looking for a noob friendly way but appreciate the feedback. Will try these when I get some time. :)