(408) 943-4100               V   Commercial Support Toggle between Light and Dark mode

Rsync Tasks

  15 minute read.

Last Modified 2021-07-30 09:07 EDT

Data often needs to be copied to another system for backup or when migrating to a new system. A fast and secure way of doing this is by using rsync. These instructions assume that TrueNAS systems are being used for both sides of the rsync task: host and remote.

Basic Requirements

Rysnc requires a dataset with the needed data on the host or remote system. Rsync provides the ability to either push or pull data. When using rsync to push, data is copied from a host system to a remote system. When using rsync to pull, data is pulled from a remote system and put on the host system.

The remote system must have the rsync service activated. Additional requirements are listed further down for either module or SSH rsync tasks.

Creating an Rsync Task

Go to Tasks > Rsync Tasks and click ADD. There are two primary Rsync Modes: Module and SSH. The requirements for each mode are different. Refer to the related tab for your desired Rsync Mode.

Module Requirements

Before you create an rsync task on the host system, you must create a module on the remote system. When TrueNAS is the remote system, create a module by going to Services and clicking for the rsync service. Click the Rsync Module tab, then click ADD. Specific creation instructions are farther down, in the Rsync Service section.

Process

Log in to the host system interface, go to Tasks > Rsync Tasks, and click ADD.

TasksRsyncTasksAddModeModule

Select the source dataset to use with the rsync task and a user account to run the rsync task. Choose a direction for the rsync task.

Select a schedule for the rsync task. If you need a custom schedule, select Custom.

Tasks Advanced Scheduler

Choosing a preset fills in the rest of the fields. To customize a schedule, enter crontab values for the Minutes/Hours/Days.

These fields accept standard cron values. The simplest option is to enter a single number in the field. The task runs when the time value matches that number. For example, entering 10 means that the job runs when the time is ten minutes past the hour.

An asterisk (*) means “match all values”.

Specific time ranges are set by entering hyphenated number values. For example, entering 30-35 in the Minutes field sets the task to run at minutes 30, 31, 32, 33, 34, and 35.

Lists of values can also be entered. Enter individual values separated by a comma (,). For example, entering 1,14 in the Hours field means the task runs at 1:00 AM (0100) and 2:00 PM (1400).

A slash (/) designates a step value. For example, while entering * in Days means the task runs every day of the month, */2 means the task runs every other day.

Combining all the above examples together creates a schedule running a task each minute from 1:30-1:35 AM and 2:30-2:35 PM every other day.

There is an option to select which Months the task runs. Leaving each month unset is the same as selecting every month.

The Days of Week schedules the task to run on specific days. This is in addition to any listed Days. For example, entering 1 in Days and setting Wed for Days of Week creates a schedule that starts a task on the first day of the month and every Wednesday of the month.

The Schedule Preview shows when the current settings mean the task runs.

Examples of CRON syntax

SyntaxMeaningExamples
*Every item.* (minutes) = every minute of the hour.
* (days) = every day.
*/NEvery Nth item.*/15 (minutes) = every 15th minute of the hour (every quarter hour).
*/3 (days) = every 3rd day.
*/3 (months) = every 3rd month.
Comma and hyphen/dashEach stated item (comma)
Each item in a range (hyphen/dash).
1,31 (minutes) = on the 1st and 31st minute of the hour.
1-3,31 (minutes) = on the 1st to 3rd minutes inclusive, and the 31st minute, of the hour.
mon-fri (days) = every Monday to Friday inclusive (every weekday).
mar,jun,sep,dec (months) = every March, June, September, December.

Days can be specified as days of month, or days of week.

With these options, flexible schedules can be created similar to these examples:

Desired scheduleValues to enter
3 times a day (at midnight, 08:00 and 16:00)months=*; days=*; hours=0/8 or 0,8,16; minutes=0
(Meaning: every day of every month, when hours=0/8/16 and minutes=0)
Every Monday, Wednesday and Friday, at 8.30 pmmonths=*; days=mon,wed,fri; hours=20; minutes=30
1st and 15th day of the month, during October to June, at 00:01 ammonths=oct-dec,jan-jun; days=1,15; hours=0; minutes=1
Every 15 minutes during the working week, which is 8am - 7pm (08:00 - 19:00) Monday to FridayNote that this requires two tasks to achieve:
(1) months=*; days=mon-fri; hours=8-18; minutes=*/15
(2) months=*; days=mon-fri; hours=19; minutes=0
We need the second scheduled item, to execute at 19:00, otherwise we would stop at 18:45. Another workaround would be to stop at 18:45 or 19:45 rather than 19:00.

Next, enter the Remote Host IP address or hostname. Use the format username@remote_host when the username differs on the remote host. Select Module in the Rsync Mode dropdown. Enter the Remote Module Name exactly as it appears on the remote system.

Configure the remaining options according to your specific needs.

Source

NameDescription
PathBrowse to the path to be copied. FreeBSD file path limits apply. Other operating systems can have different limits which might affect how they can be used as sources or destinations.
UserSelect the user to run the rsync task. The user selected must have permissions to write to the specified directory on the remote host.
DirectionDirect the flow of data to the remote host. During a push, the dataset transfers to the remote module. During a pull, the dataset stores files from the remote system.
DescriptionEnter a description of the rsync task.

Schedule

NameDescription
ScheduleSelect a schedule preset or choose Custom to open the advanced scheduler.
RecursiveSet to include all subdirectories of the specified directory. When unset, only the specified directory is included.

Remote

NameDescription
Remote HostEnter the IP address or hostname of the remote system that will store the copy. Use the format username@remote_host if the username differs on the remote host.
Rsync ModeChoose to either use a custom-defined remote module of the rsync server or to use an SSH configuration for the rsync task.

More Options

NameDescription
TimesSet to preserve modification times of files.
CompressSet to reduce the size of data to transmit. Recommended for slow connections.
ArchiveWhen set, rsync is run recursively, preserving symlinks, permissions, modification times, group, and special files. When run as root, owner, device files, and special files are also preserved. Equivalent to passing the flags -rlptgoD to rsync.
DeleteDelete files in the destination directory that do not exist in the source directory.
QuietSet to suppress informational messages from the remote server.
Preserve PermissionsSet to preserve original file permissions. This is useful when the user is set to root.
Preserve Extended AttributesExtended attributes are preserved, but must be supported by both systems.
Delay UpdatesSet to save the temporary file from each updated file to a holding directory until the end of the transfer when all transferred files are renamed into place.
Auxiliary ParametersAdditional rsync(1) options to include. Separate entries by pressing Enter. Note: The “” character must be escaped with a backslash (\.txt) or used inside single quotes ('*.txt').
EnabledEnable this rsync task. Unset to disable this rsync task without deleting it.

The Module mode adds one field to the Remote section:

Remote Module Name : At least one module must be defined in rsyncd.conf(5) of the rsync server or in the Rsync Modules of another system.

Unsetting Enabled disables the task schedule. You can still save the rsync task and run it manually.

SSH Requirements

The remote system must have SSH enabled. To enable SSH in TrueNAS, go to Services and toggle SSH.

The host system needs an established SSH connection to the remote for the rsync task. To create the connection, go to System > SSH Connections and click Add. Configure a Semi-automatic connection and set Private Key to Generate New.

To use a command line, go to the Shell on the host system. When the rsync task is managed by a TrueNAS account other than root, enter su - {USERNAME}, where {USERNAME} is the TrueNAS user account that runs the rsync task. Enter ssh-keygen -t rsa to create the key pair. When prompted for a password, press Enter without setting a password (a password breaks the automated task). Here is an example of running the command:

truenas# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:NZMgbuPvTHeEqi3SA/U5wW8un6AWrx8ZsRQdbJJHmR4 tester@truenas.local
The key's randomart image is:
+---[RSA 2048]----+
|      . o=o+     |
|     . .ooE.     |
|      +.o==.     |
|     o.oo+.+     |
|     ...S+. .    |
|    . ..++o.     |
|     o oB+. .    |
|    . =Bo+.o     |
|     o+==oo      |
+----[SHA256]-----+

The default public key location is ~/.ssh/id_rsa.pub. Enter cat ~/.ssh/id_rsa.pub to see the key and copy the file contents. Copy it to the corresponding user account on the remote system in Accounts > Users. Click EDIT and paste the key in SSH Public Key.

Next, copy the host key from the remote system to the host system user’s .ssh/known_hosts directory, using ssh-keyscan. On the host system, open the Shell and enter ssh-keyscan -t rsa {remoteIPaddress} >> {userknown_hostsDir} where {remoteIPaddress} is the remote system IP address and {userknown_hostsDir} is the known_hosts directory on the host system. Example: ssh-keyscan -t rsa 192.168.2.6 >> /root/.ssh/known_hosts.

Process

Go to Tasks > Rsync Tasks and click ADD.

TasksRsyncTasksAddModeSSH

Configure the SSH settings first by selecting SSH in the Rsync Mode dropdown and entering the Port number and Remote Path.

Next, define the Source dataset to use for the rsync task and select a User account. The User must be identical to the SSH Connection Username.

Choose a direction for the rsync task, either Push or Pull and then define the task Schedule. If you need a custom schedule, select Custom.

Tasks Advanced Scheduler

Choosing a preset fills in the rest of the fields. To customize a schedule, enter crontab values for the Minutes/Hours/Days.

These fields accept standard cron values. The simplest option is to enter a single number in the field. The task runs when the time value matches that number. For example, entering 10 means that the job runs when the time is ten minutes past the hour.

An asterisk (*) means “match all values”.

Specific time ranges are set by entering hyphenated number values. For example, entering 30-35 in the Minutes field sets the task to run at minutes 30, 31, 32, 33, 34, and 35.

Lists of values can also be entered. Enter individual values separated by a comma (,). For example, entering 1,14 in the Hours field means the task runs at 1:00 AM (0100) and 2:00 PM (1400).

A slash (/) designates a step value. For example, while entering * in Days means the task runs every day of the month, */2 means the task runs every other day.

Combining all the above examples together creates a schedule running a task each minute from 1:30-1:35 AM and 2:30-2:35 PM every other day.

There is an option to select which Months the task runs. Leaving each month unset is the same as selecting every month.

The Days of Week schedules the task to run on specific days. This is in addition to any listed Days. For example, entering 1 in Days and setting Wed for Days of Week creates a schedule that starts a task on the first day of the month and every Wednesday of the month.

The Schedule Preview shows when the current settings mean the task runs.

Examples of CRON syntax

SyntaxMeaningExamples
*Every item.* (minutes) = every minute of the hour.
* (days) = every day.
*/NEvery Nth item.*/15 (minutes) = every 15th minute of the hour (every quarter hour).
*/3 (days) = every 3rd day.
*/3 (months) = every 3rd month.
Comma and hyphen/dashEach stated item (comma)
Each item in a range (hyphen/dash).
1,31 (minutes) = on the 1st and 31st minute of the hour.
1-3,31 (minutes) = on the 1st to 3rd minutes inclusive, and the 31st minute, of the hour.
mon-fri (days) = every Monday to Friday inclusive (every weekday).
mar,jun,sep,dec (months) = every March, June, September, December.

Days can be specified as days of month, or days of week.

With these options, flexible schedules can be created similar to these examples:

Desired scheduleValues to enter
3 times a day (at midnight, 08:00 and 16:00)months=*; days=*; hours=0/8 or 0,8,16; minutes=0
(Meaning: every day of every month, when hours=0/8/16 and minutes=0)
Every Monday, Wednesday and Friday, at 8.30 pmmonths=*; days=mon,wed,fri; hours=20; minutes=30
1st and 15th day of the month, during October to June, at 00:01 ammonths=oct-dec,jan-jun; days=1,15; hours=0; minutes=1
Every 15 minutes during the working week, which is 8am - 7pm (08:00 - 19:00) Monday to FridayNote that this requires two tasks to achieve:
(1) months=*; days=mon-fri; hours=8-18; minutes=*/15
(2) months=*; days=mon-fri; hours=19; minutes=0
We need the second scheduled item, to execute at 19:00, otherwise we would stop at 18:45. Another workaround would be to stop at 18:45 or 19:45 rather than 19:00.

Next, enter the remote host IP address or hostname. Use the format username@remote_host if the username differs on the remotehost. Configure the remaining options according to your specific needs.

Source

NameDescription
PathBrowse to the path to be copied. FreeBSD file path limits apply. Other operating systems can have different limits which might affect how they can be used as sources or destinations.
UserSelect the user to run the rsync task. The user selected must have permissions to write to the specified directory on the remote host.
DirectionDirect the flow of data to the remote host. During a push, the dataset transfers to the remote module. During a pull, the dataset stores files from the remote system.
DescriptionEnter a description of the rsync task.

Schedule

NameDescription
ScheduleSelect a schedule preset or choose Custom to open the advanced scheduler.
RecursiveSet to include all subdirectories of the specified directory. When unset, only the specified directory is included.

Remote

NameDescription
Remote HostEnter the IP address or hostname of the remote system that will store the copy. Use the format username@remote_host if the username differs on the remote host.
Rsync ModeChoose to either use a custom-defined remote module of the rsync server or to use an SSH configuration for the rsync task.

More Options

NameDescription
TimesSet to preserve modification times of files.
CompressSet to reduce the size of data to transmit. Recommended for slow connections.
ArchiveWhen set, rsync is run recursively, preserving symlinks, permissions, modification times, group, and special files. When run as root, owner, device files, and special files are also preserved. Equivalent to passing the flags -rlptgoD to rsync.
DeleteDelete files in the destination directory that do not exist in the source directory.
QuietSet to suppress informational messages from the remote server.
Preserve PermissionsSet to preserve original file permissions. This is useful when the user is set to root.
Preserve Extended AttributesExtended attributes are preserved, but must be supported by both systems.
Delay UpdatesSet to save the temporary file from each updated file to a holding directory until the end of the transfer when all transferred files are renamed into place.
Auxiliary ParametersAdditional rsync(1) options to include. Separate entries by pressing Enter. Note: The “” character must be escaped with a backslash (\.txt) or used inside single quotes ('*.txt').
EnabledEnable this rsync task. Unset to disable this rsync task without deleting it.

Additional options for the SSH Rsync Mode:

  • Remote SSH Port : Enter the SSH port number of the remote system. By default, 22 is reserved in TrueNAS.
  • Remote Path : Browse to the existing path on the remote host to sync with. Maximum path length is 255 characters.
  • Validate Remote Path : Set to automatically create the defined Remote Path when it does not exist.

Unsetting Enabled disables the task schedule without deleting the configuration. You can still run the rsync task by going to Tasks > Rsync Tasks and clicking , then RUN NOW.

Rsync Service and Modules

The rsync task does not work when the related system service is not turned on. To turn the rsync service on, go to Services and toggle rsync. To activate the service whenever TrueNAS boots, set Start Automatically.

Click the to configure the service. There are two sections for rsync configuration: basic Configure options and Rsync Module creation and management.

ServicesRsyncConfigure

NameDescription
TCP Portrsyncd listens on this port.
Auxiliary ParametersEnter any additional parameters from rsyncd.conf(5).

Use the default settings unless a specific change is required. Don’t forget to click SAVE after changing any settings.

All created modules are listed here. To create a new module, click ADD.

ServicesRsyncModuleAdd

General

NameDescription
NameModule name that matches the name requested by the rsync client.
PathBrowse to the pool or dataset to store received data.
CommentDescribe this module.
EnabledActivate this module for use with Rsync. Unset this field to deactivate the module without completely removing it.

Access

NameDescription
Access ModeChoose permissions for this rsync module.
Max ConnectionsMaximum number of connections to this module. 0 is unlimited.
UserTrueNAS user account that runs the rsync command during file transfers to and from this module.
GroupTrueNAS group account that runs the rsync command during file transfers to and from this module.
Hosts AllowFrom rsyncd.conf(5). A list of patterns to match with the hostname and IP address of a connecting client. The connection is rejected if no patterns match. Separate entries by pressing Enter.
Hosts DenyFrom rsyncd.conf(5). A list of patterns to match with the hostname and IP address of a connecting client. The connection is rejected when the patterns match. Separate entries by pressing Enter.

Other Options

NameDescription
Auxiliary ParametersEnter any additional parameters from rsyncd.conf(5).
When a Hosts Allow list is defined, only the IPs and hostnames on the list are able to connect to the module.

To EDIT or DELETE a module, go to the Rsync Modules list and click for an entry.