(408) 943-4100               V   Commercial Support

Cloud Sync Tasks

  9 minute read.

Last Modified 2021-06-29 11:19 EDT

TrueNAS can send, receive, or synchronize data with a Cloud Storage provider. Cloud Sync tasks allow for single time transfers or recurring transfers on a schedule and are an effective method to back up data to a remote location.

Using the Cloud means that data can go to a third party commercial vendor not directly affiliated with iXsystems. Please investigate and fully understand that vendor’s pricing policies and services before creating any Cloud Sync task. iXsystems is not responsible for any charges incurred from the use of third party vendors with the Cloud Sync feature.

TrueNAS supports major providers like Amazon S3, Google Cloud, and Microsoft Azure, along with a variety of other vendors. To see the full list of supported vendors, go to Credentials > Backup Credentials > Cloud Credentials click Add and open the Provider dropdown.

Requirements

  • All system Storage must be configured and ready to receive or send data.
  • A Cloud Storage provider account and a cloud storage location, like an Amazon S3 bucket, must be available.
  • Cloud Storage account credentials must be saved to System > Cloud Credentials before creating the sync task. See Cloud Credentials for specific instructions.

Creating a Cloud Sync Task

Go to Data Protection > Cloud Sync Tasks and click Add.

DataProtectionCloudSyncAdd

Give the task a memorable Description and select an existing cloud Credential. TrueNAS connects to the chosen Cloud Storage Provider and shows the available storage locations. Decide if data is transferring to (PUSH) or from (PULL) the Cloud Storage location (Remote). Choose a Transfer Mode:

Sync keeps all the files identical between the two storage locations. If the sync encounters an error, files are not deleted in the destination. This includes a common error when the Dropbox copyright detector flags a file as copyrighted.

Note that syncing to a Backblaze B2 bucket does not delete files from the bucket, even when those files have been deleted locally. Instead, files are tagged with a version number or moved to a hidden state. To automatically delete old or unwanted files from the bucket, adjust the Backblaze B2 Lifecycle Rules.

Files stored in Amazon S3 Glacier or S3 Glacier Deep Archive cannot be deleted by a sync. These files must first be restored by another means, like the Amazon S3 console.

Copy duplicates each source file into the destination, overwriting any files in the destination that have the same name as the source. Copying is the least potentially destructive option.
Move transfers the files from the source to the destination and deletes the original source files. Files with the same names on the destination are overwritten.

Next, Control when the task runs by defining a Schedule. When a specific Schedule is required, choose Custom and use the Advanced Scheduler.

Tasks Advanced Scheduler

Choosing a preset fills in the rest of the fields. To customize a schedule, enter crontab values for the Minutes/Hours/Days.

These fields accept standard cron values. The simplest option is to enter a single number in the field. The task runs when the time value matches that number. For example, entering 10 means that the job runs when the time is ten minutes past the hour.

An asterisk (*) means “match all values”.

Specific time ranges are set by entering hyphenated number values. For example, entering 30-35 in the Minutes field sets the task to run at minutes 30, 31, 32, 33, 34, and 35.

Lists of values can also be entered. Enter individual values separated by a comma (,). For example, entering 1,14 in the Hours field means the task runs at 1:00 AM (0100) and 2:00 PM (1400).

A slash (/) designates a step value. For example, while entering * in Days means the task runs every day of the month, */2 means the task runs every other day.

Combining all the above examples together creates a schedule running a task each minute from 1:30-1:35 AM and 2:30-2:35 PM every other day.

There is an option to select which Months the task runs. Leaving each month unset is the same as selecting every month.

The Days of Week schedules the task to run on specific days. This is in addition to any listed Days. For example, entering 1 in Days and setting Wed for Days of Week creates a schedule that starts a task on the first day of the month and every Wednesday of the month.

The Schedule Preview shows when the current settings mean the task runs.

Examples of CRON syntax

SyntaxMeaningExamples
*Every item.* (minutes) = every minute of the hour.
* (days) = every day.
*/NEvery Nth item.*/15 (minutes) = every 15th minute of the hour (every quarter hour).
*/3 (days) = every 3rd day.
*/3 (months) = every 3rd month.
Comma and hyphen/dashEach stated item (comma)
Each item in a range (hyphen/dash).
1,31 (minutes) = on the 1st and 31st minute of the hour.
1-3,31 (minutes) = on the 1st to 3rd minutes inclusive, and the 31st minute, of the hour.
mon-fri (days) = every Monday to Friday inclusive (every weekday).
mar,jun,sep,dec (months) = every March, June, September, December.

Days can be specified as days of month, or days of week.

With these options, flexible schedules can be created similar to these examples:

Desired scheduleValues to enter
3 times a day (at midnight, 08:00 and 16:00)months=*; days=*; hours=0/8 or 0,8,16; minutes=0
(Meaning: every day of every month, when hours=0/8/16 and minutes=0)
Every Monday, Wednesday and Friday, at 8.30 pmmonths=*; days=mon,wed,fri; hours=20; minutes=30
1st and 15th day of the month, during October to June, at 00:01 ammonths=oct-dec,jan-jun; days=1,15; hours=0; minutes=1
Every 15 minutes during the working week, which is 8am - 7pm (08:00 - 19:00) Monday to FridayNote that this requires two tasks to achieve:
(1) months=*; days=mon-fri; hours=8-18; minutes=*/15
(2) months=*; days=mon-fri; hours=19; minutes=0
We need the second scheduled item, to execute at 19:00, otherwise we would stop at 18:45. Another workaround would be to stop at 18:45 or 19:45 rather than 19:00.

Unsetting Enable makes the configuration available without allowing the Schedule to run the task. To manually activate a saved task, go to Data Protection > Cloud Sync Tasks, click for the cloud sync task you want to run. You will be prompted to CONTINUE or CANCEL the Run Now operation.

The remaining options allow tuning the task to your specific requirements.

Transfer

NameDescription
DescriptionEnter a description of the Cloud Sync Task.
DirectionPUSH sends data to cloud storage. PULL receives data from cloud storage. Changing the direction resets the Transfer Mode to COPY.
Transfer ModeSYNC: Files on the destination are changed to match those on the source. If a file does not exist on the source, it is also deleted from the destination. COPY: Files from the source are copied to the destination. If files with the same names are present on the destination, they are overwritten. MOVE: After files are copied from the source to the destination, they are deleted from the source. Files with the same names on the destination are overwritten.
Directory/FilesSelect the directories or files to be sent to the cloud for Push syncs, or the destination to be written for Pull syncs. Be cautious about the destination of Pull jobs to avoid overwriting existing files.

Remote

NameDescription
CredentialSelect the cloud storage provider credentials from the list of available Cloud Credentials.

Control

NameDescription
ScheduleSelect a schedule preset or choose Custom to open the advanced scheduler.
EnabledEnable this Cloud Sync Task. Unset to disable this Cloud Sync Task without deleting it.

Advanced Options

NameDescription
Follow SymlinksFollow symlinks and copy the items to which they link.
Pre-ScriptScript to execute before running sync.
Post-ScriptScript to execute after running sync.
ExcludeList of files and directories to exclude from sync. Separate entries by pressing Enter. See rclone filtering for more details about the --exclude option.

Advanced Remote Options

NameDescription
Remote EncryptionPUSH: Encrypt files before transfer and store the encrypted files on the remote system. Files are encrypted using the Encryption Password and Encryption Salt values. PULL: Decrypt files that are being stored on the remote system before the transfer. Transferring the encrypted files requires entering the same Encryption Password and Encryption Salt that was used to encrypt the files. Additional details about the encryption algorithm and key derivation are available in the rclone crypt File formats documentation.
TransfersNumber of simultaneous file transfers. Enter a number based on the available bandwidth and destination system performance. See rclone –transfers.
Bandwidth limitA single bandwidth limit or bandwidth limit schedule in rclone format. Separate entries by pressing Enter. Example: 08:00,512 12:00,10MB 13:00,512 18:00,30MB 23:00,off. Units can be specified with the beginning letter: b, k (default), M, or G. See rclone –bwlimit.

Scripting and Environment Variables

Advanced users can write scripts that run immediately before or after the Cloud Sync task. The Post-script field is only run when the Cloud Sync task successfully completes. You can pass a variety of task environment variables into the Pre- and Post- script fields:

  • CLOUD_SYNC_ID
  • CLOUD_SYNC_DESCRIPTION
  • CLOUD_SYNC_DIRECTION
  • CLOUD_SYNC_TRANSFER_MODE
  • CLOUD_SYNC_ENCRYPTION
  • CLOUD_SYNC_FILENAME_ENCRYPTION
  • CLOUD_SYNC_ENCRYPTION_PASSWORD
  • CLOUD_SYNC_ENCRYPTION_SALT
  • CLOUD_SYNC_SNAPSHOT

There also are provider-specific variables like CLOUD_SYNC_CLIENT_ID or CLOUD_SYNC_TOKEN or CLOUD_SYNC_CHUNK_SIZE.

Remote storage settings:

  • CLOUD_SYNC_BUCKET
  • CLOUD_SYNC_FOLDER

Local storage settings:

  • CLOUD_SYNC_PATH

Testing Settings

Test the settings before saving by clicking Dry Run. TrueNAS connects to the Cloud Storage Provider and simulates a file transfer. No data is actually sent or received. A dialog shows the test status and allows downloading the task logs. This is also ran by clicking after the task is created.

DataProtectionCloudSyncAddBoxDryRun

Cloud Sync Behavior

Saved tasks activate according to their schedule or by clicking . An in-progress cloud sync must finish before another can begin. Stopping an in-progress task cancels the file transfer and requires starting the file transfer over.

To view logs about a running or the most recent run of a task, click the task State.

Cloud Sync Restore

To quickly create a new Cloud Sync that uses the same options but reverses the data transfer, select for an existing Cloud Sync on the Data Protection page.

DataProtectionsCloudSyncRestore

Enter a new Description for this reversed task, define the path to a storage location for the transferred data and click Restore.

The restored cloud sync is saved as another entry in Data protection > Cloud Sync Tasks.

In case the restore destination dataset is the same as the original source dataset, the restored files might have their ownership altered to root. If the original files were not created by root and a different owner is required, you can recursively reset ACL Permissions of the restored dataset through the GUI or by running chown from the CLI.