Cloud Sync Tasks
9 minute read.Last Modified 2021-12-09 13:08 EST
TrueNAS can send, receive, or synchronize data with a cloud storage provider. Cloud sync tasks allow for single time transfers or recurring transfers on a schedule and are an effective method to back up data to a remote location.
Using the cloud means that data can go to a third party commercial vendor not directly affiliated with iXsystems. Investigate and fully understand that cloud vendor’s pricing policies and services before creating any cloud sync task. iXsystems is not responsible for any charges incurred from the use of third party vendors with the Cloud Sync feature.
TrueNAS supports major providers like Amazon S3, Google Cloud, and Microsoft Azure, along with a variety of other vendors. To see the full list of supported vendors, go to Credentials > Backup Credentials > Cloud Credentials click Add and open the Provider dropdown.
- All system Storage must be configured and ready to receive or send data.
- A cloud storage provider account and a cloud storage location, like an Amazon S3 bucket, must be available.
- A cloud storage account credentials must be saved to System > Cloud Credentials before creating the sync task. See Cloud Credentials for specific instructions.
Go to Data Protection > Cloud Sync Tasks and click Add.
Type a memorable task description in the Description field and select an existing cloud on the Credential dropdown list. TrueNAS connects to the chosen cloud storage provider and shows the available storage locations. Select the option if data is transferring to (PUSH) or from (PULL) the cloud storage location (Remote). Select a Transfer Mode:
Sync keeps all the files identical between the two storage locations. If the sync encounters an error, files are not deleted in the destination. This includes a common error when the Dropbox copyright detector flags a file as copyrighted.
Note that syncing to a Backblaze B2 bucket does not delete files from the bucket, even when those files were deleted locally. Instead, files are tagged with a version number or moved to a hidden state. To automatically delete old or unwanted files from the bucket, adjust the Backblaze B2 Lifecycle Rules.
Files stored in Amazon S3 Glacier or S3 Glacier Deep Archive cannot be deleted by a sync. These files must first be restored by another means, like the Amazon S3 console.
Next, Use the Control options. Define when the task runs using Schedule. When a specific schedule is required, select Custom and use the Advanced Scheduler.
Choosing a Presets option populatess in the rest of the fields.
To customize a schedule, enter crontab values for the
These fields accept standard cron values. The simplest option is to enter a single number in the field. The task runs when the time value matches that number. For example, entering 10 means that the job runs when the time is ten minutes past the hour.
An asterisk (
*) means match all values.
You can set specific time ranges by entering hyphenated number values. For example, entering 30-35 in the Minutes field sets the task to run at minutes 30, 31, 32, 33, 34, and 35.
You can also enter lists of values.
Enter individual values separated by a comma (
For example, entering 1,14 in the Hours field means the task runs at 1:00 AM (0100) and 2:00 PM (1400).
A slash (
/) designates a step value.
For example, entering
* in Days runs the task every day of the month. Entering
*/2 runs it every other day.
Combining the above examples creates a schedule running a task each minute from 1:30-1:35 AM and 2:30-2:35 PM every other day.
TrueNAS has an option to select which Months the task runs. Leaving each month unset is the same as selecting every month.
The Days of Week schedules the task to run on specific days in addition to any listed days. For example, entering 1 in Days and setting Wed for Days of Week creates a schedule that starts a task on the first day of the month and every Wednesday of the month.
The Schedule Preview dipslays when the current settings mean the task runs.
|*||Every item.||* (minutes) = every minute of the hour.|
* (days) = every day.
|*/N||Every Nth item.||*/15 (minutes) = every 15th minute of the hour.|
*/3 (days) = every 3rd day.
*/3 (months) = every 3rd month.
|Comma and hyphen/dash||Each stated item (comma)|
Each item in a range (hyphen/dash).
|1,31 (minutes) = on the 1st and 31st minute of the hour.|
1-3,31 (minutes) = on the 1st to 3rd minutes inclusive, and the 31st minute, of the hour.
mon-fri (days) = every Monday to Friday inclusive (every weekday).
mar,jun,sep,dec (months) = every March, June, September, December.
You can specify days of the month or days of the week.
TrueNAS lets users create flexible schedules using the available options. The table below has some examples:
|Desired schedule||Values to enter|
|3 times a day (at midnight, 08:00 and 16:00)||months=*; days=*; hours=0/8 or 0,8,16; minutes=0|
(Meaning: every day of every month, when hours=0/8/16 and minutes=0)
|Every Monday/Wednesday/Friday, at 8.30 pm||months=*; days=mon,wed,fri; hours=20; minutes=30|
|1st and 15th day of the month, during October to June, at 00:01 am||months=oct-dec,jan-jun; days=1,15; hours=0; minutes=1|
|Every 15 minutes during the working week, which is 8am - 7pm (08:00 - 19:00) Monday to Friday||Note that this requires two tasks to achieve:|
(1) months=*; days=mon-fri; hours=8-18; minutes=*/15
(2) months=*; days=mon-fri; hours=19; minutes=0
We need the second scheduled item, to execute at 19:00, otherwise we would stop at 18:45. Another workaround would be to stop at 18:45 or 19:45 rather than 19:00.
Remove the checkmark from the Enable checkbox to make the configuration available without allowing the specified schedule to run the task. To manually activate a saved task, go to Data Protection > Cloud Sync Tasks, click for the cloud sync task you want to run. You are prompted to CONTINUE or CANCEL the Run Now operation.
The remaining options allow tuning the task to your specific requirements.
|Description||Enter a description of the Cloud Sync Task.|
|Direction||PUSH sends data to cloud storage. PULL receives data from cloud storage. Changing the direction resets the Transfer Mode to COPY.|
|Transfer Mode||SYNC: Files on the destination are changed to match those on the source. If a file does not exist on the source, it is also deleted from the destination. COPY: Files from the source are copied to the destination. If files with the same names are present on the destination, they are overwritten. MOVE: After files are copied from the source to the destination, they are deleted from the source. Files with the same names on the destination are overwritten.|
|Directory/Files||Select the directories or files to be sent to the cloud for Push syncs, or the destination to be written for Pull syncs. Be cautious about the destination of Pull jobs to avoid overwriting existing files.|
|Credential||Select the cloud storage provider credentials from the list of available Cloud Credentials.|
|Schedule||Select a schedule preset or choose Custom to open the advanced scheduler.|
|Enabled||Enable this Cloud Sync Task. Unset to disable this Cloud Sync Task without deleting it.|
|Follow Symlinks||Follow symlinks and copy the items to which they link.|
|Pre-Script||Script to execute before running sync.|
|Post-Script||Script to execute after running sync.|
|Exclude||List of files and directories to exclude from sync. Separate entries by pressing Enter.|
Examples of proper syntax used to exclude files/directories are:
Advanced Remote Options
|Remote Encryption||PUSH: Encrypt files before transfer and store the encrypted files on the remote system. Files are encrypted using the Encryption Password and Encryption Salt values. PULL: Decrypt files that are being stored on the remote system before the transfer. Transferring the encrypted files requires entering the same Encryption Password and Encryption Salt that was used to encrypt the files. Additional details about the encryption algorithm and key derivation are available in the rclone crypt File formats documentation.|
|Transfers||Number of simultaneous file transfers. Enter a number based on the available bandwidth and destination system performance. See rclone –transfers.|
|Bandwidth limit||A single bandwidth limit or bandwidth limit schedule in rclone format. Separate entries by pressing Enter. Example: |
Advanced users can write scripts that run immediately before or after the Cloud Sync task. The Post-script field is only run when the cloud sync task successfully completes. You can pass a variety of task environment variables into the Pre- and Post- script fields:
There also are provider-specific variables like CLOUD_SYNC_CLIENT_ID or CLOUD_SYNC_TOKEN or CLOUD_SYNC_CHUNK_SIZE.
Remote storage settings:
Local storage settings:
To test the settings before saving, click Dry Run. TrueNAS connects to the cloud storage provider and simulates a file transfer. No data is sent or received. A dialog box displays with the test status and allows you to download the task logs. You can also run this by clicking loop after the task is created.
Saved tasks activate according to their schedule or by clicking play_arrow. An in-progress cloud sync must finish before another can begin. Stopping an in-progress task cancels the file transfer and requires starting the file transfer over.
To view logs about a running or the most recent run of a task, click the task State.
To quickly create a new cloud sync that uses the same options but reverses the data transfer, select history for an existing cloud sync on the Data Protection page.
Type a new description for this reversed task, define the path to a storage location for the transferred data and click Restore.
The restored cloud sync is saved as another entry in Data protection > Cloud Sync Tasks.
In case the restore destination dataset is the same as the original source dataset, the restored files might have their ownership altered to root. If the original files were not created by root and a different owner is required, you can recursively reset ACL permissions of the restored dataset through the GUI or by running
chown from the CLI.