Cloud Sync Tasks
10 minute read.Last Modified 2022-06-01 15:15 EDT
TrueNAS can send, receive, or synchronize data with a cloud storage provider. Cloud sync tasks allow for single-time transfers or recurring transfers on a schedule. They are an effective method to back up data to a remote location.
Using the cloud means data can go to a third-party commercial vendor not directly affiliated with iXsystems. You should fully understand vendor pricing policies and services before using them for cloud sync tasks.
iXsystems is not responsible for any charges incurred from using third-party vendors with the cloud sync feature.
TrueNAS supports major providers like Amazon S3, Google Cloud, and Microsoft Azure. It also supports many other vendors. To see the full list of supported vendors, go to Credentials > Backup Credentials > Cloud Credentials click Add and open the Provider drop-down list.
- You must have all system storage configured and ready to receive or send data.
- You must have a cloud storage provider account and location (like an Amazon S3 bucket).
- You must have cloud storage account credentials saved to System > Cloud Credentials before creating the sync task. See Cloud Credentials for specific instructions.
Go to Data Protection > Cloud Sync Tasks and click Add.
Type a memorable task description in the Description field. Use the Credential dropdown list to select an existing cloud or create a new one with the + Add a backup credential option.
Once you choose a cloud credential from the dropdown list, TrueNAS automatically validates access to that cloud sync provider. Invalid credentials results in the following Alert:
Click FIX CREDENTIAL. TrueNAS directs you to the Cloud Credentials entry view.
Check your provider credentials and update the applicable fields within the Authentication section, then click Verify Credential. The Credential is valid will display if TrueNAS successfully accesses your provider. Click Save and return to Data Protection > Cloud Sync Tasks > Add.
TrueNAS connects to the chosen cloud storage provider and shows the available storage locations. Select the option if data is transferring to (PUSH) or from (PULL) the cloud storage location (Remote). Select a Transfer Mode:
Sync keeps all the files identical between the two storage locations. If the sync encounters an error, it does not delete files in the destination. One common error occurs when the Dropbox copyright detector flags a file as copyrighted.
Note that syncing to a Backblaze B2 bucket does not delete files from the bucket, even when you deleted those files locally. Instead, files are tagged with a version number or moved to a hidden state. To automatically delete old or unwanted files from the bucket, adjust the Backblaze B2 Lifecycle Rules.
Sync cannot delete files stored in Amazon S3 Glacier or S3 Glacier Deep Archive. These files must first be restored by another means, like the Amazon S3 console.
Next, use the Control options. Define when the task runs using Schedule. If you need a specific schedule, select Custom and use the Advanced Scheduler.
Choosing a Presets option populatess in the rest of the fields.
To customize a schedule, enter crontab values for the
These fields accept standard cron values. The simplest option is to enter a single number in the field. The task runs when the time value matches that number. For example, entering 10 means that the job runs when the time is ten minutes past the hour.
An asterisk (
*) means match all values.
You can set specific time ranges by entering hyphenated number values. For example, entering 30-35 in the Minutes field sets the task to run at minutes 30, 31, 32, 33, 34, and 35.
You can also enter lists of values.
Enter individual values separated by a comma (
For example, entering 1,14 in the Hours field means the task runs at 1:00 AM (0100) and 2:00 PM (1400).
A slash (
/) designates a step value.
For example, entering
* in Days runs the task every day of the month. Entering
*/2 runs it every other day.
Combining the above examples creates a schedule running a task each minute from 1:30-1:35 AM and 2:30-2:35 PM every other day.
TrueNAS has an option to select which Months the task runs. Leaving each month unset is the same as selecting every month.
The Days of Week schedules the task to run on specific days in addition to any listed days. For example, entering 1 in Days and setting Wed for Days of Week creates a schedule that starts a task on the first day of the month and every Wednesday of the month.
The Schedule Preview dipslays when the current settings mean the task runs.
|*||Every item.||* (minutes) = every minute of the hour.|
* (days) = every day.
|*/N||Every Nth item.||*/15 (minutes) = every 15th minute of the hour.|
*/3 (days) = every 3rd day.
*/3 (months) = every 3rd month.
|Comma and hyphen/dash||Each stated item (comma)|
Each item in a range (hyphen/dash).
|1,31 (minutes) = on the 1st and 31st minute of the hour.|
1-3,31 (minutes) = on the 1st to 3rd minutes inclusive, and the 31st minute, of the hour.
mon-fri (days) = every Monday to Friday inclusive (every weekday).
mar,jun,sep,dec (months) = every March, June, September, December.
You can specify days of the month or days of the week.
TrueNAS lets users create flexible schedules using the available options. The table below has some examples:
|Desired schedule||Values to enter|
|3 times a day (at midnight, 08:00 and 16:00)||months=*; days=*; hours=0/8 or 0,8,16; minutes=0|
(Meaning: every day of every month, when hours=0/8/16 and minutes=0)
|Every Monday/Wednesday/Friday, at 8.30 pm||months=*; days=mon,wed,fri; hours=20; minutes=30|
|1st and 15th day of the month, during October to June, at 00:01 am||months=oct-dec,jan-jun; days=1,15; hours=0; minutes=1|
|Every 15 minutes during the working week, which is 8am - 7pm (08:00 - 19:00) Monday to Friday||Note that this requires two tasks to achieve:|
(1) months=*; days=mon-fri; hours=8-18; minutes=*/15
(2) months=*; days=mon-fri; hours=19; minutes=0
We need the second scheduled item, to execute at 19:00, otherwise we would stop at 18:45. Another workaround would be to stop at 18:45 or 19:45 rather than 19:00.
Clear the Enable checkbox to make the configuration available without allowing the specified schedule to run the task. To manually activate a saved task, go to Data Protection > Cloud Sync Tasks, click for the cloud sync task you want to run. You are prompted to CONTINUE or CANCEL the Run Now operation.
The remaining options allow tuning the task to your specific requirements.
|Description||Enter a description of the Cloud Sync Task.|
|Direction||PUSH sends data to cloud storage. PULL receives data from cloud storage. Changing the direction resets the Transfer Mode to COPY.|
|Transfer Mode||SYNC: Files on the destination are changed to match those on the source. If a file does not exist on the source, it is also deleted from the destination. COPY: Files from the source are copied to the destination. If files with the same names are present on the destination, they are overwritten. MOVE: After files are copied from the source to the destination, they are deleted from the source. Files with the same names on the destination are overwritten.|
|Directory/Files||Select the directories or files to be sent to the cloud for Push syncs, or the destination to be written for Pull syncs. Be cautious about the destination of Pull jobs to avoid overwriting existing files.|
|Credential||Select the cloud storage provider credentials from the list of available Cloud Credentials.|
|Schedule||Select a schedule preset or choose Custom to open the advanced scheduler.|
|Enabled||Enable this Cloud Sync Task. Unset to disable this Cloud Sync Task without deleting it.|
|Follow Symlinks||Follow symlinks and copy the items to which they link.|
|Pre-Script||Script to execute before running sync.|
|Post-Script||Script to execute after running sync.|
|Exclude||List of files and directories to exclude from sync. Separate entries by pressing Enter.|
Examples of proper syntax used to exclude files/directories are:
Advanced Remote Options
|Remote Encryption||PUSH: Encrypt files before transfer and store the encrypted files on the remote system. Files are encrypted using the Encryption Password and Encryption Salt values. PULL: Decrypt files that are being stored on the remote system before the transfer. Transferring the encrypted files requires entering the same Encryption Password and Encryption Salt that was used to encrypt the files. Additional details about the encryption algorithm and key derivation are available in the rclone crypt File formats documentation.|
|Transfers||Number of simultaneous file transfers. Enter a number based on the available bandwidth and destination system performance. See rclone –transfers.|
|Bandwidth limit||A single bandwidth limit or bandwidth limit schedule in rclone format. Separate entries by pressing Enter. Example: |
Advanced users can write scripts that run immediately before or after the cloud sync task. The Post-script field only runs when the cloud sync task succeeds. You can pass a variety of task environment variables into the Pre- and Post- script fields:
There also are provider-specific variables like CLOUD_SYNC_CLIENT_ID or CLOUD_SYNC_TOKEN or CLOUD_SYNC_CHUNK_SIZE.
Remote storage settings:
Local storage settings:
To test the settings before saving, click Dry Run. TrueNAS connects to the cloud storage provider and simulates a file transfer but does not send or receive data. A dialog box displays with the test status and allows you to download the task logs. You can also run this by clicking loop after creating the task.
Saved tasks activate according to their schedule or by clicking play_arrow. An in-progress cloud sync must finish before another can begin. Stopping an in-progress task cancels the file transfer and requires starting the file transfer over.
To view logs about a running task, or its most recent run, click State.
To quickly create a new cloud sync that uses the same options but reverses the data transfer, select history for an existing cloud sync on the Data Protection page.
Type a new description for this reversed task, then define the path to a storage location for the transferred data and click Restore.
TrueNAS saves the restored cloud sync as another entry in Data protection > Cloud Sync Tasks.
If you set the restore destination to the source dataset, TrueNAS may alter ownership of the restored files to root. If root did not create the original files and you need them to have a different owner, you can recursively reset their ACL permissions through the GUI or run
chown from the CLI.