Creating Cloud Sync Tasks
7 minute read.Last Modified 2022-05-10 12:28 EDT
Cloud sync tasks let TrueNAS integrate with a Cloud Storage provider for additional backup storage. Cloud Sync tasks allow for single time transfers or recurring transfers on a schedule, and are an effective method to back up data to a remote location.
Using the Cloud means that data can go to a third party commercial vendor not directly affiliated with iXsystems. Please investigate and fully understand vendor pricing policies and services before creating any Cloud Sync task. iXsystems is not responsible for any charges incurred from the use of third party vendors with the Cloud Sync feature.
Transferring data from TrueNAS to the Cloud requires saving Cloud Storage Provider credentials on the system.
Go to System > Cloud Credentials and click ADD.
Enter a credential Name and choose a Provider. The rest of the options vary by Provider.
Enter the required Authentication strings to enable saving the credential.
Some providers can automatically populate the required Authentication strings by logging in to the account. To automatically configure the credential, click Login to Provider and entering your account username and password.
We recommend verifying the credential before saving it.
- All system Storage configured and ready to receive or send data.
- A Cloud Storage provider account and a cloud storage location (like an Amazon S3 bucket).
- Cloud Storage account credentials must be saved in System > Cloud Credentials.
Go to Tasks > Cloud Sync Tasks and click ADD.
Give the task a Description and select a cloud credential. TrueNAS connects to the chosen Cloud Storage Provider and shows the available storage locations.
Decide if data is transferring to (PUSH) or from (PULL) the Cloud Storage location (Remote).
Choose a Transfer Mode:
SYNC keeps all the files identical between the two storage locations. If a sync encounters an error, the destination does not delete the files.
Syncing to a Backblaze B2 bucket does not delete files from the bucket, even when you delete those files locally. Instead, Backblaze tags files with a version number or moves them to a hidden state. To automatically delete old or unwanted files from the bucket, adjust the Backblaze B2 Lifecycle Rules.
A sync cannot delete files stored in Amazon S3 Glacier or S3 Glacier Deep Archive. You must first restore the files by another means, like the Amazon S3 console.
COPY duplicates each source file into the destination, overwriting any destination files with the same name as the source. Copying is the least potentially destructive option.
MOVE transfers the files from the source to the destination and deletes the original source files. It also overwrites files with the same names on the destination.
Next, select a Schedule from the drop-down, or unset Enable to make the task available without running on a schedule.
Choosing a Presets option populates the rest of the fields.
To customize a schedule, enter crontab values for the
These fields accept standard cron values. The simplest option is to enter a single number in the field. The task runs when the time value matches that number. For example, entering 10 means that the job runs when the time is ten minutes past the hour.
An asterisk (
*) means match all values.
Specific time ranges are set by entering hyphenated number values. For example, entering 30-35 in the Minutes field sets the task to run at minutes 30, 31, 32, 33, 34, and 35.
You can also enter lists of values.
Enter individual values separated by a comma (
For example, entering 1,14 in the Hours field means the task runs at 1:00 AM (0100) and 2:00 PM (1400).
A slash (
/) designates a step value.
For example, while entering
* in Days means the task runs every day of the month,
*/2 means the task runs every other day.
Combining all the above examples together creates a schedule running a task each minute from 1:30-1:35 AM and 2:30-2:35 PM every other day.
There is an option to select which Months the task runs. Leaving each month unset is the same as selecting every month.
The Days of Week schedules the task to run on specific days. This is in addition to any listed days. For example, entering 1 in Days and setting Wed for Days of Week creates a schedule that starts a task on the first day of the month and every Wednesday of the month.
The Schedule Preview displays when the current settings mean the task runs.
|*||Every item.||* (minutes) = every minute of the hour.|
* (days) = every day.
|*/N||Every Nth item.||*/15 (minutes) = every 15th minute of the hour (every quarter hour).|
*/3 (days) = every 3rd day.
*/3 (months) = every 3rd month.
|Comma and hyphen/dash||Each stated item (comma)|
Each item in a range (hyphen/dash).
|1,31 (minutes) = on the 1st and 31st minute of the hour.|
1-3,31 (minutes) = on the 1st to 3rd minutes inclusive, and the 31st minute, of the hour.
mon-fri (days) = every Monday to Friday inclusive (every weekday).
mar,jun,sep,dec (months) = every March, June, September, December.
Days can be specified as days of month, or days of week.
With these options, you can create flexible schedules similar to these examples:
|Desired schedule||Values to enter|
|3 times a day (at midnight, 08:00 and 16:00)||months=*; days=*; hours=0/8 or 0,8,16; minutes=0|
(Meaning: every day of every month, when hours=0/8/16 and minutes=0)
|Every Monday, Wednesday and Friday, at 8.30 pm||months=*; days=mon,wed,fri; hours=20; minutes=30|
|1st and 15th day of the month, during October to June, at 00:01 am||months=oct-dec,jan-jun; days=1,15; hours=0; minutes=1|
|Every 15 minutes during the working week, which is 8am - 7pm (08:00 - 19:00) Monday to Friday||Note that this requires two tasks to achieve:|
(1) months=*; days=mon-fri; hours=8-18; minutes=*/15
(2) months=*; days=mon-fri; hours=19; minutes=0
We need the second scheduled item, to execute at 19:00, otherwise we would stop at 18:45. Another workaround would be to stop at 18:45 or 19:45 rather than 19:00.
Advanced users can write scripts that run immediately before or after the Cloud Sync task. The Post-script field runs when the Cloud Sync task successfully completes. You can pass a variety of task environment variables into the Pre-script and Post-script fields:
There also are provider-specific variables like CLOUD_SYNC_CLIENT_ID or CLOUD_SYNC_TOKEN or CLOUD_SYNC_CHUNK_SIZE.
Remote storage settings:
Local storage settings:
Test the settings before saving by clicking DRY RUN. TrueNAS connects to the Cloud Storage Provider and simulates a file transfer without sending or receiving data.
Saved tasks activate based on their schedule, or when you click RUN NOW. An in-progress cloud sync must finish before another can begin. Stopping an in-progress task cancels the file transfer and requires starting the file transfer over.
To view logs about a running task or a task most recent run, click the task status.
To quickly create a new cloud sync task that uses the same options but reverses the data transfer, expand () on an existing task and click RESTORE.
Give the new task a Description and define the path to a storage location for the transferred data.
TrueNAS saves the restored cloud sync task as another entry in Tasks > Cloud Sync Tasks.
If the restore destination dataset is the same as the original source dataset, the restored files might have their ownership altered to root. If root did not create the original files and they need a different owner, you can recursively reset ACL Permissions of the restored dataset through the GUI or by running
chown from the CLI.