Adding Cloud Sync Tasks
7 minute read.Last Modified 2022-08-31 10:13 EDT
TrueNAS can send, receive, or synchronize data with a cloud storage provider. Cloud sync tasks allow for single-time transfers or recurring transfers on a schedule. They are an effective method to back up data to a remote location.
Using the cloud means data can go to a third-party commercial vendor not directly affiliated with iXsystems. You should fully understand vendor pricing policies and services before using them for cloud sync tasks.
iXsystems is not responsible for any charges incurred from using third-party vendors with the cloud sync feature.
TrueNAS supports major providers like Amazon S3, Google Cloud, and Microsoft Azure. It also supports many other vendors. To see the full list of supported vendors, go to Credentials > Backup Credentials > Cloud Credentials click Add and open the Provider dropdown list.
- You must have all system storage configured and ready to receive or send data.
- You must have a cloud storage provider account and location (like an Amazon S3 bucket).
You can create the cloud storage account credentials using Credentials > Backup Credentials > Cloud Credentials before creating the sync task or add it at the time you create the cloud sync task on Data Protection > Cloud Sync Task > Add Cloud Sync Task. See the Cloud Credentials article for instructions on adding a backup credential using cloud credentials.
(Required) Type a memorable task description in Description.
Select an existing backup credential from the Credential dropdown list or select + Add a backup credential to add a new backup credential. + Add a backup credential opens a backup credential configuration window.
a. (Required) Enter a name for the backup credential.
b. Select the cloud storage provider from the Provider dropdown list. The authentication fields displayed vary by provider. See Adding Cloud Credentials for more information on authentication settings for each provider.
c. Enter the authentication credentials for the selected provider.
d. Click Verify Credentials. TrueNAS attempts to connect to the selected provider with the authentication settings entered.
e. Click Save. The backup credentials window closes and the Credentials field displays the newly created backup credential. The Bucket field displays if using S3 to connect.
Select the direction for the sync task. PULL brings files from the cloud storage provider to the location specified in Directory/Files (this is the location on TrueNAS SCALE). PUSH sends files from the location in Directory/Files to the cloud storage provider location you specify in Folder.
Select the transfer method from the Transfer Mode dropdown list. Sync keeps files identical on both TrueNAS SCALE and the remote cloud provider server. If the sync encounters an error, destination server files are not deleted. Copy duplicates files on both the TrueNAS SCALE and remote cloud provider server. Move transfer the files to the destination server and then deleted the copy on server that transferred the files. It also overwrites files with the same names on the destination.
Enter or browse to the dataset or folder directory using thearrow to the left of / under the Directory/Files and Folder fields. Select the TrueNAS SCALE dataset path in Directory/Files and the Google Drive path in Folder. If PUSH is the selected Direction, this is where on TrueNAS SCALE the files you want to copy, sync or move transfer to the provider. If Direction is set to PULL this is the location where on TrueNAS SCALE you want to copy, sync or move files to.
Click theto the left of / to collapse the folder tree.
Select the preset from the Schedule dropdown that defines when the task runs. For a specific schedule, select Custom and use the Advanced Scheduler. Clear the Enable checkbox to make the configuration available without allowing the specified schedule to run the task.
To manually activate a saved task, go to Data Protection > Cloud Sync Tasks, click for the cloud sync task you want to run. Click CONTINUE or CANCEL for the Run Now operation.
(Optional) Set any advanced option you want or need for your use case or define environment variables in either the Pre-script or Post-script fields. These fields are for advanced users.
Click then click Dry Run to test your settings before you click Save. TrueNAS connects to the cloud storage provider and simulates a file transfer but does not send or receive data.
The new cloud sync task displays on the Cloud Sync Tasks widget with the status of PENDING until it completes. If the task completes without issue the status becomes SUCCESS.
See Using Scripting and Environment Variables for more information on environment variables.
After you choose a cloud credential from the dropdown list, TrueNAS automatically validates access to that cloud sync provider. Invalid credentials results in the following alert:
Click FIX CREDENTIAL opens the Credentials > Cloud Credentials > Edit Cloud Credentials screen for the cloud service selected in Credentials.
Check your provider credentials and update the applicable authentication fields on the Edit Cloud Credentials screen, and then click Verify Credential. If TrueNAS successfully accesses the provider the system displays the The Credential is valid dialog. Click Save and then return to Data Protection > Cloud Sync Tasks > Add to try again.
Sync keeps all the files identical between the two storage locations. If the sync encounters an error, it does not delete files in the destination.
One common error occurs when the Dropbox copyright detector flags a file as copyrighted.
Syncing to a Backblaze B2 bucket does not delete files from the bucket, even when you deleted those files locally. Instead, files are tagged with a version number or moved to a hidden state. To automatically delete old or unwanted files from the bucket, adjust the Backblaze B2 Lifecycle Rules.
Sync cannot delete files stored in Amazon S3 Glacier or S3 Glacier Deep Archive. First restore these files by another means, like the Amazon S3 console.
Advanced users can write scripts that run immediately before or after the cloud sync task. Using either the Add Cloud Sync Task or Edit Cloud Sync Task screens, enter environment variables to either the Pre-script or Post-script fields. The Post-script field only runs when the cloud sync task succeeds.
There also are provider-specific variables like CLOUD_SYNC_CLIENT_ID or CLOUD_SYNC_TOKEN or CLOUD_SYNC_CHUNK_SIZE.
Remote storage settings:
Local storage settings:
Saved tasks activate according to their schedule or you can use the Run Now option the Cloud Sync Task widget.
To run the sync task before the saved schedule for the task, click on the cloud sync task to open the edit configuration screen for that task. If not already cleared, select Enable below the Schedule field to clear the checkbox, and then click Save.
On the Cloud Sync Task widget, click the Run Now play_arrow button.
An in-progress cloud sync must finish before another can begin. Stopping an in-progress task cancels the file transfer and requires starting the file transfer over.
To view logs about a running task, or its most recent run, click State.
To create a new cloud sync task that uses the same options but reverses the data transfer, select history for an existing cloud sync on the Data Protection page. The Restore Cloud Sync Task window opens.
Enter a name in Description for this reversed task.
Select the Transfer Mode and then define the path for a storage location on TrueNAS scale for the transferred data.
TrueNAS saves the restored cloud sync as another entry in Data protection > Cloud Sync Tasks.
If you set the restore destination to the source dataset, TrueNAS may alter ownership of the restored files to root. If root did not create the original files and you need them to have a different owner, you can recursively reset their ACL permissions through the GUI or run
chown from the CLI.