TrueNAS SCALE Nightly Development Documentation
This content follows experimental early release software. Use the Product and Version selectors above to view content specific to a stable software release.

Cloud Sync Tasks Screens

12 minute read.

Last Modified 2024-04-16 09:41 EDT

The Cloud Sync Tasks widget on the Data Protection screen shows configured cloud sync tasks, and provides access to configuration screens to add single-time or scheduled recurring transfers between TrueNAS SCALE and a cloud storage provider. Cloud sync tasks are an effective method to back up data to a remote location.

These providers are supported for Cloud Sync tasks in TrueNAS SCALE:

OpenStack Swift
pCloud
SSH File Transfer Protocol (SFTP)
Storj iX
WebDAV
Yandex

The Cloud Sync Task widget shows a list of tasks configured on the system.

The widget shows No Cloud Sync Tasks configured until cloud sync task are added.

Click on the widegt header to open the Cloud Sync Task screen that lists all tasks configured on the system.

Add on the widget and the Cloud Sync Task screen opens the Cloudsync Task Wizard.

Each task includes five icons for various functions:

The edit Edit icon opens the Edit Cloud Sync Task screen populated with with the settings for that task.
The Run Now icon starts the cloud sync, running it outside of the scheduled time.
The loop Dry Run icon performs the same function as the Dry Run button on the add and edit configuration screens. It performs a test of the configured settings. When doing a dry run, you can close the window and monitor the task using the Jobs option on the top toolbar.
The restore Restore icon creates a new cloud sync task from an existing task. The new task has the same settings but reverses the data transfer.
The delete Delete icon opens a confirmation dialog before the system deletes the task.

State displays the status of the next cloud sync task as SUCCESS for completed tasks, FAILED if the task fails to complete the sync, and PENDING for tasks that have not run yet. Click on the state oval to open the Logs dialog for that task. Download Logs saves a copy of the current task logs.

Cloud Sync Task Screen

The Cloud Sync Task screen lists all tasks configured on the system.

Expand any task to see details on the configured task, such as the cloud sync provider, direction, transfer mode, path to the dataset or directories and other options for that task.

Buttons for these task options perform the same functions as the icons on the widget:

Run Now starts the task outside of the scheduled period.
Dry Run performs a test of the configuration. This is the same function as the Dry Run button on the Edit Cloud Sync Task screen and the Advanced Options for the Cloudsync Task Wizard.
Restore opens the Restore Cloud Sync Task window where you can create a new cloud sync task from an existing task with the same options but the new task reverses the transfer from PUSH to PULL and vice-versa.
Figure 6: Restore Cloud Sync Tasks
Edit opens the Edit Cloud Sync Task screen.
Delete opens a dialog where you confirm the action before the system deletes the task.

Cloudsync Task Wizard

The Cloud Sync Task wizard screens simplify the task creation process. It includes two screens, Provider and What and When.

Provider Wizard Screen

The Provider wizard screen allows you to select the cloud sync provider with the Credentials dropdown. Select the provider from the dropdown list to show the additional credential settings that provider requires to establish a connection.

Figure 7: Cloudsync Task Wizard Provider Screen

Select Add New to open the Cloud Credentials screen. This is the same configuration screen as when you click Add on the Credentials > Backup Credentials screen.

Advanced Options opens a screen with the same settings as the Edit Cloud Sync Task screen.

Verify Credentials tests the settings before you advance to the settings on the What and When wizard screen.

What and When Wizard Screen

The What and When screen sets the direction (PUSH or PULL), transfer mode (move, copy, or sync), the datasets or directories source and destination, and sets the schedule for the transfer. The Bucket field displays for providers that use buckets to hold transferred files, folders, etc.

The Advanced Options button shows at the bottom of this screen as well.

Advanced Options and Edit Cloud Sync Task Screens

The Advanced Options accessed from the Cloudsync Task Wizard and Edit Cloud Sync Task display the same settings. Settings are grouped into the the Transfer, Remote, Control, and Advanced Options for advanced users.

The Manage Credentials link opens the Backup Credentials screen.

Transfer Settings

Transfer settings change the cloud sync task direction (PUSH or PULL), data transfer method (COPY, MOVE, or SYNC), and allows selecting the dataset or directory to use in the task. Selecting the dataset or file populates the Directory/Files field.

Figure 9: Add Cloud Sync Task Remote Settings

Browsing to select a path

Click the arrow to the left of the folder icon to expand that folder and show any child datasets and directories. A solid folder icon shows for datasets and an outlined folder for directories. A selected dataset or directory folder and name shows in blue.

Settings	Description
Description	Enter a description of the cloud sync task.
Direction	Select a direction option from the dropdown list. PUSH sends data to cloud storage. PULL receives data from cloud storage and is the default setting.
Transfer Mode	Select the transfer mode type from the dropdown list. To keep all files identical between the two storage locations, select SYNC. This changes files on the destination to match those on the source. If a file does not exist on the source, it is also deleted from the destination. There are three options: COPY - Select to duplicate each source file into the destination and overwrite destination files using the same source. Copies files from the source to the destination. If files with the same names are present on the destination, they are overwritten. MOVE - Select to transfer files from the source to the destination and delete source files. Copies files from the source to the destination and then deletes them from the source. Files with the same names on the destination are overwritten. SYNC - Select to change files on the destination to match those on the source. If a file does not exist on the source, it is also deleted from the destination.
Directory/Files	Enter or click the arrow_right arrow to the left of /mnt folder to expand and show datasets and directories within that folder. When you locate the dataset or directory location to send to the cloud for push syncs, or as the destination to write to for pull syncs. Be cautious with pull destinations to avoid overwriting existing files. Click the arrow_right arrow to the left of /mnt folder again to collapse the directory tree.

Remote Settings

The Remote settings specify the cloud sync provider and destination storage locations. The option selected in Credential changes settings displayed in the Remote settings area. The Manage Credentials link opens the Backup Credentials screen where you can add a new provider credential.

Settings	Description
Credential	Select an exiting backup cloud storage provider credential from the dropdown list. A Bucket setting displays after selecting a credential that uses S3, like Amazon S3. TrueNAS automatically validates the selected credential.
Bucket	Select the pre-defined bucket S3 to use.
Folder	Enter or click the arrow_right arrow to the left of the folder icon and at each dataset or directory to reach the storage location to use for this task.

Control Settings

Control settings establish a schedule for when the cloud sync task occurs.

Settings	Description
Schedule	Select a schedule preset or choose Custom to open the advanced scheduler.
Enabled	Select to enable this cloud sync task. Leave clear to disable the task without deleting it and keep the configuration available without allowing the specified schedule to run the task. You can use the toggle in the Enable column on the Cloud Sync Tasks widget to enable or disable the task.

Advanced Scheduler

Choosing a Presets option populates the rest of the fields. To customize a schedule, enter crontab values for the Minutes/Hours/Days.

These fields accept standard cron values. The simplest option is to enter a single number in the field. The task runs when the time value matches that number. For example, entering 10 means that the job runs when the time is ten minutes past the hour.

An asterisk (*) means match all values.

Specific time ranges are set by entering hyphenated number values. For example, entering 30-35 in the Minutes field sets the task to run at minutes 30, 31, 32, 33, 34, and 35.

You can also enter lists of values. Enter individual values separated by a comma (,). For example, entering 1,14 in the Hours field means the task runs at 1:00 AM (0100) and 2:00 PM (1400).

A slash (/) designates a step value. For example, while entering * in Days means the task runs every day of the month, */2 means the task runs every other day.

Combining all the above examples together creates a schedule running a task each minute from 1:30-1:35 AM and 2:30-2:35 PM every other day.

There is an option to select which Months the task runs. Leaving each month unset is the same as selecting every month.

The Days of Week schedules the task to run on specific days. This is in addition to any listed days. For example, entering 1 in Days and setting Wed for Days of Week creates a schedule that starts a task on the first day of the month and every Wednesday of the month.

The Schedule Preview displays when the current settings mean the task runs.

Examples of CRON syntax

Syntax	Meaning	Examples
*	Every item.	* (minutes) = every minute of the hour. * (days) = every day.
*/N	Every N^th item.	/15 (minutes) = every 15th minute of the hour (every quarter hour). /3 (days) = every 3rd day. */3 (months) = every 3rd month.
Comma and hyphen/dash	Each stated item (comma) Each item in a range (hyphen/dash).	1,31 (minutes) = on the 1st and 31st minute of the hour. 1-3,31 (minutes) = on the 1st to 3rd minutes inclusive, and the 31st minute, of the hour. mon-fri (days) = every Monday to Friday inclusive (every weekday). mar,jun,sep,dec (months) = every March, June, September, December.

Days can be specified as days of month, or days of week.

With these options, you can create flexible schedules similar to these examples:

Desired schedule	Values to enter
3 times a day (at midnight, 08:00 and 16:00)	months=; days=; hours=0/8 or 0,8,16; minutes=0 (Meaning: every day of every month, when hours=0/8/16 and minutes=0)
Every Monday, Wednesday and Friday, at 8.30 pm	months=*; days=mon,wed,fri; hours=20; minutes=30
1st and 15th day of the month, during October to June, at 00:01 am	months=oct-dec,jan-jun; days=1,15; hours=0; minutes=1
Every 15 minutes during the working week, which is 8am - 7pm (08:00 - 19:00) Monday to Friday	Note that this requires two tasks to achieve: (1) months=; days=mon-fri; hours=8-18; minutes=/15 (2) months=*; days=mon-fri; hours=19; minutes=0 We need the second scheduled item, to execute at 19:00, otherwise we would stop at 18:45. Another workaround would be to stop at 18:45 or 19:45 rather than 19:00.

Advanced Options Settings

Advanced Options settings are for advanced users. Selecting Push in Direction adds the Take Snapshot option in Advanced Options.

Figure 11: Add Cloud Sync Task Control and Advanced Options

Settings	Description
Take Snapshot	Displays if Direction is set to Push. Select to take a snapshot before transferring data to the cloud storage provider.
Create empty source dirs on destination after sync	Select to create an empty source directory in the cloud storage provider folder when pushing data to the cloud provider location, or in TrueNAS if pulling data from the cloud storage provider.
Follow Symlinks	Select to follow symbolic links (symlinks) and copy the items to which they link.
Pre-Script	For advanced users. Enter a script to execute before running sync. See the Cloud Sync tutorial for more information.
Post-Script	For advanced user. Enter a script to execute after running sync. See the Cloud Sync tutorial for more information.
Exclude	Enter a list of files and directories to exclude from sync. Separate entries by pressing `Enter`. Examples of proper syntax to exclude files/directories are: `photos</code>` excludes a file named photos `/photos`> excludes a file named photos from root directory (but not subdirectories) `photos/` excludes a directory named photos `/photos/` excludes a directory named photos* from root directory (but not subdirectories). See rclone filtering for more details about the `--exclude` option.

Advanced Remote Options

The Advanced Remote Options settings are for advanced users to configure remote encryption and transfer bandwidth speed and limit.

Figure 12: Advanced Remote Encryption Options

Settings	Description
Remote Encryption	Select to use rclone crypt encryption during pull and push transfers. Selecting PUSH in Direction encrypts files before transfer and stores the encrypted files on the remote system. Files are encrypted using the encryption password and encryption salt values. Selecting PULL decrypts files stored on the remote system before the transfer. Transferring the encrypted files requires entering the same encryption password and encryption salt used to encrypt the files. Additional details about the encryption algorithm and key derivation are available in the rclone crypt File formats documentation.
Filename Encryption	Selected by default. When selected, the pull and push tranfers encrypt or decrypt file names with the rclone Standard file name encryption mode. The original directory structure of the files is preserved. When enabled, file names are encrypted, file names are limited to 143 characters, directory structure is visible, and files with identical names have identical uploaded names. File names can use sub-paths, single copy files, and shortcuts to shorten the directory recursion. When disabled, encryption does not hide file names or directory structure, file names can be 246 characters long, use sub-paths, and copy single files.
Encryption Password	Enter the password to encrypt and decrypt remote data. Warning: Always securely back up this password! Losing the encryption password results in data loss.
Encryption Salt	Enter a long string of random characters for use as salt for the encryption password. Warning: Always securely back up the encryption salt value! Losing the salt value results in data loss.
Transfers	Enter the number of simultaneous file transfers. Enter a number based on the available bandwidth and destination system performance. See rclone –transfers.
Bandwidth limit	Enter a single bandwidth limit or bandwidth limit schedule in rclone format. Separate entries by pressing <kbdEnter. Example: 08:00,512 12:00,10MB 13:00,512 18:00,30MB 23:00,off. You can specify units with the beginning letter b, k (default), M, or G. See rclone –bwlimit.