TrueNAS SCALE Nightly Development Documentation
This content follows experimental early release software. Use the Product and Version selectors above to view content specific to a stable software release.

Adding a Google Photos Cloud Sync Task

11 minute read.

Last Modified 2024-07-18 11:56 EDT

Google Photos cloud sync tasks in TrueNAS SCALE use the rclone backend for the Google Photos API to authenticate credentials and transfer data.

Configuring a Google Photos cloud sync task is a multi-part procedure where you:

Plan your deployment and selecting a local dataset.
Generate Google API credentials on the Google Cloud API dashboard.
Install rclone and generate a token on your remote client OS.
Add Google Photos cloud credentials on TrueNAS SCALE.
Configure the cloud sync task on SCALE.

Before You Begin

Review your storage and data protection requirements and consider your options before setting up a Google Photos cloud sync task. Refer to the rclone Google Photos backend documentation for more information on using rclone to sync Google Photos, including standard options and limitations of the Google Photos API, that might help you plan your deployment.

Consider how you want to manage your media files on Google Photos and in your local dataset. Decide on the cloud sync task direction and transfer mode, remote folder to target, and new or existing local dataset to pull to or push from, that best fit your needs.

Choosing a Sync Direction and Mode

A Google Photos cloud sync task can either pull files from Google Photos to a local dataset on TrueNAS SCALE or push local files to Google Photos. Select the direction that best fits the way you intend to manage your media files.

Choose to pull data from Google Photos if you prefer to manage media files via the Google Photos UI and use the local dataset as a backup target.

Choose to push data to Google Photos if you prefer to manage media files in the local dataset and use Google Photos as a cloud backup location.

Next, select the data transfer mode that best fits the way you want to manage file retention between the source and destination. There are three options:

SYNC - Select to change files on the destination to match those on the source. If a file does not exist on the source, it is also deleted from the destination.
COPY - Select to duplicate each source file into the destination. If files with the same names are present on the destination, they are overwritten.
MOVE - Select to transfer files from the source to the destination and delete source files. Copies files from the source to the destination and then deletes them from the source. Files with the same names on the destination are overwritten.

Choosing a Target Folder

After choosing the direction and mode for your cloud sync task, choose the remote Google Photos folder that rclone targets to sync data. Because of the way rclone interacts with the Google Photos API, each target folder option has specific file management and structure requirements. This is due to the way rclone interacts with the Google Photos API. A cloud sync task cannot target the root level folder (/).

Folder	Recommended	Direction	Description
/album	Yes	Push or Pull	Use this option for push tasks or if you prefer to organize the Google Photos library by sorting media files into one or more discrete albums. All files must be in distinct albums or child directories of the local dataset. Media files uploaded to Google Photos but not assigned to an album are not pulled to a local dataset mirroring /album. Files uploaded to the base level of the local dataset instead of a child directory (an album) are not pushed to /album.
/media/all	Yes	Pull	Use this option if you prefer to use the Google Photos library as single directory, without organizing media files into discrete albums. The local dataset of a /media/all sync task contains all media files stored on the Google Photos account at the same level, with no further organization into subdirectories. Using /media/all allows you to upload new files to Google Photos, without needing to organize them into albums, and then pull them to TrueNAS.
/upload	No	Push	Media files pushed from the local dataset to /upload are then uploaded to Google Photos and not sorted into an album. Because /upload is a temporary storage location, it can not accurately synchronize from one task to the next. Pushing to this folder does not preserve metadata and can result in duplicated files, poor performance, file name instability.

Selecting the Dataset and Organizing Files

Select TrueNAS SCALE local dataset or create a new one to use as the source or destination.

Creating a Dataset

To create a basic dataset, go to Datasets. Default settings include those inherited from the parent dataset.

Select a dataset (root, parent, or child), then click Add Dataset.

Enter a value in Name.

Select the Dataset Preset option you want to use. Options are:

Generic for non-SMB share datasets such as iSCSI and NFS share datasets or datasets not associated with application storage.
Multiprotocol for datasets optimized for SMB and NFS multi-mode shares or to create a dataset for NFS shares.
SMB for datasets optimized for SMB shares.
Apps for datasets optimized for application storage.

Generic sets ACL permissions equivalent to Unix permissions 755, granting the owner full control and the group and other users read and execute privileges.

SMB, Apps, and Multiprotocol inherit ACL permissions based on the parent dataset. If there is no ACL to inherit, one is calculated granting full control to the owner@, group@, members of the builtin_administrators group, and domain administrators. Modify control is granted to other members of the builtin_users group and directory services domain users.

Apps includes an additional entry granting modify control to group 568 (Apps).

ACL Settings for Dataset Presets

	ACL Type	ACL Mode	Case Sensitivity	Enable atime
Generic	POSIX	n/a	Sensitive	Inherit
SMB	NFSv4	Restricted	Insensitive	On
Apps	NFSv4	Passthrough	Sensitive	Off
Multiprotocol	NFSv4	Passthrough	Sensitive	Off

If creating an SMB or multi-protocol (SMB and NFS) share the dataset name value auto-populates the share name field with the dataset name.

If you plan to deploy container applications, the system automatically creates the ix-applications dataset, but this dataset is not used for application data storage. If you want to store data by application, create the dataset(s) first, then deploy your application. When creating a dataset for an application, select Apps as the Dataset Preset. This optimizes the dataset for use by an application.

If you want to configure advanced setting options, click Advanced Options. For the Sync option, we recommend production systems with critical data use the default Standard choice or increase to Always. Choosing Disabled is only suitable in situations where data loss from system crashes or power loss is acceptable.

Select either Sensitive or Insensitive from the Case Sensitivity dropdown. The Case Sensitivity setting is found under Advanced Options and is not editable after saving the dataset.

Click Save.

Review the Dataset Preset and Case Sensitivity under Advanced Options on the Add Dataset screen before clicking Save. You cannot change these or the Name setting after clicking Save.

Configure file management structure inside the local dataset (for push tasks) or albums in the Google Photos (for pull tasks) as required by your direction, mode, and target selections (see above).

Creating the API Credentials

On the Google API dashboard, click the dropdown menu to the right of the Google Cloud logo and select your project.

If you do not have a project, click NEW PROJECT and enter a value in Project name, Organization, and Location. Click Create.

Enable API

After you select your project, click Enabled APIs & Services on the left menu, then click + ENABLE APIS AND SERVICES.

Enter photos library api in the search bar, then select Photos Library API and click ENABLE.

Configure Authentication

Click OAuth consent screen on the left menu, select EXTERNAL, then click CREATE.

Enter a value in App name and User support email.

Enter an email address in Developer contact information, then click SAVE AND CONTINUE.

Figure 7: Enter Developer Contact Information

Continue to the Test users section and click + ADD USERS, enter your email address, then click ADD.

On the OAuth consent screen, click PUBLISH APP under Testing and push the app to production.

Can I leave the app in testing mode?

You can leave the app in testing mode, but testing app credentials expire after seven days. Cloud sync tasks fail when credentials expire.

Create Credentials

Click Credentials on the left menu, then click + CREATE CREDENTIALS and select OAuth client ID.

Select Desktop app in the Application type dropdown, then enter a name for the client ID and click CREATE.

Copy and save your client ID and secret, or download the JSON file.

Configuring Rclone

Download rclone for your client OS and open it in a command line utility following the rclone installation instructions. The example photos in this article use Powershell in Windows OS.

Enter rclone config, then enter n to create a new remote.

Enter a name for the new remote, then enter the number from the list corresponding to Google Photos.

Enter the client id and secret you saved when you created the Google Photos API credentials, then enter false or press Enter to allow the Google Photos backend to request full access.

Figure 14: Configure rclone Client ID and Secret

Do not edit the advanced config.

Enter y to authenticate rclone using the web browser. A browser window opens to authorize rclone access. Click Allow.

In the command line, enter y to confirm rclone uploads media items with full resolution and complete the configuration.

Copy and save the type, client_id, client_secret, and token, then enter y to save the new remote.

Adding Google Photos Cloud Credentials

In the TrueNAS Web UI, go to Credentials > Backup Credentials and click Add in the Cloud Credentials widget.

Select Google Photos as the Provider and enter a name.

Paste the Google Photos API client ID and client secret in the OAuth Client ID and OAuth Client Secret fields.

Paste your rclone token into the Token field.

Click Verify Credential to ensure the credentials are valid, then click Save.

Creating the Cloud Sync Task

Go to Data Protection > Cloud Sync Tasks and click Add. The Cloud Sync Task Wizard opens.

Select the Google Photos backup credentials from the Credentials dropdown list.
Click Verify Credential to ensure the credentials are valid then click Next.
Select the Direction as PUSH or PULL and select the Transfer Mode as SYNC, COPY, or MOVE. Select the Google Photos location to back up data to or from in Folder. Browse to and select the album folder or enter /album.
Figure 18: Cloud Sync Task Wizard - What and When
Select the local dataset in Directory/Files. This is the dataset sent to Google Photos for push tasks or the write destination for pull tasks.
Push tasks containing media files saved to the local dataset root level fail with the error: Failed to sync: can’t upload files here.
Save files to child directories, not to the root level of the TrueNAS dataset. Directories under the local dataset correspond to albums in the Google Photos library.
Enter a Description for the cloud sync task.
Select the time to run the task from the Schedule options.
Click Save to add the task.

TrueNAS adds the task to the Cloud Sync Task widget with the status Pending, until the task runs on schedule.

Click Dry Run to test the task by connecting to Google Photos and simulating transferring a file. During a dry run, TrueNAS sends or receives no data. A dry run can report successful even for a task that fails to transfer data due to misconfiguration

Click Run Job to start the cloud sync task immediately.

Troubleshooting

If a Google Photos cloud sync task fails, go to Data Protection and click the FAILED status in State on the Cloud Sync Tasks widget. Review the logged error message(s). Common error messages for failed Google Photos tasks include:

Failed to copy: can't upload files here

Problem: A push task is trying to upload files to the root level / folder or the /media/all folder.

Solution: Reconfigure the push task to target the /album folder (and organize your files into one or more subfolders/albums) or change the direction of the task to pull from Google Photos and target the /media/all folder.

Pulling from the root directory is not allowed. Please, select a specific directory

Problem: A pull or push task is targeting the root level / folder.

Solution: Review Before You Begin above and change the target folder to /album or /media/all. Ensure the selected folder does not conflict with the direction of the task.

Failed to copy: directory not found

Problem: A pull task is targeting the /upload folder.

Solution: The /upload folder functions as a temporary queue for rclone to upload files to Google Photos. rclone cannot pull from /upload. Review Before You Begin above and change the target folder to /album or /media/all. Organize your Google Photos library or local dataset as needed for the selected target.

After reviewing available logs, click Edit on the task and review the configuration. Compare configured options to the requirements in Before You Begin above and correct any issues.

If a pull task is successful but some or all files are missing from the local dataset, review your library organization in Google Photos. Pull tasks configured with /album as the target folder only transfer files organized into albums. Files uploaded to Google Photos but not added to an album are not transferred. Using the Google Photos UI, create one or more albums and add all files to an album then click Run Job to re-run the cloud sync task.