Cloud Sync Tasks stuck on " Running" ? Google Drive

[FrancoisC]

Hello all!

Anyone experienced this:

My Google Drive Sync Task are never "SUCCESS". It only shows running although it seems to be complete... ETA 0s.

Any idea?
How can I verify it's completed (logs?)

Thank you!

Regards,

Francois

 

[binarybcc]

Same issue. Trying to copy a google drive directory and subdirectories to truenasscale.

 

[brahmy]

I experienced this with Google Photos. Just to leave a trail of breadcrumbs for others: what I think happened is that the initial (very first) sync butted up against a Google API limit (10,000 hits per day). Hard to troubleshoot, no error, TrueNAS was just stuck at 100%. I aborted the task after many hours, but after another day of getting stuck at 100%, it's now syncing daily since it's synced that initial slug of photos.

Since I have it set up as SYNC I'm assuming that if it was at up as COPY it would hit that daily API limit every time.

Hope this helps someone else... Be patient.

 

[brahmy]

I experienced this with Google Photos. Just to leave a trail of breadcrumbs for others: what I think happened is that the initial (very first) sync butted up against a Google API limit (10,000 hits per day). Hard to troubleshoot, no error, TrueNAS was just stuck at 100%. I aborted the task after many hours, but after another day of getting stuck at 100%, it's now syncing daily since it's synced that initial slug of photos.

Since I have it set up as SYNC I'm assuming that if it was at up as COPY it would hit that daily API limit every time.

Hope this helps someone else... Be patient.

Update (sorry, can't figure out how to edit): even if there are just a few new files to transfer, the checking logic will hot the 10,000 API request limit... So if you have more than 10k photos in Google, the sync will get stuck every time. I'll figure out how to report this as a bug or issue somewhere...

 

[brahmy]

Just one more piece of info from my logs. Transferred 12 files overnight but tried to check 19k.

<6>INFO :
Transferred: 6.346Mi / 6.346 MiByte, 100%, 22.256 KiByte/s, ETA 0s
Errors: 128 (retrying may help)
Checks: 19133 / 19133, 100%
Transferred: 12 / 12, 100%
Elapsed time: 10h42m24.0s

 

[brahmy]

Another update, I've submitted a bug report here: https://ixsystems.atlassian.net/browse/NAS-119154

After four straight days of hitting a 10k API limit, I was able to sync 22 photos today in 8,400 API requests... so either the sync logic finished working through the complete file/folder backlog, or it's hit-and-miss every day with how many API requests it takes to sync stuff.

PS. Sorry if this keeps bumping the thread. I like leaving a trail of breadcrumbs troubleshooting as I know it can be useful for others in the future.

 

[scrool]

From what I understand TrueNAS uses rclone for Cloud sync. In the case of Google Drive I found following information on their documentation:

When you use rclone with Google drive in its default configuration you are using rclone's client_id. This is shared between all the rclone users. There is a global rate limit on the number of queries per second that each client_id can do set by Google. rclone already has a high quota and I will continue to make sure it is high enough by contacting Google.

It is strongly recommended to use your own client ID as the default rclone ID is heavily used.

I assume that instead of rclone's credentials, iXsystems one is used here. This quote comes from Making your own client_id and it provides nice guide how to get OAuth Client ID and OAuth Client Secret.

After you follow that, you'll need one more component for TrueNAS - token. For that I have installed rclone on my workstation and ran `rclone config` and went through the wizard, where I entered my newly created client_id and client_secret. In browser I have authrorized access to my account and I ended up having token in the terminal. It is whole json - within curly braces.

Finally I used all these three components to create new Cloud credential in TrueNAS.

First test with this sync works. I'm not sure if that token will be refreshed automatically or at all.