Cloud Sync

[svtkobra7]

General question regarding Cloud Sync ...

Let's say you have the following scenario: (a) Push Sync with a source dataset of several hundred GBs or even TBs, (b) Provider = Google Drive (unlimited), (c) max upload bandwidth = 70 mbps

If the sync task is interrupted for some reason (whether it be unintentional outage or intentional reboot) and you receive a failed status, and if the sync job has been running for days and should be at X% of the initial sync, when resumed does does it pick up at X% or 0%???

I ask as I'm trying to determine whether it best to (a) set up sync tasks which may take more than a week to complete initially, or (b) if I need to create many smaller tasks, i.e. I don't want to start syncing, have an outage after 7 days, have to start the sync job over and realize I'm starting at 0%.

Thanks in advance for any help you can provide.

 

[dlavigne]

Did you find the answer to your question?

 

[svtkobra7]

Did you find the answer to your question?

Thanks for the follow up - I did not, but would still love to know. ;)

 

[svtkobra7]

[apologies - incomplete prior response - I may have thought of a way to answer my own question]

Since Cloud Sync is based on rclone, and based upon my knowledge thereof, my unsubstantiated guess would be that resumption of the Cloud Sync "task" means you pick up at X% (not 0%), as that would be how rclone operates (independently of FreeNAS).

It just struck me that supposition could be confirmed through the following approach (can't believe I didn't think of this prior):

• Create/execute/stop a push Cloud Sync task (where sync is purposely incomplete)
• Create/execute a pull Cloud Sync task for the above push
  
• The payload of the pull sync would answer the question:
  • If no data pulled, or an error encountered, then the answer to my question is that an interrupted push sync would result in starting at 0% upon re-initiation.
  • If some data pulled, then the answer to my question is that an interrupted push sync would result in starting at X% upon re-initiation.

I would love to test and report back; however, unfortunately I cannot at the moment:

• I knew that I would be taking my server offline for network "retooling" when I asked the question. As luck would have it, I'm in the middle of that planned outage now.
  
• I reverted from 11.2 BETA2 to 11.1 U5 (but could flip back when I'm back up). This functionality, i.e. inclusion of Google Drive as a Cloud Sync provider, was one of the primary drivers for moving to a beta release as opposed to remaining on a stable release channel (as would be my preference, generally). Lame rationale (11.1 U5 => 11.2 BETA2) I know.

That being said, I would imagine that I'm not the only FreeNAS user who would appreciate clarity here? There must be other users with no ISP data cap, unlimited cloud storage, and the desire to sync multi-TB datasets to the cloud. Again, thanks for your kind follow up.

 

[ian.m.taylor]

I had a long cloud sync fail, but with Backblaze as a provider. I would imagine the results would be the same for Google drive, though. Here's what happened:

1. There were files in my Backblaze bucket, that were successfully copied.
2. When I restarted the task, it reported starting at 0%, but the total number of files it actually needed to copy was smaller than before.

In essence, rclone will say it's starting from scratch, but with a smaller workload to complete (only the remainder). Does that answer your question?