SOLVED Can I create a Cloud Sync Task to Google Drive to run every minute

[CheeryFlame]

Hello,

I just built a NAS and finished copying all my data to it. I've setup all my tasks but the cloud sync. I was using the Google Drive app on Mac before which was great because it would backup my files in real time once a change was detected. This doesn't seem possible with TrueNAS but I wonder if I could do a SYNC custom schedule set to run every minute? This seems like the closest thing to in real time cloud sync.

Will this analyze my whole drive for changes every minute or will this know which files have changed in the last minute and only sync those files? Are there downside speaking of performance?

Thank you!

 

[winnielinnie]

I don't believe rclone (the backend used for TrueNAS Cloud Sync Tasks) supports real-time continuous syncing.

The problem with creating a sync task every minute is you are very likely to overlap sync jobs with the same source-destination, which could cause issues, unless TrueNAS checks to see if a sync task is currently running before starting a new one.

Will this analyze my whole drive for changes every minute or will this know which files have changed in the last minute and only sync those files? Are there downside speaking of performance?

It will scan the entire directory tree (whatever path you configured as the source) for any new files and changes in modification times or sizes of existing files, for every file. This is how it determines what must be transferred to the destination (including "modified" files.)

 

[CheeryFlame]

Thank you for replying, I guess the best way is to try it and if it doesn't work I could just change it to every hour.

Configuring my sync task brought up a concern to my files already present on Google Drive. I have made a "Backups" folder and dragged everything I had on my drive into that folder so I have a backup in case the sync erases some work files I had to edit during the NAS configuration that took me about a month.

SYNC: Files on the destination are changed to match those on the source. If a file does not exist on the source, it is also deleted from the destination.

From what I understand since my TrueNAS backup is in the root of my Google Drive, would this task be removing the Backups folder I created containing my Google drive backup? Or file does not exist on the source, it is also deleted from the destination is only valid for the files and folders that are the same on source AND destination?

 

[winnielinnie]

I've never used Cloud Sync Tasks with TrueNAS. However, rclone follows the same principles as rsync.

The tooltips and guide are a bit ambiguous.

If it's meant to work similar to rsync, then I assume the following:

"COPY" is like using rsync without the --delete flag. If a file is "new" or "modified" on the source, it will be copied over to the destination (overwriting files of the same name). If there are files on the destination that do not exist on the source, they are left alone on the destination (not deleted.)

"SYNC" is like using rsync with the --delete flag. Same thing as "COPY". Except that if a file exists on the destination, but not on the source, it will be deleted on the destination.

would this task be removing the Backups folder I created containing my Google drive backup?

"SYNC" will result in the exact same directory tree and files on the destination which currently exists on the source, starting from the specified "root" on each end.

If your Google Drive's root folder contains a subfolder named "Backups", and your Google Drive's root folder is specified as the destination, then yes, the "SYNC" task will delete this subfolder and all its files if no such folder exists on your source's specified root.

To get around this, you can create another subfolder inside your Google Drive's root called "truenas", and then use this subfolder as the "destination". (This means "truenas" is the specified root directory on the destination side.) The "SYNC" cannot go any further "up" from here.

But I don't even know if TrueNAS lets you configure this type of granularity in the Cloud Sync Tasks. (EDIT: I think it depends on the provider. Once you select your Google Drive credentials, it might present you with a directory tree on your Google Drive cloud storage.)

 

[CheeryFlame]

Thank you for the detailed answer, I could have made a big mistake here. Then it may not be a good idea to have TrueNAS backed up to the root of Google Drive.

Google Sheets and Google Docs files are updated in real time on Google Drive, I wonder if rclone could break those files? It wouldn't make sense since a .gdoc file weight 2kb and is only a link file. I wonder how those files are stored on Google Drive.

 

[CheeryFlame]

I would also like to know if I do a SYNC PULL from Google Drive and a SYNC Push from TrueNAS, so basically 2 tasks for both ways. Would these tasks get in conflict or will this be the equivalent of using the Google Drive app that sync changes with the newest file no matter on what side I've done the changes?

 

[Davvo]

But I don't even know if TrueNAS lets you configure this type of granularity in the Cloud Sync Tasks. (EDIT: I think it depends on the provider. Once you select your Google Drive credentials, it might present you with a directory tree on your Google Drive cloud storage.)

FWIW, MEGA does.

I would also like to know if I do a SYNC PULL from Google Drive and a SYNC Push from TrueNAS, so basically 2 tasks for both ways. Would these tasks get in conflict or will this be the equivalent of using the Google Drive app that sync changes with the newest file no matter on what side I've done the changes?

I believe that would be ugly, from my understanding it's not possible.
But you could run a VM, share a path with it and use gdrive on that.

 

[winnielinnie]

I believe that would be ugly

I concur. Since with TrueNAS, you're not using the files directly on your Google Drive. You do stuff locally (on TrueNAS), then remotely sync the changes. The scope of "Cloud Sync Tasks" doesn't go beyond that. If you configure a PUSH and PULL for the same source-destination, you could end up inadvertently deleting files.

Google Sheets and Google Docs files are updated in real time on Google Drive, I wonder if rclone could break those files?

I'd imagine if you're PUSHing from TrueNAS to the Google Drive using "SYNC", it may in fact try to overwrite the file you're working on, as it's modification timestamp has changed. (PULLing from Google Drive to TrueNAS might work without issue, since it'll just pull in the latest version of the document.)



The purpose of Cloud Sync Tasks is meant specifically to use cloud storage as a place to store or retrieve backups.

Trying to make it work with "real-time" updates and using cloud office suite software breaks its scope.

 

[CheeryFlame]

For anyone wondering how I've end up setting up my cloud sync is that I'm pulling the stuff from Google Drive that I need to keep on Google Drive, mostly shared folders with other people like work, clients, projects. I then access the latest version of those files on my computers with the Google Drive desktop app. Everything that I don't need the latest version and that I work locally like my media library is managed directly on my server and is pushed to Google Drive strictly as a backup and viewing purposes.

I still have one theory that may work but seems to defeat the purpose of my server which would be to use Google's desktop app to sync in real time the data on the server by accessing the files through the SMB share on Windows. It's kind of weird since the computer now acts as the server instead of your server and your server becomes a file tree with wasted ressources. Another downside is that it would waste your computer ressources since your server should be doing it and also it doesn't feel like a stable option. If someone is willing to try it and share their experience, go on!

 

[Davvo]

I still have one theory that may work but seems to defeat the purpose of my server which would be to use Google's desktop app to sync in real time the data on the server by accessing the files through the SMB share on Windows. It's kind of weird since the computer now acts as the server instead of your server and your server becomes a file tree with wasted ressources. Another downside is that it would waste your computer ressources since your server should be doing it and also it doesn't feel like a stable option. If someone is willing to try it and share their experience, go on!

That's why I suggested you using a VM insider your server.

 

[CheeryFlame]

That's why I suggested you using a VM insider your server.

Damn totally misread that part, that's a neat idea! I'll try it for sure!