Cloud Sync + Onedrive Business issues

[gummigreger]

Hi!

I'm managing a couple of Truenas-systems here at work, and have been working on our backups a bit.
We have one Truenas Core-box that we actually work against with SMB-shares and Active Directory-ACL's(connected to our Domain controller). We'll call this one TRUENAS1.

Then I've deployed a second Truenas Core-box which only serves as on-site backup for TRUENAS1, let's call this one TRUENAS2, and this one is of course also connected to our Domain, but no SMB-shares are exposed to the users to prevent any files being open when running sync tasks & such.

TRUENAS1 sends ZFS snapshots on a regular basis to TRUENAS2(Every evening at 20.00) via a built-in Replications Task(PUSH).

Then I've set up a Cloud Sync task to sync(PUSH/SYNC) all the data from TRUENAS2 to our Onedrive Business-storage since we have plenty of space there(E5-licenses so basically unlimited storage with some caveats). At first I had it run directly from TRUENAS1 but I figured it would fail if any files were left open so I moved the task to TRUENAS2 for peace of mind.

I started the initial upload task Thursday last week and now after a few days off from work I realize the task has been restarting several times(it's now at 57%). I've now changed the replication & cloud sync tasks to only be once a month(just until I've gotten the initial upload done) in case either of the operations are messing with me.

I'm a bit worried that the SYNC-task isn't ignoring files that are already on the storage correctly for some reason though, since if I go to the Onedrive storage-folder I can see roughly ~1TB of data in that folder. When i check the source-folder on the TRUENAS2 it's roughly 1.38TB of data. My understanding was that with SYNC it should skip uploading files that are aleady on the target destination and only upload files that have changed, been removed or been added on the source since the last task. Have I misunderstood this? If it's gonna upload all the 1.38TB of data every time I run the task I'll have to solve this some other way, I was so content that I'd found the ultimate solution for an off-site backup without having to get expensive cloud storage when we already have the Onedrive storage already.

 

[gummigreger]

Found what I'd missed. I had a subfolder that contained a daily backup of two Hyper-V-servers that wasn't excluded correctly, somehow it started to reupload those everytime they got updated.

For anyone else reading this later, I hadn't understood the rclone exclusions correctly. The "Directory/Files"-field in my Cloud Sync task had the following: /mnt/Backup/Replication/Server and so I assumed the exclusion should look like so: /mnt/Backup/Replication/Server/FOLDERTOBEEXCLUDED/** which was wrong. Inputting FOLDERTOBEEXCLUDED/** worked. So everything in the "Directory/Files"-field should be omitted when entering exclusions which wasn't obvious to me at least.