Cloudsync - Transfer Mode confusion

[ryanwclark]

I found that as of 11.3 the functionality in CloudSync was changed or at least the documentation would lead me to believe it was changed from 11.2 to 11.3. I am currently using 11.3 but want the behavior described in 11.2, as I would prefer not to copy all files at my various cloud storage providers as described in the 11.3 documentation but rather copy only the new files and skip already existing identical files. I also found the documentation on rclone.org for rclone copy, which I thought (could be way off here) was being used for this feature in FreeNas/TrueNas.

Does anyone have any additional insight?


Rclone.org

rclone copy

Copy files from source to dest, skipping identical files.

rclone.org

Sync the source to the destination, changing the destination only. Doesn't transfer unchanged files, testing by size and modification time or MD5SUM. Destination is updated to match source, including deleting files if necessary.


11.2
Sync (default) makes files on destination system identical to those on the source. Files removed from the source are also removed from the destination, similar to rsync --delete.

Copy copies files from the source to the destination and skips files that are identical, similar to rsync.

Move copies files from the source to the destination and deletes the source files after the copy, similar to mv.


11.3
SYNC: Files on the destination are changed to match those on the source. If a file does not exist on the source, it is also deleted from the destination. There are exceptions to this behavior.

COPY: Files from the source are copied to the destination. If files with the same names are present on the destination, they are overwritten.

MOVE: After files are copied from the source to the destination, they are deleted from the source. Files with the same names on the destination are overwritten.

 

[sroot]

I'm here with the same question, posting so I might remember to come back if I find the answer today, or at least get a message when someone else does

 

[sroot]

I didn't find an answer so I ran some tests. Actually just testing 'sync' as that did what I needed.

I would prefer not to copy all files at my various cloud storage providers as described in the 11.3 documentation but rather copy only the new files and skip already existing identical files.

I want to do this too

I'm using Backblaze
https://www.truenas.com/docs/core/tasks/cloudsynctasks/ says

Sync

keeps all the files identical between the two storage locations. If the sync encounters an error, files are not deleted in the destination. This includes a common error when the Dropbox copyright detector flags a file as copyrighted.

Note that syncing to a Backblaze B2 bucket does not delete files from the bucket, even when those files have been deleted locally. Instead, files are tagged with a version number or moved to a hidden state. To automatically delete old or unwanted files from the bucket, adjust the Backblaze B2 Lifecycle Rules.

So I created a new data set with some some small pngs and a 300mb file.
I synced once, it worked.
I copied the files, synced again, it worked.
I synced again - result was almost instant so I'm inferring it didn't copy the files again (it took 5-10mins when there are new files)
I copied the files, deleted the originals, synced again, it worked.

For deleted files, just like https://www.truenas.com/docs/core/tasks/cloudsynctasks/ said, the original file name got a suffix which expanded to show a hidden file of zero bytes and the original file. This is what I need. It means the sync can't remove files and I do that manually. It also lets me see the files. Here's a screen shot showing the end result where;

• 1.png has been deleted, the (2) link expanded to show both the original version of the file and the 0 byte place holder that infers it has been deleted from the source
• 2.png, 3.png - also deleted and links not expanded.
• 300-1.iso has been deleted, link expanded to show original and zero byte place holder
• 300-2.iso is a regular file, present through several repeated sync tests, not deleted and clicking the blue link allows the file to be downloaded