Rsync to Truenas: --checksum function not working as intended

[Benni.blanko]

Hi there,
We are starting to do backups from Windows to Truenas 13-U5.3 with rsync.
Rsync for that reason, that the Windows machines traveling worldwide and VPN connections are sometimes slow and error prune.
At the moment we are doing directory synchronizing by simply using shares via VPN and using a Windows directory synchronizer.
And we discovered that the SMB transfer corrupted several files on our Truenas shares. The files look good (filesize okay, date/time also), but they contain huge portions of only #00 in it).
So we need something more robust and better: Rsync.
I already have a running Rsync configuration from the Windows client to Truenas. Working great.
Rsync-Cygwin version used: 3.2.7 (actual version)

Now I'm simulating a defect file on Truenas; like one of the previously broken SMB transferred files:
*run a rsync synchronisation (source: Windows PC, destination: Truenas RSync)
*files are identical on both system
*now I'm faking a defect file on Truenas by using a hex editor to mangle the file (date, time, size unchanged)
*run rsync from Windows again WITH --checksum (-c) option, but the file differences are not detected; no checksumming on both ends and finding the different file (as far as I can tell)

Soooo... I'm a little out of ideas here....

Here's the last rsync call I'm doing on the Windows box; a little parameter overkill, but that's the result of several tests and reading in the documentation.

rsync -ac -v -H -h -x --no-perms --ignore-times --stats --partial --progress --inplace --append --progress --no-whole-file --itemize-changes --delete-delay --log-file="/cygdrive/c/temp/rsync_to_nas_log.txt" /cygdrive/c/test1 rsync://username@truenas:873/rsyncmodulename/test1/

Number of files: 8 (reg: 6, dir: 2)
Number of created files: 0
Number of deleted files: 0
Number of regular files transferred: 0
Total file size: 13.14M bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 341
Total bytes received: 13

sent 341 bytes received 13 bytes 708.00 bytes/sec
total size is 13.14M speedup is 37,117.21

And two of these 8 files are different on Truenas, so they should be found and copied again. But they are not.

Why isn't rsync --checksum not detecting the different files and copies them again?
Anybody any idea or hint?

Thanks for your support.

 

[winnielinnie]

*now I'm faking a defect file on Truenas by using a hex editor to mangle the file (date, time, size unchanged)

Why? ZFS already creates checksums for every record. Not only will it know about corruption upon accessing the file or running a scrub, but it will attempt to automatically repair it if you use redundancy.

That's one of the main reasons to use ZFS in the first place.

And two of these 8 files are different on Truenas, so they should be found and copied again. But they are not.

Not that it matters (since you should leave file integrity to ZFS), but it's probably because you're using "append".

rsync -ac -v -H -h -x --no-perms --ignore-times --stats --partial --progress --inplace --append --progress --no-whole-file --itemize-changes --delete-delay --log-file="/cygdrive/c/temp/rsync_to_nas_log.txt" /cygdrive/c/test1 rsync://username@truenas:873/rsyncmodulename/test1/

 

[Benni.blanko]

Thanks for your comment.
I'm aware of that. ZFS Is making everything right!

The previously used Windows file synchronizer has produced defect files (several hundred or thousand files in a total of 7TB). ZFS just stores what it get's and observes that with it's checksum. But the files already arrived broken as ZFS wrote them.
A
lot of the big video files contains 10-50% only zeros in Truenas filesystem. I assume the file transfer just broke or timed out, but the file was created with the target size on start of a file copy. So size and date/time are indistinguishable between good and bad files.
Even deleting all files and re-sychronzing them again with some windows tool has the risk of creating faulty files again.
Doing a file read/compare after write even slows down the process again over limited VPN connections. And we have to read/compare everything (TB) over VPN to even find the bad files.

Rsync offers a better protection here as the transfer is secured with checksums (also recovering procedures like transferring only delta of broken files).

But: Rsync is not detecting the different files on source and target files, although it should with the --checksum option.
That's what I don't understand or where I made a mistake here.

 

[winnielinnie]

The previously used Windows file synchronizer has produced defect files (several hundred or thousand files in a total of 7TB). ZFS just stores what it get's and observes that with it's checksum. But the files already arrived broken as ZFS wrote them.

Hopefully you're not seeing something similar to this:

Transferring file via share results in a corrupted file with different checksum each time

I use a SMB share to access my files through ethernet on my local network. Recently, when I tried to copy some files, they all ended up corrupted. For example, this image is obviously corrupted If I use 7zip to get the SHA256 value of this file, it will return a different result each time...

www.truenas.com

A lot of the big video files contains 10-50% only zeros in Truenas filesystem. I assume the file transfer just broke or timed out, but the file was created with the target size on start of a file copy. So size and date/time are indistinguishable between good and bad files.
Even deleting all files and re-sychronzing them again with some windows tool has the risk of creating faulty files again.

...or bad backup software? The incessant runs of zeroes are concerning. Is this software attempting to preallocate the entire file with zeroes before even transferring?

But: Rsync is not detecting the different files on source and target files, although it should with the --checksum option.
That's what I don't understand or where I made a mistake here.

It's in my post above:

Not that it matters (since you should leave file integrity to ZFS), but it's probably because you're using "append".

Rsync offers a better protection here as the transfer is secured with checksums (also recovering procedures like transferring only delta of broken files).

Rsync always uses an integrity check to confirm the file was transferred and written correctly on the destination. This is unrelated to the "-c" option, which computes a hash of the the file on both ends (source and destination) to determine if it should be transferred again.