File copy reliability

[tony95]

I am adding a disk to my Z2 pool and this is requiring me to copy 38TB to another pool and then copy it back. Typically with NTFS I can be sure at least some data will be corrupted with such a large copy. With NTFS I usually do a checksum compare after copying data. Is this necessary with ZFS? Is there a built-in mechanism to ensure data is copied accurately such as a checksum on both ends or do I need to worry about it the same as NTFS?

 

[Patrick M. Hausen]

Use zfs send|zfs receive to copy entire datasets and you can be sure that the data is ok if you don't get an error, anywhere. You can do an additional zpool scrub of the destination afterwards if you like.

 

[tony95]

These copies are being done from BeyondCompare in windows, but they must be getting handed off to some process local to TrueNAS because there is no network activity at all.

 

[HoneyBadger]

Is there a built-in mechanism to ensure data is copied accurately such as a checksum on both ends

Unless it is explicitly stated otherwise (eg: vdev removal) ZFS is always checksumming your data on read and write, but a quick zpool scrub after each step of the copy certainly won't hurt anything and will help give you extra assurance.

but they must be getting handed off to some process local to TrueNAS because there is no network activity at all.

Samba 4.10 introduced support for Server Side Copy (SSC) which lets the local (TrueNAS) Samba server handle the actual copy without needing the network round-trip.

Server-Side Copy - SambaWiki

wiki.samba.org

Your client software (BeyondCompare) might generate some traffic afterwards if it's calculating client-side hashes (MD5/SHA) to verify file integrity, but I'd have no issues myself copying large chunks of data directly on TrueNAS and then just doing a scrub afterwards.

 

[tony95]

Unless it is explicitly stated otherwise (eg: vdev removal) ZFS is always checksumming your data on read and write, but a quick zpool scrub after each step of the copy certainly won't hurt anything and will help give you extra assurance.
...

I have been doing checksums with TrueNAS for a while and I stopped because I never saw errors after a write that completed successfully and I did see file copy errors where I believe NTFS would have just let the corrupted data sit in the target directory so I suspected this is what was happening. This will save me a lot of time and effort knowing I can trust the file system. Pretty sweet.

 

[Mlovelace]

If the file transfer is all on the same LAN pushing the data around with zfs send | zfs receive piped through netcat will be significantly faster.

Replication Stream Compression

Hi All, This is just a bit of knowledge sharing so hope somebody finds this helpful. For a long time now all my systems have been using LZ4 compression during replication as after testing (a long time ago) this seemed to be a good idea. I must add that all my datasets are LZ4 by default...

www.truenas.com