Does zfs verify checksums when using cp?

[bazantino1]

I wonder if i use cp to copy files within dataset/pool/system, does ZFS perform checksum verification of copied files or do i have to do this manually?

I've searched for answer, but haven't found a clear statement of how it works. Only declarations that zfs verify checksums on read.

 

[joeschmuck]

As I understand it, when you use the cp command (or move or rename), the data will be read and checked as it's read, then it will be written and will be verified by reading the data it wrote was written correctly. If it didn't then ZFS would not be a good file system for error free data retention.

 

[Ericloewe]

I wonder if i use cp to copy files within dataset/pool/system, does ZFS perform checksum verification

Yes, always.

I've searched for answer, but haven't found a clear statement of how it works. Only declarations that zfs verify checksums on read.

What more do you want to know, specifically? The checksums are stored in the parent block pointer and compared against the calculated checksum from whatever the disks sent back to ZFS.

 

[winnielinnie]

Yes, always.

I believe the answer is more nuanced than that?

From what I understand, any time a record is read from a dataset, ZFS always confirms its integrity by hashing it against its stored checksum. A tool like "cp" will invoke a read-and-check operation by ZFS, as will "rsync", as will any application that reads the data into RAM.

However, after this point, it's out of ZFS's hands to verify that the copied file matches the checksum of the original file. This is because you could be "cp'ing" to a different filesystem, or even to a different dataset (in which new records will be written with their own checksums if the recordsize property is different), or even to the same dataset, which once again equally applies (unless using deduplication.)

Since ZFS is not "file-based" (but rather "block-based"), it doesn't verify the file's integrity against the original: it just verifies the existing record(s)' integrity when it is read into RAM. We assume the "cp'd" file will be a 100% match to the original file, since if ZFS gives the greenlight that the records read into RAM check out (yay!), and the write operation to the destination filesystem did not hit any I/O errors (whew!), then there's no reason to think that the copied file will have undergone any corruption.


Here's an example of where a "cp" of a file ends up corrupted on the destination:

1. File exists on ZFS dataset (checksums of its records are also stored)
2. "cp" command is issued to copy the file to another filesystem
3. All records of this existing file are read into RAM, ZFS checksums are okay!
4. From RAM (all good!) the data of this file is written to the destination filesystem
5. The destination filesystem (Ext4, NTFS, XFS) is on a failing device and a few bits are incorrectly written
6. ZFS has no way to know about this. It already "did its part".
7. If you don't compare the checksums between the two files, you'll be none-the-wiser of "corruption"

When it comes to snapshots and replications, then ZFS plays its part in every step of the process, unlike other end-user applications and commands, such as "cp".

I could be way off base and would love to be ridiculed!

* Unlike "cp", when you use "rsync", it confirms that the transferred chunks written to the destination match the checksum of the same chunks read from the source. (This is a "one-time" check to assure the files were written correctly during transfer and has nothing to do with stored data integrity after-the-fact.) "cp" does no such thing. So if you want better peace of mind, use "rsync" (rather than "cp") to copy files when feasible.

EDIT: I realized the OP was only concerned with ZFS to ZFS?

 

[Ericloewe]

You make a valid and important distinction between the block checksums (what ZFS deals with) and the file checksums. To some extent, it's beyond what a filesystem can do without encroaching on the rest of the OS and user applications, but it's important to keep exactly that in mind.

@bazantino1 can clarify, but I took the question to mean something like "is there a sequence of actions with ZFS that results in data being returned that was not checked against its checksum", to which the answer is "not unless you go far out of your way to disable it". If the concern is a different one, that's something we can address too.

 

[joeschmuck]

LOL, I knew this would happen. Many of us tend to maybe get a little too detailed in our answers when I feel a simple answer will be adequate. But I do love all the details, it really is great to spread the knowledge.

I think the OP really just wanted to know if copying a file using cp would do it with a verification that it was copied accurately. Of course maybe they did want more details.

 

[bazantino1]

Sorry for being unclear in my question.

What more do you want to know, specifically? The checksums are stored in the parent block pointer and compared against the calculated checksum from whatever the disks sent back to ZFS.

I wasn't sure if there is a collaboration between the command and the file system. What I mean is, does ZFS know that it is making a copy, or does it just know that it has been asked to read some data (and verify it against a written checksum) and then write some (for filesystem unrelated) data elsewhere, assume it is good, and calculate a new checksum for it.

In this second case, the only thing that would protect the data would be ECC memory.

However, after this point, it's out of ZFS's hands to verify that the copied file matches the checksum of the original file. This is because you could be "cp'ing" to a different filesystem, or even to a different dataset (in which new records will be written with their own checksums if the recordsize property is different), or even to the same dataset, which once again equally applies (unless using deduplication.)

That's exactly what was my concern.

EDIT: I realized the OP was only concerned with ZFS to ZFS?

Yep, only ZFS to ZFS. Otherwise i always verify with rsync.

You make a valid and important distinction between the block checksums (what ZFS deals with) and the file checksums.

It doesn't really matter to me whether it's at the block level or the whole file level, as long as ZFS knows it's making a copy and somehow verifies it or just treats it as two separate operations.

I think the OP really just wanted to know if copying a file using cp would do it with a verification that it was copied accurately.

I know that under Linux and, for example, ext4 will not and I was curious if, since ZFS is so secure, would it work differently here.

 

[bazantino1]

ZFS knows it's making a copy and somehow verifies it or just treats it as two separate operations.

*ZFS knows it's making a copy and somehow verifies it. If it just treats it as two separate operations then I need to use rsync.

I don't see edit post option :/

 

[Ericloewe]

So the concern boils down to applications potentially corrupting data. There’s not much that ZFS can do there, since the abstraction used does not foresee such checks by the filesystem (which would really break down as soon as things deviate from your nominal scenario).
There is one exception, namely the kernel copy_range function, but I don’t think any filesystem does a readback to confirm all is well.
This is one of those scenarios where you sorta have to trust your computer, on the premise that a lot of things would be obviously broken if data was being corrupted before reaching the destination disk.

 

[bazantino1]

Ok, thanks for your help.

 

[danb35]

ZFS knows it's making a copy and somehow verifies it

ZFS doesn't check the checksum of the copy against the checksum of the original. If you're concerned about corruption "in flight" (e.g., through defective RAM if you aren't using ECC), ZFS won't address that issue in this context.