Snapshots show 0 used even when not true?

[FreeNASBob]

I have a periodic snapshot task enabled every twelve hours with a 6-month lifetime. When I use Windows "Previous Versions" functionality I see several previous versions of a file as expected, but when I use the FreeNAS GUI to browse the snapshots taken on the same dates as the files that appear in Windows' "Previous Versions" all of the snapshots show "0" in the "Used" column. When I filter the "Used" column to show every row that is not "0" I only get some snapshots from months earlier and the used size is much too small to contain all of the changes to those files since then. What am I missing? What's the right way to find the snapshot containing actual changes in the FreeNAS 9.3 GUI?

 

[FreeNASBob]

Here are screenshots of what I'm talking about.

 

[SweetAndLow]

Did you ever modify or delete any data that would require the snapshot to have data.

 

[FreeNASBob]

Did you ever modify or delete any data that would require the snapshot to have data.

Yes. Windows "Previous Versions" only shows files that have changed.

 

[dlavigne]

Which build version (from System -> Information)?

 

[FreeNASBob]

Which build version (from System -> Information)?

FreeNAS-9.3-STABLE-201512121950

It has since come to my attention that when ZFS claims a snapshot "uses" 0 bytes it doesn't mean that a snapshot occupies 0 bytes on disk. Instead it means that there are zero bytes unique to this particular snapshot. This leads to the bizarre situation where taking two snapshots of the same data leads to having two snapshots that both report 0 bytes used, even though they might both actually contain 100 GB of changes to the current filesystem. Because those changes are shared between the two snapshots ZFS reports that they each use 0 bytes.

I can see the logic in that approach, but I think the "used" terminology is ambiguous and will lead to misunderstandings because it is contrary to an already established paradigm where two filesystem pointers refer to the same data on disk (i.e., hard links). In the case of hard links, two file names pointing to the same inode will both report their file size as the size of the data on disk. If two files refer to the same 10 MB object in the filesystem both files will report their size as 10 MB. If one of the files is deleted no disk space is freed because there is still another filename referencing the 10 MB filesystem object. This is the behavior I was expecting with ZFS snapshots, and I expect anybody familiar with hard links would expect the same.

A better terminology might be something to indicate that what's being reported is the number of bytes unique to a particular snapshot, not the actual size of changed data stored in that snapshot.

 

[depasseg]

This leads to the bizarre situation where taking two snapshots of the same data leads to having two snapshots that both report 0 bytes used, even though they might both actually contain 100 GB of changes to the current filesystem.

It's not bizarre, it's just that you don't actually understand what is in a snapshot and what "used" means. Do some more reading about zfs snapshot sizes.
Here's a quick explanation:
https://blogs.oracle.com/observatory/entry/understanding_the_space_used_by

 

[FreeNASBob]

It's not bizarre, it's just that you don't actually understand what is in a snapshot and what "used" means. Do some more reading about zfs snapshot sizes.
Here's a quick explanation:
https://blogs.oracle.com/observatory/entry/understanding_the_space_used_by

To most people who have never encountered Oracle's particular interpretation of the word, used is the adjective form of the third person present uses. Linguistically, 0 bytes used ≡ the filesystem changes referenced by this snapshot occupy 0 bytes on the filesystem.

I think if we used a non-IT example, the bizarre-ness of ZFS' nomenclature would be clear. Imagine we have two construction workers, Bob and Jeff, who share a miter saw between them. Our company inventories tools by person so our inventory shows a miter saw with the same serial number listed under Bob's tools and also under Jeff's tools (like two snapshots referencing the same blocks of data). No person would say, the number of miter saws used by Bob is zero, and the number of miter saws used by Jeff is also zero, and justify that because firing Bob or Jeff wouldn't free up a miter saw. That is what ZFS is doing here.

ZFS' use of the word used is contrary to standard English and contrary to established precedent (even in ZFS) where two different names point to the same data in the filesystem (e.g., hard links). Even in ZFS two file names that point to the same filesystem object will each report their size to be the size of that object even though deleting one will not free any disk space at all.

Code:

[root@freenas /mnt/pool/foo]# dd if=/dev/random of=myfile.dat bs=1024 count=10240                                                                 
10240+0 records in                                                            
10240+0 records out                                                           
10485760 bytes transferred in 0.436770 secs (24007512 bytes/sec)

[root@freenas /mnt/pool/foo]# ls -lh                              
total 20493                                                                   
-rwx------+ 1 root  wheel    10M Jan  1 10:28 myfile.dat

[root@freenas /mnt/pool/foo]# ln myfile.dat myfile2.dat           

[root@freenas /mnt/pool/foo]# ls -lh                              
total 40986                                                                   
-rwx------+ 2 root  wheel    10M Jan  1 10:28 myfile.dat                      
-rwx------+ 2 root  wheel    10M Jan  1 10:28 myfile2.dat

[root@freenas /mnt/pool/foo]# df /mnt/pool                       
Filesystem  512-blocks Used       Avail Capacity  Mounted on                  
pool      13986032957 3069 13986029888     0%    /mnt/pool

[root@freenas /mnt/pool/foo]# rm myfile2.dat

[root@freenas /mnt/pool/foo]# ls -lh                              
total 20493                                                                   
-rwx------+ 1 root  wheel    10M Jan  1 10:28 myfile.dat

[root@freenas /mnt/pool/foo]# df /mnt/pool                       
Filesystem  512-blocks Used       Avail Capacity  Mounted on                  
pool      13986032957 3069 13986029888     0%    /mnt/pool

In conclusion, ZFS' use of the word used is not consistent with how the word is formally defined, nor with how it is used in everyday speech. It is also internally inconsistent with how ZFS treats other cases where two names point to the same filesystem object. That makes ZFS' use of the word used out of the ordinary, and as illustrated by the non-IT example, strikingly so. When something is strikingly out of the ordinary, it is bizarre by definition.

 

[depasseg]

Your assumption in your linguistic example is that "used" is in referenced to some arbitrary previous point in time, vs the reality is that it references against the previous snapshot.
And in your tool example, if you were trying to apply 3-dimensional rules to a 4 dimensional problem, I would also say that you need to read up on 4-dimensional physics to understand how both Bob and Jeff can both use (and not use) a miter saw at the same time.

You are applying Windows logic to ZFS which just doesn't apply.

 

[Bidule0hm]

Ok, but what label do you suggest then?

It is also internally inconsistent with how ZFS treats other cases where two names point to the same filesystem object.

This is the biggest problem to me. If one snapshot then used = changes size, but if 2+ snapshots then used = 0.

Whatever used refers to (the size of the snapshot itself, the size of the changes referenced by the snapshot, both added together, ...) it should always be the same.

I proposed a logical (but less than perfect...) used value in this post: https://forums.freenas.org/index.ph...g-periodic-snapshots.40152/page-2#post-252453 where the rule is: If one snapshot then [used] = [changes size], but if 2+ snapshots then [used (for one snapshot)] = [changes size] / [number of snapshots referencing it] :)

 

[FreeNASBob]

Your assumption in your linguistic example is that "used" is in referenced to some arbitrary previous point in time, vs the reality is that it references against the previous snapshot.

I just assume a snapshot references the blocks that were changed at some point after the snapshot was made, and more than one snapshot can reference the same blocks. If each successive snapshot is only a reference to the previous snapshot, that would mean snapshots are similar in concept to incremental backups rather than differential backups. Is that the case?

And in your tool example, if you were trying to apply 3-dimensional rules to a 4 dimensional problem, I would also say that you need to read up on 4-dimensional physics to understand how both Bob and Jeff can both use (and not use) a miter saw at the same time.

You are applying Windows logic to ZFS which just doesn't apply.

Nobody's trying to use two snapshots at the same time. I don't even think it's possible.

What is "Windows logic"? Is that some reference to making at least a cursory attempt to keep things consistent within an OS environment?

 

[depasseg]

I just assume a snapshot references the blocks that were changed at some point after the snapshot was made, and more than one snapshot can reference the same blocks.

Hence the difference between "used" and "referred" in the zfs article I shared.

Nobody's trying to use two snapshots at the same time. I don't even think it's possible.

You realize that snapshots build on each other right? And when you clone, you are just branching off a snapshot. You can create a clone of a 20TB dataset and the net effect is 0 bytes difference. Hence the ability for Bob and Jeff to use the same saw at the same time.

 

[FreeNASBob]

You realize that snapshots build on each other right?

No. From all the documentation I have read I got the impression that a snapshot was more like a differential backup than an incremental backup (best comparison I could think of), but even then each incremental backup would still report its size to be the size of the difference between it and the previous backup. It wouldn't suddenly report that it was zero bytes in size if the subsequent backup yielded no changes. Only the last would report its size as zero bytes.

 

[jgreco]

This is mainly a question of how snapshots work and how you visualize the process. You're not wrong, but you're also not right. Your expectations of what the system will tell you are simply not matching up with the more practical reality.

 

[FreeNASBob]

Your expectations of what the system will tell you are simply not matching up with the more practical reality.

I agree that it's not how it works, but I can't visualize how it's more practical than letting users know the actual size of a snapshot just like ZFS does with hard links. I can see that it's useful in the specific case where an admin wants to know exactly how much space will be freed by deleting a solitary snapshot, but that seems like an edge case.

Ideally ZFS would report the actual size of the data represented by a snapshot and the size of the data shared with other snapshots. This allows the user to have all of the information about what is going on with snapshots. In its current form used is a calculated number, derived from those two values. That means ZFS already "knows" these two values and all we're talking about is exposing them to the user. No need to refactor snapshots or ZFS.

 

[Ericloewe]

That means ZFS already "knows" these two values and all we're talking about is exposing them to the user.

Not really. It would probably require iterating through the whole chain to calculate those values.

 

[FreeNASBob]

Not really. It would probably require iterating through the whole chain to calculate those values.

It has to know those values or it would have no way to know to display 0 bytes for a snapshot that shares its data with another.

 

[Bidule0hm]

It probably knows that there's more than one snapshot, but not the size.

 

[Ericloewe]

It has to know those values or it would have no way to know to display 0 bytes for a snapshot that shares its data with another.

It's referenced to the previous snapshot. The snapshot just goes

• Pointer to previous snapshot
• Changes since previous snapshot
  • File A is now composed of blocks XYZ located at addresses xyz

So, the snapshot's size is the block size multiplied by the number of blocks changed. No changes? That's a zero.

There's no "sharing" of data, snapshots are simply built on top of the previous one.

 

[FreeNASBob]

So, the snapshot's size is the block size multiplied by the number of blocks changed. No changes? That's a zero.

That's the behavior I expect, but it's not the behavior displayed. In your example, snapshot1 has 3 blocks worth of changes (384K). Snapshot A should report 384K used. I think we're all on the same page up to here. Snapshot2 in your example has no changes, so it reports 0K used. I think we're all still in agreement that this behavior is good and well. What I think is completely insane is what happens to snapshot1 after snapshot2 is created: now that snapshot2 exists and points to all of the blocks in snapshot1, snapshot1 reports that it is also 0K even though snapshot1 has not been modified. That's insanity to me. How can a filesystem object's size change when it hasn't been modified?

I can't even imagine how crazy system administration would be if this concept were applied consistently across an entire file system. It would be like the Mad Hatter's version of dedupe. Every time a new data block was written that was identical to 1 or more other blocks on the file system, all the files sharing that block would report that their file size had decreased by 128K.

I know other people don't share my view about this, and I don't mean to imply that I think they're crazy. I'm just trying to explain why it seems crazy to me. To some extent I think the problem is how ZFS is revolutionary. Old farts like me have been doing "a file is a discrete object consisting of a pointer to x sectors" for almost 50 years. Along with that concept comes a lot of other logical constructs, like file size. What does file size even mean when two or more files can have pointers to the same data? It's like dividing by zero.