Help verifying periodic snapshots

[FreeNASBob]

I have a dataset on my FreeNAS box that is a mirror of my PC. I run rsync on the PC periodically to update files on FreeNAS. I have a periodic snapshot task set to run every 12 hours on that dataset. My thought was that the periodic snapshot task would take care of versioning the changes to the dataset. As I posted in another thread, the FreeNAS GUI of the periodic snapshots is not showing anything for the "used" column on snapshots that should show over 100 MB worth of changes.

Although I suspect that the issue is with the GUI and not the actual snapshots, I don't have a way to verify that right now. Can anybody offer any assistance on how I can check if the snapshots are malfunctioning? I don't want to find out one day that I have uploaded and replaced a file with a corrupted version only to find out that the snapshots I thought I was making aren't working.

Thanks.

 

[FreeNASBob]

More detail:

Using FreeNAS-9.3-STABLE-201512121950

Periodic snapshot task is set to run on pool/foo from 0:00 to 23:59 every 12 hours Sun - Sat. The snapshot task runs as expected and generates an entry in snapshots GUI.

If I run rsync on my PC and it updates files on pool/foo, I would then expect that the snapshot from the last time I ran rsync would grow in size to match the changes in the newly uploaded files. This isn't happening, at least not in the GUI. If I go back to the snapshot that was generated after the last sync, it still shows "0" in the used column. I'm pretty sure this is a GUI issue because if I browse pool/foo from Windows the "Previous versions" correctly shows a version from the last time I ran rsync.

I suppose the worst case scenario would be that I have to manually discover every corrupted file and manually use Windows to restore each one, but that kind of defeats the purpose of having ZFS snapshots.

Even the GUI tips you off that it's lying. Check out the "Refer" column where you see the size of the dataset change from 381.7 GB to 381.9 GB. That's 200 MB of changes (it's files that are growing in size), but the snapshot size is still 0. I don't understand. It's almost like the snapshot task is file-based instead of sector-based and it doesn't recognize that a file with the same name has changed.

 

[depasseg]

I'd open a bug (and post the # or link here and in the bug reporting section).

 

[FreeNASBob]

I'd open a bug (and post the # or link here and in the bug reporting section).

I want to, but I'm not confident I know where the problem is yet. I would like to have a reproducible test case first. Using zfs list -t snapshot I was able to see that the problem exists there as well, so it's not a GUI issue. I would think that the snapshots didn't contain the data at all if it weren't for the fact that they exist in Windows' "Previous versions" and the previous versions actually are different versions. It's just something about those files that are growing in size that aren't registering in the snapshot's size that's driving me crazy.

Take a look at the file properties for two versions of the same file from Windows Explorer.

You see that the version last modified 12/19/2015 is 852,392 bytes smaller than the version modified 12/25/2015. If I understand correctly, that means the next snapshot following 9:37 AM on 12/19/2015 should at least report 832 KB taken. In fact, it should be a lot more but I'm just limiting the discussion to this one file. The fact that I was able to use Windows Explorer to restore the previous version is proof enough that the snapshot data exists, but when we look at the snapshots taken on 12/19/2015 they all list 0 bytes taken.

As in the previous case, you can see the size of the data referenced in the snapshot changed between the snapshot from 12/19 at midnight and the one at noon later that day as expected, but zfs list still reports 0 bytes taken by the snapshot.

If this bug were as simple as some bug in zfs list I couldn't be the first person to notice it. There has to be some peculiarity to my setup that is causing this, and I want to find it before making a report. The only thing I can think of is GELI encryption.

Any ideas?

 

[FreeNASBob]

OK. I think I'm about to feel like an idiot, but here goes: It occurs to me that most of the file changes are appends. With two versions of the same file, the first x bytes of the latest version would be the same as the previous version. Theoretically that would mean each snapshot of this file would in fact be zero bytes because you can reproduce the previous versions by truncating the current version - no data needed. I knew that already.

What I had assumed was that ZFS snapshots were sector-based. The probability of the end of a file landing exactly on a sector boundary is 1/n, where n is the sector size (4096 in my case), which is usually a very small probability (<0.0002 in my case). That would mean that even if data were only appended to a file, you would expect that the last sector of the file had changed almost every time data were appended. That would cause the snapshot to occupy at least n bytes. What's going to make me feel like an idiot is if ZFS is smarter than being strictly sector-based (and blind to the bytes in that sector), and "knows" that the only change to the file is purely appended data ... even though that appended data changes a sector that it had seen before.

Does that make any sense? If that's the case it's really awesome on the one hand because I can basically have infinite versions that occupy zero space (other than overhead), and when I do see a positive number for the "Used" column I'll know some kind of corruption has taken place in the file on my PC. On the other hand it's kind of inconvenient because I have no easy way to identify which snapshots actually contain a change without manually looking through them to find the date and time when the size of the referred data changes.

 

[jgreco]

No, ZFS is a CoW filesystem, so an append is going to allocate a new block and write the updated data in that new block.

 

[FreeNASBob]

Bug filed. Periodic snapshots report "0 bytes used" when >0 bytes used.

 

[titan_rw]

Maybe I'm not understanding, but you do realize that the 'used' field for snapshots only list size that is 'used' for that snapshot, and only that snapshot?

Take this example. Dataset contains 1GB file. 2 snapshots are created back to back. User deletes 1GB file. 'Used' by snapshot will show 0 for each snapshot. That's because the 1GB of blocks that the file occupied is 'tied' to both snapshots. Delete either one of the snapshots, and the remaining one will show 'used' of 1GB.

I recently had this with a dataset that I had deleted a bunch (~500 gigs) of data. There were two snapshots. Each snapshot showed around 20 gigs 'used'. I deleted one of the snapshots, and now the remaining one shows that it's using ~450 gigs or so. This means that most (but not all) of the stuff I deleted was common to both snapshots.

Also, appending to a file doesn't 'use' more snapshot space. Why should it? No data is being deleted or changed. The referenced number will go up of course.

For my media dataset, that is mostly static, I can take a snapshot and a month later, it is still 'using' very little. Maybe a few gigs. Although I may have added 100's of gigs of data to the dataset since it was created. This will show up in the 'referenced' field.

If I remember right, you can tell how much space ALL the snapshots in a particular dataset are using with this:

Code:

zfs get usedbysnapshots tank/dataset

 

[jgreco]

The act of appending to a file implies a change, at least to the last block of file data.

 

[titan_rw]

The act of appending to a file implies a change, at least to the last block of file data.

This would be quite small though, no? Mostly a metadata change? I think the OP was thinking that appending 10 gigs to a 10 gig file would show 10 or 20 gigs of 'used' in a snapshot.

This demonstrates the 'used' for snapshots when they share common data:

Use GUI to create dataset TEST:

Code:

root@nas nas1pool # zfs list nas2pool/test
NAME  USED  AVAIL  REFER  MOUNTPOINT
nas2pool/test  192K  9.23T  192K  /mnt/nas2pool/test

Use GUI to create 1st snapshot:

Code:

root@nas nas1pool # zfs list -t all -r nas2pool/test
NAME  USED  AVAIL  REFER  MOUNTPOINT
nas2pool/test  192K  9.23T  192K  /mnt/nas2pool/test
nas2pool/test@1-emptydataset  0  -  192K  -

Put data on dataset: (compression disabled)

Code:

root@nas test # dd if=/dev/zero of=1gb bs=1m count=1k
1024+0 records in
1024+0 records out
1073741824 bytes transferred in 0.397269 secs (2702807910 bytes/sec)
root@nas test # ls -lah
total 1048609
drwxr-xr-x  2 root  wheel  3B Dec 30 13:19 .
drwxr-xr-x  13 root  wheel  13B Dec 30 13:17 ..
-rw-r--r--  1 root  wheel  1.0G Dec 30 13:19 1gb

Use GUI to create 2nd snapshot:

Code:

root@nas test # zfs list -t all -r nas2pool/test
NAME  USED  AVAIL  REFER  MOUNTPOINT
nas2pool/test  1.00G  9.22T  1.00G  /mnt/nas2pool/test
nas2pool/test@1-emptydataset  128K  -  192K  -
nas2pool/test@2-onegigofdata  0  -  1.00G  -

Use GUI to create 3rd snapshot:

Code:

root@nas test # zfs list -t all -r nas2pool/test
NAME  USED  AVAIL  REFER  MOUNTPOINT
nas2pool/test  1.00G  9.22T  1.00G  /mnt/nas2pool/test
nas2pool/test@1-emptydataset  128K  -  192K  -
nas2pool/test@2-onegigofdata  0  -  1.00G  -
nas2pool/test@3-onegigofdata  0  -  1.00G  -

Delete data:

Code:

root@nas test # rm 1gb

'USED' shows 0 because both snapshots share the deleted data:

Code:

root@nas test # zfs list -t all -r nas2pool/test
NAME  USED  AVAIL  REFER  MOUNTPOINT
nas2pool/test  1.00G  9.22T  1.00G  /mnt/nas2pool/test
nas2pool/test@1-emptydataset  128K  -  192K  -
nas2pool/test@2-onegigofdata  0  -  1.00G  -
nas2pool/test@3-onegigofdata  0  -  1.00G  -

If we deleted all snapshots, we would reclaim 'usedbysnapshots':

Code:

root@nas test # zfs get usedbysnapshots nas2pool/test
NAME  PROPERTY  VALUE  SOURCE
nas2pool/test  usedbysnapshots  1.00G  -

Use GUI to delete 2nd snapshot. The remaining snapshot (3rd) shows 'used' now because it's the only one refrencing the deleted data.

Code:

root@nas test # zfs list -t all -r nas2pool/test
NAME  USED  AVAIL  REFER  MOUNTPOINT
nas2pool/test  1.00G  9.22T  192K  /mnt/nas2pool/test
nas2pool/test@1-emptydataset  128K  -  192K  -
nas2pool/test@3-onegigofdata  1.00G  -  1.00G  -

Use GUI to delete 3rd snapshot. Verify dataset disk usage has gone down:

Code:

root@nas test # zfs list -t all -r nas2pool/test
NAME  USED  AVAIL  REFER  MOUNTPOINT
nas2pool/test  320K  9.23T  192K  /mnt/nas2pool/test
nas2pool/test@1-emptydataset  128K  -  192K  -

Code:

root@nas test # zfs get usedbysnapshots nas2pool/test
NAME  PROPERTY  VALUE  SOURCE
nas2pool/test  usedbysnapshots  128K  -

 

[FreeNASBob]

Maybe I'm not understanding, but you do realize that the 'used' field for snapshots only list size that is 'used' for that snapshot, and only that snapshot?

I understand that.

Also, appending to a file doesn't 'use' more snapshot space. Why should it? No data is being deleted or changed. The referenced number will go up of course.

That's not true at all. To make a simple example, say I have a 1-byte file. That file still occupies the entire 4,096-byte sector. If I make a snapshot, append another byte to that file, and make another snapshot the used snapshot size for the first snapshot should be 4kB because it has to keep the original sector as it was with the 1 byte of data and the filesystem now has another sector with 2 bytes of data.

For my media dataset, that is mostly static, I can take a snapshot and a month later, it is still 'using' very little. Maybe a few gigs. Although I may have added 100's of gigs of data to the dataset since it was created. This will show up in the 'referenced' field.

All of my snapshots save for six show as 0 bytes used. That's simply impossible.

If I remember right, you can tell how much space ALL the snapshots in a particular dataset are using with this:

Code:

zfs get usedbysnapshots tank/dataset

Thanks for that. Here's the output of that command on my dataset:

Code:

[root@freenas ~]# zfs get usedbysnapshots pool/foo
NAME PROPERTY VALUE SOURCE
pool/foo usedbysnapshots 76.0G -

76 GB used by snapshots, but if you add up the used column in the GUI for all snapshots from that dataset it totals 1870.2 KB.

The only filters applied to the listing of snapshots are "used is not 0" and "Volume/Dataset begins with pool/foo".

Something very fishy is going on here.

 

[titan_rw]

Just to test the append thing myself:

Create 10 gig file:

Code:

root@nas test # dd if=/dev/zero of=10gb bs=1m count=10k
10240+0 records in
10240+0 records out
10737418240 bytes transferred in 14.659084 secs (732475396 bytes/sec)

Create snapshot:

Code:

root@nas test # zfs list -t all -r nas2pool/test
NAME  USED  AVAIL  REFER  MOUNTPOINT
nas2pool/test  10.0G  9.22T  10.0G  /mnt/nas2pool/test
nas2pool/test@1-10gig  0  -  10.0G  -

Append 5 gigs to existing 10 gig file:

Code:

root@nas test # dd if=/dev/zero bs=1m count=5k >> 10gb
5120+0 records in
5120+0 records out
5368709120 bytes transferred in 4.825214 secs (1112636446 bytes/sec)

'Used' is now 144k, I assume due to metadata updates required to append to the file? But the snapshot properly shows it's referencing 10 gigs:

Code:

root@nas test # zfs list -t all -r nas2pool/test
NAME  USED  AVAIL  REFER  MOUNTPOINT
nas2pool/test  14.1G  9.21T  14.1G  /mnt/nas2pool/test
nas2pool/test@1-10gig  144K  -  10.0G  -

Create another snapshot. Verify Refer is 15G:

Code:

root@nas test # zfs list -t all -r nas2pool/test
NAME  USED  AVAIL  REFER  MOUNTPOINT
nas2pool/test  15.0G  9.21T  15.0G  /mnt/nas2pool/test
nas2pool/test@1-10gig  144K  -  10.0G  -
nas2pool/test@2-15gig  0  -  15.0G  -

Deleting all snapshots would free 144k. You would lose the ability to revert back to the 10 gig file:

Code:

root@nas test # zfs get usedbysnapshots nas2pool/test
NAME  PROPERTY  VALUE  SOURCE
nas2pool/test  usedbysnapshots  144K  -

 

[FreeNASBob]

It seems as though you're suggesting, and maybe FreeBSD and FreeNAS intend, that the "used" nomenclature isn't intended to describe the number of bytes that differ between a snapshot and the filesystem from which it originated. Instead, it seems, that "used" is intended to mean "unique bytes found only in this snapshot." If so, that would be a bizarre metric as I can't think of how it would be useful in the real world.

 

[titan_rw]

That's not true at all. To make a simple example, say I have a 1-byte file. That file still occupies the entire 4,096-byte sector. If I make a snapshot, append another byte to that file, and make another snapshot the used snapshot size for the first snapshot should be 4kB because it has to keep the original sector as it was with the 1 byte of data and the filesystem now has another sector with 2 bytes of data.

Excluding metadata, why would the snapshot show the original 1 byte as 'used'? It hasn't changed. It's static. Snapshots only use space when data gets deleted, or changed. The first 'chunk' of your file, regardless of how big, doesn't change. See my example I posted.

All of my snapshots save for six show as 0 bytes used. That's simply impossible.

It's very possible. Again, the snapshots 'used' field only shows deleted / changed data that's exclusive to that snapshot. Because of this, you can't add non-zero sized snapshots together and expect to reach the 'usedbysnapshots' number. My example shows this clearly.

Thanks for that. Here's the output of that command on my dataset:

Code:

[root@freenas ~]# zfs get usedbysnapshots pool/foo
NAME PROPERTY VALUE SOURCE
pool/foo usedbysnapshots 76.0G -

76 GB used by snapshots, but if you add up the used column in the GUI for all snapshots from that dataset it totals 1870.2 KB.

The only filters applied to the listing of snapshots are "used is not 0" and "Volume/Dataset begins with pool/foo".

Something very fishy is going on here.

So you have 76 gigs of changed / deleted data that your snapshots are holding on to. If MOST of this data was referenced by MORE than ONE snapshot, then the 'used' numbers for the snapshots won't add up to the total usedbysnapshots.

Everything seems fine to me.

 

[Bidule0hm]

It seems as though you're suggesting, and maybe FreeBSD and FreeNAS intend, that the "used" nomenclature isn't intended to describe the number of bytes that differ between a snapshot and the filesystem from which it originated. Instead, it seems, that "used" is intended to mean "unique bytes found only in this snapshot." If so, that would be a bizarre metric as I can't think of how it would be useful in the real world.

I understand the used value as the space used by the snapshots to retain the state of the data it refers to. If you just add data but don't delete anything then the snapshot just need to keep track of some metadata. If you delete (or change) some data the snapshot needs to keep the deleted data the used value is the small metadata tracking + the space used by the data.

 

[titan_rw]

It seems as though you're suggesting, and maybe FreeBSD and FreeNAS intend, that the "used" nomenclature isn't intended to describe the number of bytes that differ between a snapshot and the filesystem from which it originated. Instead, it seems, that "used" is intended to mean "unique bytes found only in this snapshot." If so, that would be a bizarre metric as I can't think of how it would be useful in the real world.

Yes, as I mentioned in my first post when I said "the 'used' field for snapshots only list size that is 'used' for that snapshot, and only that snapshot"

This is a ZFS thing. As per how oracle designed it:

http://docs.oracle.com/cd/E19253-01/819-5461/gbcxc/index.html

Snippet from above URL: "As the file system changes, disk space that was previously shared becomes unique to the snapshot, and thus is counted in the snapshot's used property. Additionally, deleting snapshots can increase the amount of disk space unique to (and thus used by) other snapshots."

 

[FreeNASBob]

Excluding metadata, why would the snapshot show the original 1 byte as 'used'?

Because ZFS snapshots are sector-based. It doesn't know anything about the file. All ZFS knows is that the sector has changed and so a copy has to be made of that sector to retain the ability to restore it.

It's very possible. Again, the snapshots 'used' field only shows deleted / changed data that's exclusive to that snapshot. Because of this, you can't add non-zero sized snapshots together and expect to reach the 'usedbysnapshots' number. My example shows this clearly.

That would be a bizarre metric, and "used" is not a fitting label for it. "Unique bytes" would be much better. "Used" implies, well ... used.

So you have 76 gigs of changed / deleted data that your snapshots are holding on to. If MOST of this data was referenced by MORE than ONE snapshot, then the 'used' numbers for the snapshots won't add up to the total usedbysnapshots.

Everything seems fine to me.

That seems crazy to me. It's not giving me information about what's contained in the snapshot. It's not telling me WHERE those 76 GB are. How does one go and find that information?

 

[titan_rw]

I understand the used value as the space used by the snapshots to retain the state of the data it refers to. If you just add data but don't delete anything then the snapshot just need to keep track of some metadata. If you delete (or change) some data the snapshot needs to keep the deleted data the used value is the small metadata tracking + the space used by the data.

Yes, but keep this in mind: If more than one snapshot 'reference' the deleted data, this deleted data won't show in ANY of the snapshots 'used' numbers.

 

[FreeNASBob]

Yes, but keep this in mind: If more than one snapshot 'reference' the deleted data, this deleted data won't show in ANY of the snapshots 'used' numbers.

That seems bizarre to me. Regardless, none of my files are deleted ... only modified.

 

[FreeNASBob]

I understand the used value as the space used by the snapshots to retain the state of the data it refers to. If you just add data but don't delete anything then the snapshot just need to keep track of some metadata.

I was under the impression that ZFS snapshots were sector-based, but you're suggesting they're file-based. That changes things.