ZFS Snapshot size calculation issue

[flotpg]

Hey Forum,

I admit it - I don't understand ZFS Snapshots and already found this: https://www.truenas.com/community/threads/snapshots-and-used-space.63810/
If I understand the mentioned post above correctly, the UI and also the output of "zfs list -ro space -t all tank/Backups" doesn't tell me what the snapshots are consuming in terms of disk space.

zfs list -ro space -t all tank/Backups = 6,6 TB but USEDSNAP = 16,4 TB - How ?

Code:

NAME                                      AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD

tank/Backups                              608G    38.5T     16.4T            22.1T             0B                          0B

tank/Backups@auto-2021-12-19_00-00      -         3.46T         -       -              -          -

tank/Backups@auto-2021-12-26_00-00      -         711G         -       -              -          -

tank/Backups@auto-2022-01-02_00-00      -         662G         -       -              -          -

tank/Backups@auto-2022-01-09_00-00      -         864G         -       -              -          -

tank/Backups@auto-2022-01-16_00-00      -         909G         -       -              -          -

But the output of this small script or gives me the same amount of roughly 6,6 TB

Code:

#!/bin/bash
# greps a list of snapshots, edit grep parameter
# or tank/ for all
dataset="tank/Backups"

zfs list -ro space -t all $dataset

for snapshot in `zfs list -H -t snapshot | cut -f 1 | grep $dataset`
do
zfs destroy -v -n $snapshot
done

Code:

NAME                                           AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD

tank/Backups                                       608G    38.5T 16.4T    22.1T      0B       0B

tank/Backups@auto-2021-12-19_00-00      -         3.46T         -       -              -          -

tank/Backups@auto-2021-12-26_00-00      -         711G         -       -              -          -

tank/Backups@auto-2022-01-02_00-00      -         662G         -       -              -          -

tank/Backups@auto-2022-01-09_00-00      -         864G         -       -              -          -

tank/Backups@auto-2022-01-16_00-00      -         909G         -       -              -          -



would destroy tank/Backups@auto-2021-12-19_00-00

would reclaim 3.46T

would destroy tank/Backups@auto-2021-12-26_00-00

would reclaim 711G

would destroy tank/Backups@auto-2022-01-02_00-00

would reclaim 662G

would destroy tank/Backups@auto-2022-01-09_00-00

would reclaim 864G

would destroy tank/Backups@auto-2022-01-16_00-00

would reclaim 909G

If I really delete it without the dry-run (-n) it reclaims much more space: 16,36 TB:


Can anyone please give me a hint ;)
Many thanks and best regards, Flo.

 

[sretalla]

Maybe this is relevant?

ZFS used space hidden in more than one snapshot

Someone was asking about this on IRC This is mostly for myself for later: hidden_in_snaps(){ bc -l <<<"( $(zfs list -H -p -o usedsnap ${1}) - $(zfs list -H -p -t snapshot -r -d 1 -o used ${1} | awk 'BEGIN{total=0} {total += $1} END{print total}') ) / 1024

blog.shalman.org

 

[flotpg]

Hey @sretalla, thanks for the link.
I will have a loot at it and let you know when I run into this next time (needed to free up space).
Many thanks and best regards, Flo.

 

[winnielinnie]

Snapshots overlap with each other, and the existence of other snapshots with shared records of "deleted" data that would otherwise be permanently lost without the existence of at least one of such snapshots, can make for some very confusing and hard-to-calculate "usage".



For example, imagine you have the live dataset, and you fill it up with a bunch of data. Some of this data consists of very large video files (maybe 100GB in total of such videos).

You then create @snap001.

You do some more stuff, here and there. Few minor deletions here, few new files there.

You then create @snap002.

Once again you later do more stuff. Some more minor changes.

You then create @snap003.

Finally, you delete the 100GB of large video files on your live dataset.

The live dataset, on its own, is now 100GB smaller in usage. Yet all three snapshots still refer to the 100GB worth of videos.

What would happen if you destroy @snap001? You might only end up freeing a few MB or so. Why won't it free 100GB? Because the remaining two snapshots still point to the records that contain the 100GB of video files.

You would have to destroy all three snapshots at the same time (or eventually all of them) to ever reclaim the 100GB. Until then, the mysterious 100GB of excess usage can be hard to pinpoint. (If you only act on one snapshot at a time, you will only ever see references to a "few MB" for each snapshot, and might never realize that any of the three simply still existing takes up 100GB.[1] The only reason we know about it in the above example is because it's a simplified example and we assume we're keeping track of the very specific snapshots that point to the same 100GB of video files.



[1] Another way to view this is how "the math doesn't add up". If you see how much each individual snapshot uses (or can potentially "free up") you'll end up with paradoxical math like this:

Destroying @snap001 will free up 64MB
Destroying @snap002 will free up 17MB
Destroying @snap003 will free up 33MB

Destroying @snap001 through @snap003 at the same time will free up 100GB

"What the heck?! 64MB + 17MB + 33MB = 100GB? That makes no sense!"

Your above script runs at a "snapshot-by-snapshot" individual basis. It doesn't run on groups of snapshots.

 

[flotpg]

@winnielinnie
thanks a lot for this great write up!
In my case I store Veeam backup files and your explanation makes totally sense here as the backup files contain backups of many VMs.
Best regards, Flo.

 

[Ericloewe]

Just to add: Determining the amount of space freed by a single snapshot is easy (size of the snapshot), as is the total space used by snapshots (the USEDSNAP property of the dataset).

Anything else is hard to define down to a single, useful number.