Snapshots not deleted after lifetime

[denningsrogue]

First, I apologize in advance for the length of this post but I am confused about snapshots, replication and end of life (EOL) deletion and I need some help.

I have 2 volumes, each comprised of 6 4TB drives in a RAIDZ-2 configuration. The second volume is a replication target for the first.

I have 4 periodic snapshot tasks set on the "push" volume.
1. hourly snapshots with a 1 day lifetime
2. daily snapshots with a 1 week lifetime
3. weekly snapshots with a 1 month lifetime
4. 4 weekly snapshots with a 1 year lifetime

Each of the snapshots is recursive and their are 16 datasets on the push volume. The snapshots tasks and replication are working properly.

The problem I am having is that the hourly snapshots are not being deleted at the end of a day. (I expect the same problem exists for the other tasks but they haven't been running long enough for me to see). I understand from the handbook that the snapshots won't be deleted on the "pull" side when they reach EOL (that's a separate question for later) but I thought that snapshots would be deleted on the push side when they reached their EOL. Furthermore, only the most recent hourly snapshot provides the rollback option (why don't the others?).

I have consulted the handbook and google to no avail. I am hoping one of the experts here can point me in the right direction to solve my problem.

 

[zambanini]

use only one snap per volume. more is not possible. at least I think so.

 

[rogerh]

use only one snap per volume. more is not possible. at least I think so.

No, you can have a snapshot task for each dataset (provided you haven't got a recursive snapshot task on the volume) and you can have multiple snapshot tasks per dataset. At least, before 9.3.1 you could. I have found snapshot tasks which are the subject of a replication task fail to delete expired snapshots since updating to 9.3.1.

https://forums.freenas.org/index.ph...-by-snapshot-task-in-9-3-1.36849/#post-225013

 

[denningsrogue]

I have updated to 9.3.1 and that's when the issue appeared. I now have more than 5000 snapshots and growing. Any suggestions on how to delete them manually?

 

[rogerh]

I hesitate to tell you how I did it, because I should have done clever things with sed and awk and epoch and 'if' statements. But the following works. It just outputs all the snapshots bare of any other info and the "grep" statements pick out the ones to destroy a day's worth at a time. Just repeat for days after the 25th until you are within somewhat over 24 hours from now, or change for different life snapshots. "Xargs -L 1" just hands out the snapshots to zfs destroy one at a time. "vola" is the pool name, mainly to exclude the boot pool as I couldn't find where to put it in the "zfs list" command.

Important note: This is not sanitised for general use. For instance, if any dataset has "1d" or another duration string in its name bad things will happen. At the very least, run it with the "| xargs -L 1 zfs destroy" missed off the end and look carefully at the list produced of snapshots that will be destroyed for any unexpected content!

zfs list -H -t snapshot -o name | grep vola | grep 1d | grep 20150825 | xargs -L 1 zfs destroy


However, there are some snapshots that won't delete. I get an unhelpful message that "the dataset is busy". I am not sure whether this is due to the current replication task, or due to a period I downgraded to a previous boot snapshot. Only a day and half's worth in my case, and more recent ones (27th and after) will delete.

 

[denningsrogue]

rogerh. Thanks. That worked perfectly.

 

[denningsrogue]

I take it this is a bug not a feature.;)

 

[rogerh]

I take it this is a bug not a feature.;)

No longer deleting stale snapshots on the receiving side of a replication task is an intended feature. The problem of not deleting snapshots on the sending side only seems to happen if one has a replication task, and it may be connected; but I don't think it is intended!

 

[rogerh]

In 201509022158 the 'problem' of deletion of snapshots on the receiving side has been solved. But so far the deletion of stale snapshots on the sending side does not seem to have been helped. @denningsrogue, can you check what effect this update has on your system?

 

[denningsrogue]

rogerh -- sorry for the delay. I have been on the road. Forgive the newbie question but how do I get the 201509022158 update?

 

[denningsrogue]

Never mind. I've got the update. It hasn't solved the problem for me on the push or pull side. I still have to delete expired snapshots manually. Instead it appears to have introduced a new issue, which is that the listing of snapshots now only presents a truncated list. For example, when I set the page to display 10 snapshots, only 6 appear. The same is true if the page is set to display 25, 50 or 100 snapshots per page. Only a truncated list appears. This is only true if I am using Safari. If I use Chrome, the list of snapshots appears as it should.

 

[rogerh]

If you look at bug https://bugs.freenas.org/issues/11227 then it seems everything except reporting info in the replication task tab (and perhaps displaying Versions in Windows, if you use this) will be sorted in the next update. You can download the relevant python files from the bug page if you want the fixes sooner. BTW, the snapshots are displayed ok here in Safari 6.2.8 (OSX 10.8.5).

 

[denningsrogue]

Problem resolved by the python files. Thank you. I also solved the display problem by disabling the Ghostery extension for Safari. With that extension disabled the correct number is displayed on the snapshot page.