Recursive snapshots not removed after lifetime expires

[mrktt77]

8.3.0-RELEASE-p1-x64

Pool
Pool\Dataset1
Pool\Dataset2

Periodic snapshot task on Pool with "Recursive" flagged.
Snapshot are created as expected on Pool and on Datasets.
The Pool's snapshots are removed correctly when lifetime is crossed. Pool\Dataset1-2 snapshots stays thereforever.

I've read other posts and bug reports but no solution. I'm thinking about an additional cron job but if someone knows how to solve or has a quick fix job already written I would be grateful
thanks

 

[ixdwhite]

Please list the configured replication jobs and their settings as well as the output of 'zfs list -t snapshot', highlighting the snapshots that should have been deleted and weren't. In addition, verify that you have no cloned snapshots mounted. Cloned snapshots will disallow the backing snapshot to be deleted and the deleter job will abort when it experiences a delete error.

I suspect either ZFS is disallowing the delete due to a dependency or its a bug in the snapshot manager (and some bugs were found related to recursive snapshot replication recently so this is fairly likely).

Note that configuring a top level Recursive job and jobs for individual subordinate datasets is redundant; the recursive snapshot is already snapping the other datasets. A top-level Recursive and individual subordinate snapshots interact poorly with replication and I don't recommend mixing them if you can avoid it. If you need the ability to rollback individual datasets then don't create the top level dataset snapshot job as Recursive.

 

[mrktt77]

There was a replication job (single one, with "recursive, remove stale snapshot" flagged) but I've removed it to see if it is related.
On the remote machine the snapshot are exactly the same, as it should be.

Can't copy
zfs list -t snapshot
because there's like 1k+ snapshots (pool + some datasets * almost a year of daily and weekly snapshots, it's a remote backup/time machine service for multiple customers. I need separate datasets for individual rollback, quotas etc. and except for this problem it works like a charm.)
but the list is as it should be:

pool@auto-yyyymmdd.hhmm-1m for the pool
pool/datasetx@auto-yyyymmdd.hhmm-1m for the other datasets

there are no cloned snapshots, no snapshots mounted, nothing at all.
only thing not done via UI is the renaming (not cloning) of the first snapshot of every dataset from -6m to -2y.
the only problem is that I have the correct number of pool@auto according to the retention policy I've in place but pool/datasetx@auto are never removed and stays there forever. They are deletable however, I'm cleaning them from time to time manually from shell to avoid filling up the pool.

I can see
autosnap.py: [tools.autosnap:56] Popen()ing: /sbin/zfs snapshot -r ***x/yyyy@auto-20130507.0900-1m
in the log. but the -r doesn't delete the snapshots on child datasets.

There's only 2 task job on the main pool (with recursive flagged) and nothing else.
1 daily on workday with 1 month of retention
1 weekly on sunday with 6 month of retention

Now I'm waiting to see if with the replication job gone something changes. The next snapshot due to be removed should be gone in 4 hours. So let see..

 

[mrktt77]

ok, confirmed. as soon as I delete the replication task the removal of snapshots goes as it should.
so, I can't have a sigle recursive snapshots job and a single recursive replication job? By design or for some kind of bug? Or am I doing it wrong?
I mean, it's possibile that the thing is gonna be fixed somehow in the next releases or I should schedule a cron job to clean the snapshots that aren't supposed to be there?
thanks

 

[cyberjock]

I think the best way to explain it would be user error based on how snapshots work. It's not a bug or really even by design. Snapshots are designed to work a certain way and your misunderstanding about how snapshots(and recursive snapshots) work lead to your issue.

 

[mrktt77]

I think the best way to explain it would be user error based on how snapshots work. It's not a bug or really even by design. Snapshots are designed to work a certain way and your misunderstanding about how snapshots(and recursive snapshots) work lead to your issue.

uhm, ok...
I suppose it's the same misunderstanding of this newbie guy too.
Old post on Ben Rockwood blog from the time of infancy of ZFS

Finally, what if we want to work on more than just a single snapshot. What if we want to send all the "home" datasets? For some time now (although just now arriving in Solaris 10) we've had recursive flags for both zfs snapshot and zfs send. Lets give it a try:

root@quadra ~$ zfs snapshot -r data/home@nov6
root@quadra ~$ zfs send -R data/home@nov6 | zfs recv -d backups

since I honestly can't see where are the differences from his test setup and mine, except he is doing it locally and not over ssh.
-r on snapshot I suppose is the flag "recursive" in UI, and -R on send I suppose is the "recursive" flag in UI on replication.
And if I do replications, snapshots and pruning of old snapshots manually from shell it's ok for me too.
But probably it's again me beeing newbie :)
thanks anyway.

 

[ixdwhite]

I believe this is related to a known issue, come to think of it. Mixed recursive and non-recursive snapshots cause the bookkeeping on whether snapshots have been transferred or not to get confused. The deleter might skip deleting snapshots that it thinks are supposed to be transferred and haven't yet. I'll have to check if there is a fix and if it was committed to the public FreeNAS tree yet.

 

[ixdwhite]

Please try replacing autorepl.py with the one out of gui/tools/autorepl.py from trunk. This uses 'zfs destroy -r' during deletion which will make the delete process more aggressive. This may also address issues with replicating Recursive and non-Recursive snapshots.

I think the best advice is to use Recursive snapshots unless it is the only one scheduled on the entire dataset hierarchy. If you need per-dataset snapshots then don't set up Recursive ones at the same time or they will conflict.

You may have to clean out previous Recursive snapshots to break any old dependencies.

 

[mrktt77]

Please try replacing autorepl.py with the one out of gui/tools/autorepl.py from trunk. This uses 'zfs destroy -r' during deletion which will make the delete process more aggressive. This may also address issues with replicating Recursive and non-Recursive snapshots.

I think the best advice is to use Recursive snapshots unless it is the only one scheduled on the entire dataset hierarchy. If you need per-dataset snapshots then don't set up Recursive ones at the same time or they will conflict.

You may have to clean out previous Recursive snapshots to break any old dependencies.

Thanks. However I've inspected autorepl.py on my box (I've even updated it to 8.3.1 p2 in the mean time) and it seems identical to the master on github. the destroy -r is already there.
I'd like to specify that I'm not mixing up recursive and non recursive snapshots. the only snapshot job is recursive.
But I've find out a funny thing in the mean time, maybe is related to the subject.
3-4 days ago, before removing the replication job, a zfs get -Hr freenas:state on the main pool or on any child datasets gave NEW for every snapshot from the beginning of time and keep in mind that all of them have been correctly replicated on the remote box.
Now, after a couple of days without replication job, zfs get -Hr freenas:state gives "- -" for the old snapshots already replicated and "NEW local" for the 2 new snapshots. No "latest" however.
Now I've readded the replication job, next snapshot is tomorrow morning, let's see..
thanks

 

[mrktt77]

For those interested.
Since I need a quick fix more than a clear idea of what was going on:

- I've reset manually the freenas:state property on the pool and on all the child datasets and snapshots on the first box.
- Deleted all the replicated data on the destination box.
- Re-added the replication task with "Initialize remote side for once"

First replica was completed before the next recursive snapshot.
Now all is going on as supposed. Same data on the two sides AND old snapshots correctly deleted on the source side.

The pool has the correct property values (LATEST / NEW before replication, - / LATEST after) on the last couple of snapshots and " - - " on the previous ones.
The child datasets however have "NEW local" on the snapshots added since reset, even if they're already been replicated with the recursive job.
If the snapshots marked as NEW are ignored by the purge code of autosnap.py I think I'll be on square one in one month, i.e., all child datasets marked as NEW local and not touched by autosnap.py.

My 0.002 newbie cent are:

if autosnap.py sets the freenas:state property to NEW recursevely

autosnap.py: [tools.autosnap:56] Popen()ing: /sbin/zfs snapshot -r -o freenas:state=NEW storez/rsync@auto-20130521.0900-1m )

shouldn't autorepl.py do the same when has done replicating recursively the pool+child datasets ?
so

autorepl.py: [common.pipesubr:69] Executing: /sbin/zfs inherit freenas:state storez/rsync@auto-20130520.0900-1m
shouldn't be
autorepl.py: [common.pipesubr:69] Executing: /sbin/zfs inherit -r freenas:state storez/rsync@auto-20130520.0900-1m

with the -r of recursion?
thanks