ZFS Rollup - A script for pruning snapshots, similar to Apple's TimeMachine

fracai

Guru
Joined
Aug 22, 2012
Messages
1,212
I've posted a Python script at https://github.com/fracai/zfs-rollup and https://bitbucket.org/fracai/zfs-rollup for "rolling up" old snapshots. The intended use is that you would configure automated snapshots at some interval of time with a long retention length and allow the script to prune these into a sparser set as they get older.

The original version is very similar to Apple's TimeMachine. All snapshots are kept for the past day (most sensible if you're making hourly snapshots), then daily snapshots are kept for the past week, and finally weekly snapshots are kept forever. If you have your snapshots set to one year of retention, those weekly snapshots will start falling off when they're a year old.

I'm now working on supporting arbitrary intervals (i.e. keep snapshots from every two hours for two days) and dropping the assumption that the system is set to hourly snapshots (i.e. snapshot from every 15 minutes). Preliminary support is there, but it still needs work and hasn't been fully tested, so the best version to use right now is from before I added that (563c442685 at github or a8ddfac at bitbucket). Arbitrary interval support has been implemented.

Suggestions, requests, and testing are welcome.

Please note that the intent of the script is to delete snapshots. It's really only the last line in the script that does the scary bit, and I generally test this pretty well, but bugs will slip through. Monitus es.

Just wanted to update this to note that the scripts have been updated for Python 3 and should be compatible with FreeNAS 11.3. Please let me know if there are any issues or feature requests.
 
Last edited:

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
I am in no way disrespecting the effort you put into this, but I must be missing something.

What is the benefit of this over just configuring FreeNAS to do multiple levels of periodics?

For example, we snap our Docs ever hour for 12 hours, every six hours for two weeks, and then every week for 10 years.

This has been working fine here since the early days of 8.0.foo, and is functionality built into the base system, which has the advantage of surviving an upgrade.
 

fracai

Guru
Joined
Aug 22, 2012
Messages
1,212
To be honest, I think I started writing this before even considering multiple periodics. Even after though, I think I'd still prefer this method.

My reasoning is basically that when you run multiple periodics you're going to end up with duplicate snapshots (when the weekly snapshot runs, you could have three snapshots at the same time). They might not add anything to used capacity, but it's additional clutter to browse through when looking for the snapshot that you need to restore from. There's also an open ticket on problems with replication for snapshots made at the same time. Again with reducing clutter, the script allows keeping just one periodic task at the highest frequency that you want. The script then cleans up the snapshots that aren't needed once they're old enough.

Plus, there's an option to prune empty snapshots as well. Not everyone is going to be interested in that, but it'll clear out some more clutter.

Even though the script isn't part of the base system, it's entirely reliant on the dates of your snapshots, so it will survive an upgrade as well. (Mine is in my user folder and scheduled as an hourly cron job.)
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Interesting. No idea. Maybe it'll bite us too. I had no idea that there were any bugs relating to that. We'd been doing multiple tiers of snapshots since 8.0.1. Yup, our oldest ZFS snap is Documents@auto-20111009.0000-10y, days after we set up 8.0.1.

Whatever bugs there may be in replication, they may be less severe than William makes it sound. I recently upgraded our fileserver here from 8.0.smthn to 8.3.0 on new hardware and a from-scratch config, 4K block size, etc., and did a zfs send style manual replication to populate the whole thing. That shoved over all the snapshots just fine. Then, having the old main server as a "backup server" to fall back on, I blew away the little fileserver we use for backups/replication, stuffed new disks in it, set up replication, and let it fresh up. The ZFS documents snap above is from THAT server so replication seems to be quite capable of dealing with snaps, even multiple times. The replication target does seem to have a few dozen more snapshots than I'd expect so there may be something amiss somewhere, but it isn't immediately obvious what that would be...
 

fracai

Guru
Joined
Aug 22, 2012
Messages
1,212
I've pushed new commits that add support for arbitrary intervals so you can set up things like: 2h:24,6h:60,1d:30,weekly:10,monthly
That's keep every two hours for 2 days (24 snapshots), every 6 hours for 15 days, 1 day for 30 days, modify the weekly task to save them for 10 weeks, and the default monthly task (made as early as possible in the month and saved for 12 snapshots).

As always, testing and ideas are appreciated.
 

bollar

Patron
Joined
Oct 28, 2012
Messages
411
I wound up needing your script today, so thanks again.

The problem with multiple periodic snapshot tasks appears to surface when a snapshot expires before it's replicated to the pull server. It was easy enough for me to duplicate the problem by starting a large replication with a weekly snapshot and then creating an every 15 minute periodic snapshot task with a two hour expiration. At some point after the first snapshot is transferred, there will be one of several types of file not found failures:
Code:
Hello,
   The system was unable to replicate snapshot tank/data to replication
======================
cannot receive incremental stream: most recent snapshot of tank/data does not
match incremental source
warning: cannot send 'tank/data/iscsi/users@auto-20121214.1003-2h': Broken pipe
cannot send 'tank/data': I/O error


Code:
Hello,
   The system was unable to replicate snapshot tank/data to replication
======================
WARNING: could not send tank/data@auto-20121211.2118-24h: does not exist
WARNING: could not send tank/data/users@auto-20121211.2118-24h: does not exist
WARNING: could not send tank/data/scratch@auto-20121211.2118-24h: does not exist
WARNING: could not send tank/data/public@auto-20121211.2118-24h: does not exist
WARNING: could not send tank/data/media@auto-20121211.2118-24h: does not exist
WARNING: could not send tank/data/iscsi@auto-20121211.2118-24h: does not exist
WARNING: could not send tank/data/iscsi/users@auto-20121211.2118-24h: does not exist
WARNING: could not send tank/data/iscsi/itunes@auto-20121211.2118-24h: does not exist
WARNING: could not send tank/data/iscsi/aperture@auto-20121211.2118-24h: does not exist
WARNING: could not send tank/data/iscsi/finalcutpro@auto-20121211.2118-24h: does not exist
Succeeded.


Anyway, I haven't been able to break a replication with only one periodic snapshot task.

If you were to get really bored, perhaps you could suggest modifications to the FreeNAS snapshot and replication scripts:

autosnap.py
autorepl.py
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
The problem with multiple periodic snapshot tasks appears to surface when a snapshot expires before it's replicated to the pull server.

Ahhh. And that's likely why we've never seen an issue. We just aren't pushing very much stuff around most of the time, it's long-term storage.
 

fracai

Guru
Joined
Aug 22, 2012
Messages
1,212
I've thought about patching autosnap to allow the sort of pruning that my script provides, but it would also require GUI changes in order to define and select the desired intervals. I don't think I'm committed to that effort unless I hear that there's more of a desire for this sort of thing; either from users or the FreeNAS developers.

For now, I'm happy with what I can achieve with an hourly cronjob. But, who knows, I have been known to get bored...
 

bollar

Patron
Joined
Oct 28, 2012
Messages
411
But, who knows, I have been known to get bored...

I hear you. Things will be slow here, so my project over the holiday is to get back up to speed with openindiana.
 

kavermeer

Explorer
Joined
Oct 10, 2012
Messages
59
I've been spending some time on setting up replication before experimenting more with the pruning scripts. Apart from the fact that I don't understand why empty snapshots on the push server are no longer empty at the pull server, it seems to work fine. So I'm back to the snapshots.

In my setup, I have a number of shares and because I want to enable the 'previous version' functionality in Windows, they all need their own snapshot tasks. So I have a bunch of auto snapshots, and one replication task. As our system is used to store data from data acquisition systems, there is sometimes quite a large amount of new/changed data to replicate.

Now I'm wondering how to schedule things. I need to make snapshots during office ours. What would be the best time to schedule the pruning job, and the replication? I was thinking about running the pruning job during the day, with the last run after the last snapshots are made. Then start replication after that.

Or would it be better to just run the pruning hourly and let the replication task run for the full 24 hours?
 

kavermeer

Explorer
Joined
Oct 10, 2012
Messages
59
I stumbled upon http://mail.opensolaris.org/pipermail/zfs-auto-snapshot/2009-February/000092.html, where they state that

Define 'empty'

Having a "used" value of 0 doesn't work, because of the way snapshot
disk space accounting works, where data referenced by two snapshots
isn't accounted for in the used value of either snapshot)

I found another thread that addresses this same topic:

Data which is in more than one snapshot will not get counted in the "zfs
list" output at all.

If this is just noise, I apologize. But I seem to sometimes have the luck of being hit by rare bugs...
 

fracai

Guru
Joined
Aug 22, 2012
Messages
1,212
Thanks, that'd be a reason not to use the --zero option. Maybe I should remove it; it's purely cosmetic anyway and I'd rather not have people deleting snapshots because I'm not properly handling a quirk of ZFS.

As for when to run the tasks, I personally run the rollup script every hour. I'd probably run the replication every hour as well, unless it took too long or consumed too many resources. In that case, I'd run it after work hours.
 

fracai

Guru
Joined
Aug 22, 2012
Messages
1,212
I removed the '--empty' option, it wasn't actually doing anything anyway. That was a casualty of adding arbitrary intervals and then testing on snapshots that were already being pruned of "empty" snapshots.
 

kavermeer

Explorer
Joined
Oct 10, 2012
Messages
59
I did really like the cleanup of the empty snapshots. I create a lot of snapshots, and have to do so for every share, but most of the time there are no changes. However, I do agree that it might not be the best idea to have it as part of a time-based clean-up script.

From the two links, it wasn't clear to me whether this is regarded as a bug (and might be fixed already) or that it is by design. In any case, both links seem to agree that removing one empty snapshot at a time is save. So we might need a new script that does just that. I might even give it a try, as I think it shouldn't be too hard.
 

fracai

Guru
Joined
Aug 22, 2012
Messages
1,212
Agreed. The downside to an empty script is that it has to check the sizes after every destroy. But yeah, a patch to my rollup script or a brand new one would be great, if you have the time.
 

kavermeer

Explorer
Joined
Oct 10, 2012
Messages
59
I'm trying to put something together. I'm sorting snapshots by date now, and remove the oldest one first. I might as well start with the newest one - I just want to be consistent. However, I'm not sure if I should do something similar for nested snapshots. I mean, if both 'bla@auto-date1' and 'bla@auto-date2' have their used property set to 0, then removing one of them may change the value of the used property for the other. But does the same also apply to 'bla@auto-date' and 'bla/bla@auto-date'? If so, I want to make sure I'm always starting from the top level (or from the bottom level).
 

fracai

Guru
Joined
Aug 22, 2012
Messages
1,212
Personally, I'd want to never remove the latest snapshot, even if its empty, and the work backwards removing empty snapshots. But, that's me. And I agree, start with the deepest datasets first.

I look forward to seeing what you come up with.
 

fracai

Guru
Joined
Aug 22, 2012
Messages
1,212
Actually, I misspoke. Datasets are independent and won't share data from parent to child, so the order shouldn't matter.
 

fracai

Guru
Joined
Aug 22, 2012
Messages
1,212
Time saving idea: build separate lists for each dataset and delete an empty snapshot from each instead of iterating over datasets one at a time.
 
Top