Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.

ZFS Rollup - A script for pruning snapshots, similar to Apple's TimeMachine

Western Digital Drives - The Preferred Drives of FreeNAS and TrueNAS CORE

dpearcefl

Member
Joined
Aug 4, 2015
Messages
137
Sorry for being dense but...

So would you set up your snaps in the FreeNAS GUI to snap every hour and then before the next snap run this script hourly via cron?
 

fracai

Neophyte Sage
Joined
Aug 22, 2012
Messages
1,212
I actually make snapshots every five minutes and run the script once an hour.

Hourly snapshots and the script on an hourly schedule should be fine as well.
 

dpearcefl

Member
Joined
Aug 4, 2015
Messages
137
So if I do this:

Code:
# ./rollup.py -t -v -r -i hourly:5,daily:4 Mirror/GMail
Mirror/GMail
                @auto-20160526.1052-2y h d - 0
        pruning @auto-20160526.1057-2y - - - 0
                @auto-20160526.1102-2y h - R 0


Can you explain the four columns at the end?
 

dpearcefl

Member
Joined
Aug 4, 2015
Messages
137
I feel a little silly:
Column 1 - Matches hourly interval?
Column 2 - Matches daily interval?
Column 3 - most Recent
Column 4 - Size of the snap shot in bytes
 

fracai

Neophyte Sage
Joined
Aug 22, 2012
Messages
1,212
Yes, I'm pretty sure you've got it. I should probably add a suffix to the size.

Any other questions or usability issues that I could improve?
 

dpearcefl

Member
Joined
Aug 4, 2015
Messages
137
Maybe just a little documentation, like an "INSTALL" file? Just good practices on how to schedule things.

This is a great script. I've been testing it today and as soon as I understood what it was doing, it worked great. Thanks.
 

fracai

Neophyte Sage
Joined
Aug 22, 2012
Messages
1,212
Yeah, documentation is indeed lacking. I'm glad you like it and I'll work on those docs as soon as I can.
 

beezel

Member
Joined
Nov 7, 2016
Messages
33
I hate to necro such an old thread, but I ran into this script and it perfectly fits the bill for what I'm needing to do. I'm just running into some issues with the intervals. Any chance there is any docs or help available out there? I can't quite make out what to do from the syntax provided in rollup.py.

I'm looking to keep 1 daily for 2 months, but would like to run something that removes everything except the last hour for testing. Any suggestions?
 

fracai

Neophyte Sage
Joined
Aug 22, 2012
Messages
1,212
The closest you can get for 1 daily for 2 months would actually be 1 daily for 60 days: "1d:60".

Only keeping the last hour would depend on how often the snapshots are being taken. 5M:12 would keep snapshots every 5 minutes for an hour (5 minutes * 12 = 60 minutes = 1 hour). Note the capital M; lowercase would specify months.

The format is "interval:count,interval:count,...". Also note that this script isn't actually making the snapshots, just removing them. It's up to another process (FreeNAS includes a snapshot task) to actually make them. I'm just stating this to be explicit in the interaction with the intervals to keep.

You can use '-t' to show what would be kept and deleted.


It wouldn't be a bad idea to create a script that deletes all snapshots before a given date or older than some time interval. I'll look in to that.
 

srit

Junior Member
Joined
Nov 29, 2011
Messages
23
hoping you can help me/ with what is probably an idiot mistake but when i run the script i get the error
Code:
clearempty.py: Command not found.
 

fracai

Neophyte Sage
Joined
Aug 22, 2012
Messages
1,212
Where did you place the script file?
What exactly are you typing when you run it?

That message sounds like you're typing: "clearempty.py" and the file isn't in your path. You'll need to do one of the following: put it somewhere that is in your path, add the location to your path, or specify the full path to the script when running it.
 

srit

Junior Member
Joined
Nov 29, 2011
Messages
23
That was it. Thanks. Thought i tried it with the full path...guess i didnt.
 

fracai

Neophyte Sage
Joined
Aug 22, 2012
Messages
1,212
Glad I could help. Let me know if you need anything else or have any suggestions regarding the scripts.
 

f4242

Member
Joined
Mar 16, 2017
Messages
94
Hi,

Doesn't work anymore with FreeNAS-11 under Python 3. I fixed the issue around echo that need parentheses, but my Python skills are not enough to fix things like that:
Traceback (most recent call last):
File "/mnt/pool0/bin/zfs-rollup/rollup.py", line 126, in <module>
name,property,value = snapshot.split('\t',3)
TypeError: a bytes-like object is required, not 'str'
 

fracai

Neophyte Sage
Joined
Aug 22, 2012
Messages
1,212
Thanks for the report. I suppose this is a good motivator to update it to python 3.

I'm not immediately sure what the fix is here, but I should hopefully have a chance to work on this later today.

Pull requests are welcome as well ;-)
 

fracai

Neophyte Sage
Joined
Aug 22, 2012
Messages
1,212
So I didn't get much time to look at this last night and I don't have my login for github right now or I'd put in some effort right now.

Can you comment if python2 is still available? That would be the easiest solution to just specifically reference that interpreter. There's even a pull request that I haven't accepted, but really should have already, that would do exactly that.

Also, did you create the latest fork on github? If so, thanks for the headstart. There are a few things that need to be fixed still (arguments to print moved back inside the parens, 'end=""' added to some of the statements) but it's a good start for me regardless.

Anyway, I should actually get a chance to work on this tonight. Sorry for the delay.
 

f4242

Member
Joined
Mar 16, 2017
Messages
94
Yeah I started to work on it but quickly reached my low python skills limits :)

There is also Python 2.7, but I still get an exception with rollup.py:
python2.7 /mnt/pool0/bin/zfs-rollup/rollup.py --recursive --intervals 30M:336,daily:30,weekly:4,monthly:1 /mnt/pool0
Traceback (most recent call last):
File "/mnt/pool0/bin/zfs-rollup/rollup.py", line 140, in <module>
print("ignoring:\t", dataset+"@")+snapshot
TypeError: can only concatenate tuple (not "str") to tuple
I also use clearempty.py and this script work fine with Python 2.7 on FreeNAS-11.
 

f4242

Member
Joined
Mar 16, 2017
Messages
94
There is also Python 2.7, but I still get an exception with rollup.py:.
Oh, forget that. This bug was introduced by me when trying to make it working with Python 3. The original script work fine with the supplied Python 2.7 in FreeNAS-11.
 

fracai

Neophyte Sage
Joined
Aug 22, 2012
Messages
1,212
I've merged the pull request which brings explicit usage of python2, which should be good for now. I'll investigate python3 this evening and weekend.

Thanks again.
 
Joined
Jul 17, 2015
Messages
2
does anyone have rollup.py working on Freenas 11.2? get the following when I try to run it.

line 87
print "invalid count: "+count
^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print("invalid count: "+count)?
 
Top