ZFS Rollup - A script for pruning snapshots, similar to Apple's TimeMachine

[fracai]

I haven't upgraded yet to check it, but I did make an update a while ago to explicitly depend on python2. I should really update it for python3 regardless.

Are you using the latest version? Alternatively, you can make exactly the edit that it suggests; change the print line to use parenthesis.

 

[KazuyaDarklight]

does anyone have rollup.py working on Freenas 11.2? get the following when I try to run it.

line 87
print "invalid count: "+count
^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print("invalid count: "+count)?

Just popping on to say I have the same thing happening as Todd, I'm also having to invoke as "python rollup.py" instead of just ./rollup.py for some reason. I get "No such file or directory" otherwise. I'm going to make the requested tweaks. I just copied this from BitBucket, so it should be the latest.

 

[KazuyaDarklight]

And having done that, I now get this. So I've exceeded my ability to debug. I think this script is fantastic in concept and cross my fingers for v11.2+ compatibility in the future.

Code:

Traceback (most recent call last):
    File "rollup.py", line 126, in <module>
          name,property,value = snapshot.split('\t',3)
TypeError: a bytes-like object is required, not 'str'

EDIT: Ah, but I fix it for myself somewhat by running as "python2 rollup.py" I may just be too much of a newb for this.

 

[KazuyaDarklight]

I do have a functional question regarding how the intervals work because I just accidently cleared a lot of ours out. (I'm fine, we have a backup unit if anything does come up, so its not actually lost.)

So, in short, what does it mean to set an interval of hourly:5,daily:14, etc? I thought that was how many of each level one was keeping, as in, hourly:48,daily:14 would mean that I keep the last 48 hourly entries, and 14 daily entries. But thats clearly not true because I ended up clearing out everything except the last 2 days, though I'll also say, that reality doen't seem to fit the verbos readout given during test. I've skimmed through the thread and read the readme. I've either hit a bug, or I'm just not getting it.

 

[fracai]

your understanding of the intervals is correct, so if it didn't do that something does seem like it's wrong. Could you post or PM the verbose output? I'd like to fix what ever is broken, or needs clarification.

 

[tortue]

Just thought I'd add a comment here that the version of rollup.py on the github page at the beginning of this resource (updated Feb 9, 2020) works with Freenas 11.3. Some may find that their existing rollup.py doesn't work after upgrading to 11.3 since Python 2 was removed. Just need to update the file with the later version which uses Python 3 and everything should keep on functioning as before.

I'm not sure when the rollup script was updated to Python 3 as I set up my cron job a handful of versions back and have never had an issue so I never touched it.

Awesome work fracai!

 

[fracai]

Thanks, I'm glad you're finding the scripts useful.

I did just recently update everything to support Python 3. Previously they just tried to force using python2, but it sounds like that's not available anymore. It was well past time upgrading them to 3 anyway.

Let me know if there are any features that would make these more useful.

 

[Mastakilla]

Thanks for the super handy script!

Isn't this script worth being a resource perhaps? That could make it easier to find...
I also struggled a bit with how to use it. I had to make a few assumptions.

To prevent new users from bumping into the same issues, here is an example of what I did:
Please let me know if I made any wrong assumptions or if there are better "practices" to follow, like:

• Using a non-root user in the wheel group
• Running it from a jail

Code:

root@data[~/zfs-rollup]# mkdir /root/zfs-rollup
root@data[~/zfs-rollup]# vi /root/zfs-rollup/rollup.py
    <copy paste the script in there>
root@data[~/zfs-rollup]# chmod 750 /root/zfs-rollup/rollup.py
root@data[~/zfs-rollup]# ./rollup.py -rv -i hourly:48,daily:14,weekly:12,monthly:24,yearly:10 hgstpool/mediads
    <lots of output>
root@data[~/zfs-rollup]# ./rollup.py -rtv -i hourly:48,daily:14,weekly:12,monthly:24,yearly:10 hgstpool/mediads
                 @auto-2020-07-25_18-00 - - w m y - 157248
                 @auto-2020-07-27_02-00 - - w - - - 50410362912
                 @auto-2020-08-01_02-00 - - - m - - 76051221792
                 @auto-2020-08-03_02-00 - - w - - - 325311168
                 @auto-2020-08-10_02-00 - - w - - - 130760448
                 @auto-2020-08-15_02-00 - d - - - - 112467264
                 @auto-2020-08-16_02-00 - d - - - - 106998528
                 @auto-2020-08-17_02-00 - d w - - - 100114560
                 @auto-2020-08-18_02-00 - d - - - - 102613056
                 @auto-2020-08-19_02-00 - d - - - - 107994432
                 @auto-2020-08-20_02-00 - d - - - - 110178432
                 @auto-2020-08-21_02-00 - d - - - - 111681024
                 @auto-2020-08-22_02-00 - d - - - - 110615232
                 @auto-2020-08-23_02-00 - d - - - - 109828992
                 @auto-2020-08-24_02-00 - d w - - - 109828992
                 @auto-2020-08-25_02-00 - d - - - - 104901888
                 @auto-2020-08-26_02-00 - d - - - - 105006720
                 @auto-2020-08-27_02-00 h d - - - - 32760000
                 @auto-2020-08-27_03-00 h - - - - - 32725056
                 @auto-2020-08-27_04-00 h - - - - - 32620224
                 @auto-2020-08-27_05-00 h - - - - - 32637696
                 @auto-2020-08-27_06-00 h - - - - - 32655168
                 @auto-2020-08-27_07-00 h - - - - - 32637696
                 @auto-2020-08-27_08-00 h - - - - - 32620224
                 @auto-2020-08-27_09-00 h - - - - - 32969664
                 @auto-2020-08-27_10-00 h - - - - - 32952192
                 @auto-2020-08-27_11-00 h - - - - - 33808320
                 @auto-2020-08-27_12-00 h - - - - - 33301632
                 @auto-2020-08-27_13-00 h - - - - - 32637696
                 @auto-2020-08-27_14-00 h - - - - - 32637696
                 @auto-2020-08-27_15-00 h - - - - - 32620224
                 @auto-2020-08-27_16-00 h - - - - - 32620224
                 @auto-2020-08-27_17-00 h - - - - - 32655168
                 @auto-2020-08-27_18-00 h - - - - - 32637696
                 @auto-2020-08-27_19-00 h - - - - - 32620224
                 @auto-2020-08-27_20-00 h - - - - - 32620224
                 @auto-2020-08-27_21-00 h - - - - - 32637696
                 @auto-2020-08-27_22-00 h - - - - - 32847360
                 @auto-2020-08-27_23-00 h - - - - - 32008704
                 @auto-2020-08-28_00-00 h - - - - - 32008704
                 @auto-2020-08-28_01-00 h - - - - - 32235840
                 @auto-2020-08-28_02-00 h d - - - - 32026176
                 @auto-2020-08-28_03-00 h - - - - - 31991232
                 @auto-2020-08-28_04-00 h - - - - - 32917248
                 @auto-2020-08-28_05-00 h - - - - - 32637696
                 @auto-2020-08-28_06-00 h - - - - - 32637696
                 @auto-2020-08-28_07-00 h - - - - - 32620224
                 @auto-2020-08-28_08-00 h - - - - - 32620224
                 @auto-2020-08-28_09-00 h - - - - - 32655168
                 @auto-2020-08-28_10-00 h - - - - - 33703488
                 @auto-2020-08-28_11-00 h - - - - - 33720960
                 @auto-2020-08-28_12-00 h - - - - - 33319104
                 @auto-2020-08-28_13-00 h - - - - - 33319104
                 @auto-2020-08-28_14-00 h - - - - - 33720960
                 @auto-2020-08-28_15-00 h - - - - - 32620224
                 @auto-2020-08-28_16-00 h - - - - - 32620224
                 @auto-2020-08-28_17-00 h - - - - - 33686016
                 @auto-2020-08-28_18-00 h - - - - - 34559616
                 @auto-2020-08-28_19-00 h - - - - - 61746048
                 @auto-2020-08-28_20-00 h - - - - - 32917248
                 @auto-2020-08-28_21-00 h - - - - - 32637696
                 @auto-2020-08-28_22-00 h - - - - - 32637696
                 @auto-2020-08-28_23-00 h - - - - - 32026176
                 @auto-2020-08-29_00-00 h - - - - - 32026176
                 @auto-2020-08-29_01-00 h - - - - R 32288256

Code:

root@data[~/zfs-rollup]# mkdir /root/zfs-rollup
root@data[~/zfs-rollup]# vi /root/zfs-rollup/clearempty.py
    <copy paste the script in there>
root@data[~/zfs-rollup]# chmod 750 /root/zfs-rollup/clearempty.py
root@data[~/zfs-rollup]# /root/zfs-rollup/clearempty.py -r hgstpool
hgstpool/backupds
         auto-2020-08-09_00-00 0
         auto-2020-08-16_00-00 0

 

[Phil1295]

thank you, I like the idea as I was missing it when I recently migrated from Netgear. They also have a periodic retention logic that can be setup from GUI (named Smart snapshots).
Anyone can confirm that teh previous post setup example is the way to go ? Best regards

 

[2twisty]

Gonna give this a bump.

Does this work with TrueNas 12 Core? I would assume so, but I don't want to risk my snapshots without more info.

 

[tortue]

Yes, I use it with truenas 12 core still.

 

[Stux]

FWIW, seems to work okay with a replicated dataset and TrueNAS Core 13-U4

More or less ended up using @Mastakilla 's instructions

Now, I think I saw a post somewhere detailing how to use with 5minute snapshots?