FreeNAS 9.10 pylibzfs

[Andreas Corneliussen]

Hi Everyone

As mentioned in my introduction thread, I would like to stop being a cheapskate regarding hardware (which if I buy it to cheap end up dying on me), but also on backup systems. This is not entirely true. I would like to start backing up, but I am still working hard to make the lowest cost solution fit my need.
I therefore have the following plan for backing up my Dataset with around 180 Gigs of images:

1. Make one startup snapshot and zfs send > initial_backup_file
2. Take the initial_backup_file and add some parity on top to make it resilient to errors
3. Send the initial_backup_file and send it to amazon glacier
4. Each day make a new snapshot, calculate the size of an incremental zfs send of this snapshot and if < 1M destroy the snapshot
5. If the snapshot is not destroyed make an incremental zfs send of the changes between the newest snapshot and the one before that
6. pad the incremental snapshots with parity and send to amazon glacier
7. When a fixed amount of incremental files have been transfered to amazon, start over with a new initial backup file
8. Keep the full two newest sets of incremental snapshots in amazon glacier

Given that I have around 180 Gigs of images and I will add something like 20 Gigs of parity to this and store two full sets of incremental changes I expect the monthly cost of this backup to be in the range of 2$.

I have grown quite fond of using python as my day to day programming language for high level programming (i.e. not embedded targets), and I have therefore taken the boto.glacier library and made a class with all the amazon glacier functionality that I have the need for in this project. For the ZFS handling I have found py-libzfs which should give handy python functions for all the actions I need to to on the Zfs pool. And this is where I get to the root of my issue that I need help with.

I have made a jail where I have installed python 2.7, ipython and py27-libzfs-1.0.20170319. When I then try to use the ipython prompt to ask for a list of pools I get the following:

import libzfs
pools = list(libzfs.ZFS().pools)
pools
[]


I may have been able to foresee this as there is no reason my jail should have any knowledge about the underlying filesystem and therefore no reason that it should be able to see, let alone make changes to, the diskpool of the base OS of FreeNAS. Therefore, the script I write will have to be executed in the context of the base OS, but here I am unable to install any packages as I understand.

Can any of you point me in a direction of solving this dilemma that is better than fetching the source code of py-libzfs and trying to make it run in the context of the python installed in the base OS of my freenas?[/CMD]

 

[Andreas Corneliussen]

Hot damn I feel stupid. I have waited for anyone to answer this thread and stopped looking at the problem. Then today I decided that I would try to investigate how much work it would take to make the libzfs library work in the base os of freenas just to find a comment in a commit of the makefile in the git project that mentioned freenas. 10 sec. later i tried testing it in an ssh to the base freenas os and found that the libzfs python bindings are already present in this os so I don't need to do anything else.
Only problem I am left with right now is that the glacier bindings I have written have been written for boto2 as that had the most natural glacier bindings as I saw it, whereas the only boto3 is installed in the freenas base os. But it should be possible to migrate the glacier bindings to boto3 and everything should be fine.