Udev Trigger ZFS Autobackup to Removable Disk [Offsite Backup]

[Jip-Hop]

I wanted to automatically trigger zfs-autobackup when I attach my (external) backup disk. So I made udev-trigger-zfs-autobackup. It's used to store (encrypted) backups offline, on disks you can easily swap and store offsite. Works with USB, eSATA and disks in hotswap trays.

The script and dependencies can be installed in any directory you like (it's using a Python virtual environment). The installation makes no modifications to the system outside of its installation directory. This is to ensure it will survive updates of TrueNAS SCALE. Just store it on one of your data pools, and not on the disks where the TrueNAS base system is installed.

It's still fresh so any feedback is welcome

 

[wolme]

Nice work, thanks, but something goes wrong on TrueNAS Core 12.0-U8.

System: TrueNAS Core 12.0-U8
Setting: WD 8 TB as backup disk, connected via eSata, Hot Plug activated
Problem: connecting the disk does not trigger the script (udev event not detected?)

For installation I followed your description on github.

1. wipefs and sgdisk commands do not work on TrueNAS Core (commands not found). Probably because it's FreeBSD, not Debian like in TrueNAS Scale (?). I used the GUI Tools in TrueNAS Core for wiping.

2a. execution of trigger.sh (and backup.sh) script: zsh (standard) shell in TrueNAS Core cannot process the $BASH_SOURCE[0] parameter ("BASH_SOURCE[0]: parameter not set"). Had to change it to ${(%):-%x} to make it work see https://stackoverflow.com/questions/9901210/bash-source0-equivalent-in-zsh

2b after successful installation of the environment by executing trigger.sh --install and config I tested your script manually. In zsh shell it quits on line 96 with "export:96 invalid option(s)". It seems to quit because command "export" with argument "-f" does not exist in zsh see https://unix.stackexchange.com/questions/59360/what-is-the-zsh-equivalent-of-bashs-export-f.

Using bash instead of zsh it quits with

Traceback (most recent call last):
File "/mnt/scripts/scripts/udev-trigger-zfs-autobackup/monitor.py", line 14, in <module>
monitor = pyudev.Monitor.from_netlink(pyudev.Context())
File "/mnt/scripts/scripts/udev-trigger-zfs-autobackup/venv/lib/python3.9/site-packages/pyudev/core.py", line 62, in __init__
self._libudev = load_ctypes_library("udev", SIGNATURES, ERROR_CHECKERS)
File "/mnt/scripts/scripts/udev-trigger-zfs-autobackup/venv/lib/python3.9/site-packages/pyudev/_ctypeslib/utils.py", line 54, in load_ctypes_library
raise ImportError("No library named %s" % name)
ImportError: No library named udev

The script backup.sh does not work because trigger.sh does not recognize and import the pool.

Would be great if you can help to adjust your script for TrueNAS Core. Would be very helpful. I'am not good at scripting, but if you need further logs, I can assist.

 

[Jip-Hop]

Thanks for testing and reporting! The bash/zsh issues are solvable but the Python ImportError: No library named udev looks like a show stopper unfortunately :( This script will likely stay exclusive to SCALE/Linux. Sorry!

 

[haifisch]

Sounds great! I would like to try it. What are the recommendations to use this now that pip is removed from Truenas?

 

[Jip-Hop]

I'd say just try it out. I don't think it relies on pip being installed on TrueNAS. But I'm not using the script myself any more so I'm not updating the script in case it breaks.

 

[ghanit]

I have found @Jip-Hop 's great script while searching for a way to backup my TrueNAS Scale box to Disks in a HotSwap bay. Because I was not happy with the default zfs-autobackup settings, I started to improve his script on a fork. It can now:

• Have custom and separate zfs-autobackup config for each backup disk
• Can deal with several backup disks thanks to the async udev-trigger
• Has a start and stop script
• Update function to update to the newest zfs-autobackup version
• Includes logging and sends an email upon completion with zfs-output
• Is written mostly in python

Because Jip-Hop is not maintaining his script anymore, I'll leave the link to my fork here. I tested it with my Scale box, but further feedback is of course welcome.

GitHub - ghan1t/udev-trigger-zfs-autobackup: Automatically snapshot and sync ZFS datasets when backup disk is attached.

Automatically snapshot and sync ZFS datasets when backup disk is attached. - ghan1t/udev-trigger-zfs-autobackup

github.com

 

[wolme]

I have found @Jip-Hop 's great script while searching for a way to backup my TrueNAS Scale box to Disks in a HotSwap bay. Because I was not happy with the default zfs-autobackup settings, I started to improve his script on a fork. It can now:

• Have custom and separate zfs-autobackup config for each backup disk
• Can deal with several backup disks thanks to the async udev-trigger
• Has a start and stop script
• Update function to update to the newest zfs-autobackup version
• Includes logging and sends an email upon completion with zfs-output
• Is written mostly in python

Because Jip-Hop is not maintaining his script anymore, I'll leave the link to my fork here. I tested it with my Scale box, but further feedback is of course welcome.

GitHub - ghan1t/udev-trigger-zfs-autobackup: Automatically snapshot and sync ZFS datasets when backup disk is attached.

Automatically snapshot and sync ZFS datasets when backup disk is attached. - ghan1t/udev-trigger-zfs-autobackup

github.com

Hi ghanit, thanks a lot for maintaining and improving the script originally made by Jip-Hop. I'am using it on Proxmox 8.14 with the following pip packages in the python env:

colorama 0.4.6
pip 23.3.2
pyudev 0.24.1
PyYAML 6.0.1
setuptools 69.0.3
wheel 0.42.0
zfs-autobackup 3.2

Two errors I have found so far which do not happen when I use the same zfs-autobackup command without your script. You can find the corresponding logfiles in the attachment.

1. If I backup large amount of data - arround 3 TB, one dataset with 10 subdatasets - with your script, zfs-autobackup will fail after sending/receiving three out of ten datasets. It works with standalone zfs-autobackup command.

2. If I use the option -- set-properties readonly=on (did not test other set-properties options) with your script, zfs-autobackup will not recognize backup-name and target-path arguments. It works with standalone zfs-autobackup command.

I have no clue about the reason of these errors.

 

[ghanit]

Hi @wolme, happy to hear that you're trying out my version of the script!

I'll try to help with my limited python and zfs knowledge and see if I can reproduce it on my system. I have opened two issues on my fork on github, so we don't blow up this thread.

Have you tried with the same versions as in the requirements.txt in my repo? At first I used the newest version of zfs-autobackup that wasn't stable and had a bug.

 

[wolme]

Hi @wolme, happy to hear that you're trying out my version of the script!

I'll try to help with my limited python and zfs knowledge and see if I can reproduce it on my system. I have opened two issues on my fork on github, so we don't blow up this thread.

Have you tried with the same versions as in the requirements.txt in my repo? At first I used the newest version of zfs-autobackup that wasn't stable and had a bug.

Hi @ghanit , thanks for your fast reply. I did not change your requirements.txt. The list with the installed pip packages:

 

[wolme]

colorama 0.4.6
pip 23.3.2
pyudev 0.24.1
PyYAML 6.0.1
setuptools 69.0.3
wheel 0.42.0
zfs-autobackup 3.2

I once tried zfs-autobackup beta 3.3, but I did not work. After that I did a fresh install of your script / python env. I will continue to report in github.

 

[ghanit]

For anyone reading here, the issues should be resolved. Further feedback is welcome.