For those you may be familiar with ZFS Autobackup: https://github.com/psy0rz/zfs_autobackup I've been using this setup for around 8 months now without issue.
I find this to be far superior to basic replication as used in Scale for various reasons. With my slow uploads to offsite replication machine, I'd found at least every month Scale basic replication wanting to restart the whole thing from scratch if my internet connection dropped and something got corrupted, which it does now and then. When it's more than 1TB, that's not good. Then there are the issues with using encryption, and encryption roots and such, no issue at all with ZFS autobackup. It also is able to restart a failed replication where it left off. It also easily handles multiple backup sets of the same datasets without any issues or worrying about snapshots affecting each other. You can also easily pick which datasets you want to replicate without a separate job for each. And so on. This is more for those ZFS folks who may be familiar with this backup tool, or for those who may need a tool to replace the Scale method if it isn't working for them. This method is to install the tool on Scale, but not within kubernetes. This is how to install it, not how to use the tool, for that see the doc in on the github site.
For purposes of these instructions, INSTALLDIR = the directory you are installing the tool in, which should be located on a dataset (or subdirectory of course) so that it is persistent.
With these instructions, we do not modify anything installed by Scale, no additional packages, etc.
That's it! This installs a virtual environment for python so it has the packages needed for ZFS autobackup but without modifying the hosts python packages.
To run the tool from a shell script, do something like:
Note that's what I am doing, you may use many different options. And note I am using ssh so I have stored my ssh key and config in the directorys shown.
If you want to use mbuffer, then, you need to install it somewhere that is included on your PATH. You can install it from the mbuffer site without using apt via simple download.
I'm using this tool to keep a replicated copy of key datasets on a rotating set of external drives (automatic from a script which detects which drive is attached), which goes to a bank vault, and, to keep a replicated copy to a backup ZFS server offsite.
To upgrade the tool to the latest, I simply rename INSTALLDIR/Autobackup to Autobackup-old, and, install again.
If Scale replication is not working for you, you might review the tools github doc to see if this might work for you.
I find this to be far superior to basic replication as used in Scale for various reasons. With my slow uploads to offsite replication machine, I'd found at least every month Scale basic replication wanting to restart the whole thing from scratch if my internet connection dropped and something got corrupted, which it does now and then. When it's more than 1TB, that's not good. Then there are the issues with using encryption, and encryption roots and such, no issue at all with ZFS autobackup. It also is able to restart a failed replication where it left off. It also easily handles multiple backup sets of the same datasets without any issues or worrying about snapshots affecting each other. You can also easily pick which datasets you want to replicate without a separate job for each. And so on. This is more for those ZFS folks who may be familiar with this backup tool, or for those who may need a tool to replace the Scale method if it isn't working for them. This method is to install the tool on Scale, but not within kubernetes. This is how to install it, not how to use the tool, for that see the doc in on the github site.
For purposes of these instructions, INSTALLDIR = the directory you are installing the tool in, which should be located on a dataset (or subdirectory of course) so that it is persistent.
With these instructions, we do not modify anything installed by Scale, no additional packages, etc.
cd INSTALLDIR
wget https://github.com/psy0rz/zfs_autobackup/archive/refs/heads/master.zip
unzip master.zip
rm master.zip
mv zfs_autobackup-master Autobackup
cd Autobackup
python -m venv autobackup-venv --without-pip --system-site-packages
source autobackup-venv/bin/activate
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python get-pip.py
pip install -r requirements.txt
deactivate
That's it! This installs a virtual environment for python so it has the packages needed for ZFS autobackup but without modifying the hosts python packages.
To run the tool from a shell script, do something like:
cd INSTALLDIR/Autobackup
autobackup-venv/bin/python -m zfs_autobackup.ZfsAutobackup -v -d --no-progress --clear-refreservation --exclude-received --keep-source=10,1d1w,1w1m,1m3m --keep-target=10,1d1w,1w1m,1m3m --allow-empty --clear-mountpoint --rollback --destroy-missing=30d --zfs-compressed --ssh-config /mnt/tank/Scripts/Keys/config --ssh-target backup --rate=800k backup Backup/ReplicateNote that's what I am doing, you may use many different options. And note I am using ssh so I have stored my ssh key and config in the directorys shown.
If you want to use mbuffer, then, you need to install it somewhere that is included on your PATH. You can install it from the mbuffer site without using apt via simple download.
I'm using this tool to keep a replicated copy of key datasets on a rotating set of external drives (automatic from a script which detects which drive is attached), which goes to a bank vault, and, to keep a replicated copy to a backup ZFS server offsite.
To upgrade the tool to the latest, I simply rename INSTALLDIR/Autobackup to Autobackup-old, and, install again.
If Scale replication is not working for you, you might review the tools github doc to see if this might work for you.
Last edited:
