[HOW TO] iocage plugin backups

[blooper98]

Look, if there's one thing you should know about me, its that I make mistakes. That's why I like robust backups. But if there's one thing nerds like me enjoy more than backups, its correcting people. Feel free to leave your corrections below, starting with "Um, actually..." to earn a point :)

The following is a mash up of info various forum posts, blogs, the help docs, and other random tid-bits. Thank you to everyone who contributes & shares.

Who is this for?
People who can't afford a dedicated virtualization box to complement their truenas box, and people who want to tinker.

Who does that leave? Home users who have just enough money for a robust NAS, but not enough for a dedicated virtualization box. Frankly, if that didn't describe me, I would argue that those people don't exist. But there must be more of us! I'm not shouting into the void... right?

The "I don't know how this thing works" method
Lets say you download a plugin and you get it working. Maybe it just works, or maybe you sit down with a guide, a glass of whiskey, and a whole bunch of headache. This won't work if you've added mount points (see the tale of two plugin types below in that case). Regardless, you're not really sure how the plugin works, and you don't care to learn - frankly, a reasonable opinion. iocage export is your friend. This tool creates a backup file of your entire plugin jail that you can replicate off to the cloud (or not!) and reinstall it with a simple iocage import command. Let's take a look at an example backup script that will be executed as a once-in-a-while (but not more than once per day) cron job:

Code:

#!/bin/sh

# substitute your jail name
JAIL_NAME=mosquitto

# create a new backup
iocage stop $JAIL_NAME
iocage export $JAIL_NAME
iocage start $JAIL_NAME

This will create two files in your pool like (assuming a pool named tank) /mnt/tank/iocage/images/mosquitto_YYYY-MM-DD.zip and /mnt/tank/iocage/images/mosquitto_YYYY-MM-DD.sha256 where YYYY-MM-DD is the date the export command is run.
All of these backups can be replicated to the cloud with your provider of choice using a "cloudsync task" via the GUI for the /mnt/tank/iocage/images directory.

OK, but that's too many backups...
Lets say you want to keep just the most recent backup. To do this use the following modified cron job:

Code:

#!/bin/sh

# substitue your jail name and pool name as needed
JAIL_NAME=mosquitto
POOL_NAME=tank

# delete previous backups
rm "/mnt/${POOL_NAME}/iocage/images/${JAIL_NAME}_*"

# create a new backup
iocage stop $JAIL_NAME
iocage export $JAIL_NAME
iocage start $JAIL_NAME

and replicate with a "cloudsync task" as before.

A tale of two plugin types
For the rest of this guide, I consider two types of plugins: those with static configuration files, and those with database-like configurations. Static configuration plugins tend to be a bit simpler, one example being mosquitto. These plugins have the distinct advantage of not requiring down time to backup, since the configuration does not live in memory. Database configurations tend to be a bit more complex: like nextcloud, plex, or emby. These do require down time to backup in iocage, since there's no tool to snapshot memory usage as well as disk.

All of the methods below require customization on a per plugin basis.

Plugins with static config files
One example of this that I have tested is the mosquitto plugin. If you can mount your backed up config file into a new plugin jail and restart the service, you're up and running again! Finding the existing configuration files is not always trivial, but a helpful tool is to run find / -name app_name where app_name the name of the plugin application. This will search the entire jail for files & directories with the same name of the plugin.

Note that all plugins must have their specific configuration files backed up, and since they each have their own unique directories and file names, each plugin backup will have to be unique.

Let's see what that looks like for a mosquitto plugin installation on a pool named tank and the existing & empty data sets or directories: tank/mosquitto, tank/mosquitto/config, tank/mosquitto/db. Then, running the following script as root will install to mount the needed configuration files.

Code:

#!/bin/sh

# install mosquitto plugin and stop the mosquitto service
iocage fetch -g https://github.com/ix-plugin-hub/iocage-plugin-index -P mosquitto --name mosquitto
iocage exec mosquitto "service mosquitto stop"

# tar up the default configuration files and log
iocage exec mosquitto "tar -cf /root/config.tar /usr/local/etc/mosquitto"
iocage exec mosquitto "tar -cf /root/db.tar /var/db/mosquitto"

# delete the default configuration files and log so that the directories can be a mount point
iocage exec mosquitto "rm -rf /usr/local/etc/mosquitto/*"
iocage exec mosquitto "rm -rf /var/db/mosquitto/*"

# mount the default configuration and log directories
iocage fstab -a mosquitto /mnt/tank/mosquitto/config /usr/local/etc/mosquitto nullfs rw 0 0
iocage fstab -a mosquitto /mnt/tank/mosquitto/db /var/db/mosquitto nullfs rw 0 0

# extract the default configuration files and log into the mounted directories
iocage exec mosquitto "tar -xf /root/config.tar"
iocage exec mosquitto "tar -xf /root/db.tar"

# restart the jail
iocage restart mosquitto

To backup the jail simply replicate the configuration files located in tank/mosquitto to another location, .e.g with a "cloudsync task." When reinstalling the jail from backup: destroy the existing jail and use the following

Code:

#!/bin/sh

# install mosquitto plugin and stop the mosquitto service
iocage fetch -g https://github.com/ix-plugin-hub/iocage-plugin-index -P mosquitto --name mosquitto
iocage exec mosquitto "service mosquitto stop"

# delete the default configuration files and log so that the directories can be a mount point
iocage exec mosquitto "rm -rf /usr/local/etc/mosquitto/*"
iocage exec mosquitto "rm -rf /var/db/mosquitto/*"

# mount the default configuration and log directories with backup up configuration files
iocage fstab -a mosquitto /mnt/tank/mosquitto/config /usr/local/etc/mosquitto nullfs rw 0 0
iocage fstab -a mosquitto /mnt/tank/mosquitto/db /var/db/mosquitto nullfs rw 0 0

# restart the jail
iocage restart mosquitto

Note that this simply omits the tar & extract lines.

Plugins with database configs
Two examples of this (that I use) are the nextcloud and emby plugins. These require a database dump to backup, and consequently some down time. Emby has the added simplicity of keeping all the configuration & database information in a single directory, so we'll use that as an example.

When installing Emby, follow a similar recipe as the static config backup, assuming installation on a pool named tank and the existing & empty data sets or directories: tank/emby, tank/emby/db. Then, running the following script as root will install to mount the needed configuration files:

Code:

#!/bin/sh

# install emby plugin and stop the emby-server service
iocage fetch -g https://github.com/ix-plugin-hub/iocage-plugin-index -P emby --name emby
iocage exec emby "service emby-server stop"

# tar up the default configuration files and database
iocage exec emby "tar -cf /root/db.tar /var/db/emby-server"

# delete the default configuration files and database so that the directories can be a mount point
iocage exec emby "rm -rf /var/db/emby-server/*"

# mount the default configuration and database directories with backup up configuration files
iocage fstab -a emby /mnt/tank/emby/db /var/db/emby-server nullfs rw 0 0

# extract the default configuration files and log into the mounted directories
iocage exec emby "tar -xf /root/db.tar"

# restart the jail
iocage restart emby

Then, create a cron job to run the following script and create a backup on a regular (daily, weekly, etc) basis:

Code:

#!/bin/sh

# stop the jail so that the data is static and on the disk
iocage stop emby

# create a snapshot of the jail data
zfs snapshot -r "tank/emby@backup-$(date +%Y-%m-%d_%H-%M)"

# start the jail again
iocage start emby

Here, I don't recommend only using a cloud sync task because your plugin would be down for the entire duration of the task, which might be time consuming. A zfs snapshot instead allows for replication of the snapshot to another dataset, e.g. tank/snapshots/emby, after the jail is up and running again. If you're luck enough to have a zfs backup target, this can be replicated directly to the backup target. If you replicate locally, a "cloud sync task" may take as long as it needs. I like this particular snapshot naming scheme because it will be recognized in the "replication tasks" form under "Also include Naming scheme" with the string backup-%Y-%m-%d_%H-%M. A downside to this method is that the snapshots won't automatically be deleted...TBD.

Making a general script
With all of this, recognizing that we might want a) a formulaic approach, b) log files, and c) a jail export anyway because I make mistakes, consider the following script:

Code:

#!/bin/sh

# this script makes some assumptions about the directory hierarchy
# another option is to manually fill each field
POOL_NAME="tank"
JAIL_NAME="emby"
SNAP_ROOT="${POOL_NAME}/${JAIL_NAME}"
BACKUP_LOG="/mnt/${POOL_NAME}/${JAIL_NAME}/backup.log"
BACKUP_DIR="/mnt/${POOL_NAME}/${JAIL_NAME}/backup"

echo "----------Backing Up Jail ${JAIL_NAME}----------" >> $BACKUP_LOG
echo $(date) >> $BACKUP_LOG

echo "...Stop and export jail" >> $BACKUP_LOG
iocage stop $JAIL_NAME >> $BACKUP_LOG
iocage export $JAIL_NAME >> $BACKUP_LOG

echo "...Delete and replace iocage backup" >> $BACKUP_LOG
rm -fv "${BACKUP_DIR}/*" >> $BACKUP_LOG
mv -v "/mnt/${POOL_NAME}/iocage/images/${JAIL_NAME}_$(date +%F).zip" $BACKUP_DIR >> $BACKUP_LOG
mv -v "/mnt/${POOL_NAME}/iocage/images/${JAIL_NAME}_$(date +%F).sha256" $BACKUP_DIR >> $BACKUP_LOG

echo "...Creating snapshot of state" >> $BACKUP_LOG
zfs snapshot -r "${SNAP_ROOT}@backup-$(date +%Y-%m-%d_%H-%M)" >> $BACKUP_LOG

echo "...Start jail" >> $BACKUP_LOG
iocage start $JAIL_NAME >> $BACKUP_LOG

# Replication taken care in GUI

echo "----------Finished Backup of ${JAIL_NAME}----------" >> $BACKUP_LOG
echo "" >> $BACKUP_LOG

Note that iocage export runs much quicker this way because the export doesn't include the mounted files.