Scale backup to non-ZFS system

[dalnew]

Hey all, sorry if this has been posted before but I couldn't find any concrete threads that explain all this in the forum. If it has been explained please feel free to point me to the relevant threads. My scenario is that I have a ZFS machine with ~100tb of storage that I need to backup to a non-zfs storage machine (Synology specifically). This means I can't really take advantage of zfs send/receive snapshotting that would obviously be preferable to backup (and restore) my server. So it seems I will be relegated to using the old rsync to push data. If I simply rely on rsync to manage the changes it can take an exceedingly long time to do the necessary checksums, etc and so I would, ideally, like to avoid that and leverage the ZFS snapshots to manage this.

My system config is roughly:

• 1 NVME boot drive (not mirrored)
• 1 NVME "scatch" drive - Contains system dataset/ix-applications/container config info/etc - (not mirrored)
• 6 drive RAIDZ2 array (long term data storage and backups of the scratch drive datasets

As such I have setup my system back like this:

1. Take snapshots of all relevant datasets.
2. Do a zfs diff on those snapshots to the previous snapshots to capture any changes
3. Parse that diff (see below) and combine into a single list of files/directories that have changed
4. Rsync only the contents of that list

For the diff I simply wrote a little script that takes the diff format and parses into a single file of changes to rsync
M /mnt/tank1/renders/A
- /mnt/tank1/renders/A/art.mp4
M /mnt/tank1/renders/B
+ /mnt/tank1/renders/B/bocce.mp4

I suspect I could simply maintain the directories that had content change and then rsync only those such that anything that was deleted would automatically be removed and anything added would be added. Otherwise I could list the specific files/directories added/removed if I knew rsync would actually delete the files that aren't there.

For non-data sets like the boot drive I was simply copying /data as well, although I've seen mention of the TrueNAS scale config being backed up to /var/db/system/configs-* as well, but I haven't been able to determine what I actually need to be backing up.

My main goal, of course, is to have a complete set of backups that could be used to restore my entire TrueNAS Scale machine should something catastrophically fail, but unfortunately it wasn't clear exactly what I needed to do to do that besides the data, configurations, etc. I recently had one of my Truenas NVME drives failed (I don't have enough slots to have a mirror unfortunately, and so I had to manually delete and restore the pools from the redundant HDD backups. It wasn't a 2 second process, but it wasn't hard either. I guess I'm trying to make sure that if other significant failures happen in the future, like the boot drive or god forbid the entire RaidZ2 array that I can restore the datasets, or at the very least the data.

Given all this here are my actual questions:

1. Right now I am only backing up data from my zfs -> synologys as it is not a zfs machine. Is there a better way to backup datasets to a non-zfs machine such that dataset (metadata?) would also be backed up but the files also be directly accessible? Ie to accelerate recreating the datasets+data once a failed zfs machine is usable again
2. What should I actually be backing up for the system/boot drive? Are /data and /var/db/system/configs-* both needed? Is one prefered?
3. Is there an intelligent way to manage this through the GUI? I couldn't find a way to setup an rsync job that does something specific like this without rsyncing everything.
4. Any way to automate the zfs diff of snapshots other than manually determining the last two, diffing, and then constructing a list? Or better yet setup an rsync task that leverages diffs directly?

Thanks for any help!

 

[Arwen]

Some answers:

1. No. You are asking for attributes of ZFS on a non-ZFS file system, not possible as far as I know. It may be possible to translate some attributes into another file system. Beyond me.
2. You don't backup anything from the boot pool, except the TrueNAS configuration.
3. This is done through a script, which can be cron'ed through the GUI
4. See #3

If this were a full backup, then you can perform a ZFS send as a file. Since the output is just a data stream, you capture the stream instead of receiving it;
zfs send PARMS POOL | ssh -nq SYNOLOGY dd of=OUTPUT_PATH
Of course I don't know if you can SSH into a Synology or if you would do it via a network mount. Same principle would apply.


Your question #1 does bring up an interesting point. Some of TrueNAS configuration is inside ZFS. But, in the case of a disaster, and complete loss of a data pool, how would you re-create it exactly the same?

 

[dalnew]

Some answers:

1. No. You are asking for attributes of ZFS on a non-ZFS file system, not possible as far as I know. It may be possible to translate some attributes into another file system. Beyond me.
2. You don't backup anything from the boot pool, except the TrueNAS configuration.
3. This is done through a script, which can be cron'ed through the GUI
4. See #3

If this were a full backup, then you can perform a ZFS send as a file. Since the output is just a data stream, you capture the stream instead of receiving it;
zfs send PARMS POOL | ssh -nq SYNOLOGY dd of=OUTPUT_PATH
Of course I don't know if you can SSH into a Synology or if you would do it via a network mount. Same principle would apply.


Your question #1 does bring up an interesting point. Some of TrueNAS configuration is inside ZFS. But, in the case of a disaster, and complete loss of a data pool, how would you re-create it exactly the same?

So it seems most of what I'm doing is correct then. The /data directory has the TrueNAS configuration and that is being backed up. What that config actually contains is a bit beyond me though which is why I was asking about the dataset configurations. Obviously the aspect of ZFS that includes snapshots and history would be lost unless I replicate that on the synology as well. That is of course possible using BTRFS by sending a snapshot from zfs to the syno and snapshotting there as well, however I don't want to get into that level of detail as replicating a bunch of old data on the backup doesn't seem super worthwhile to me.

The pool/dataset configuration is a bit of a mystery to me. It would be awesome if it were possible to replicate the configuration of the pools and dataset(s). When my NVME failed I basically had to start over with all the pool config. For some reason it wouldn't let me just replace the drive and keep all the config, as it didn't even have a drive available to "replace". The datasets were easily re-instantiated because they were replicated in my RAIDZ2 pool as a backup, however if that also failed I would only have the files, not the dataset info.

For example: I could have a pool with ashift=12 and other specific params and a dataset hierarchy like:
tank
dataset1 (readonly, record size 128k)
dataset2 (record size 1MB, dedup off)
dataset3 (dedup on, inherit on)
tank2
dataset4 (recordsize 256k, dedup on)

and each dataset might have it's own configuration that I believe I will have to manually recreate if all I was left with was the Synology data... or maybe I'm wrong and the Truenas config backs that info up? If that is the case then this is much easier. I just need to backup the TrueNAS config, reconstruct the failed ZFS machine and all the pools datasets would be recreated and then I just need to "fill in" the data from the synology.

It can't be that easy can it? :)

 

[Arwen]

For my non-NAS computers, (desktop, laptop & media server), I have a special script that saves the ZFS configurations of the 2 / all pools, (OS pool & remaining storage). It captures pool attributes, dataset attributes and dataset tree. Runs every hour, over writing the prior hour of the day, but keeps 7 days worth. These get backed up along with other recovery information.

One of the failings of backup schemes is that full recovery needs some information before recovery begins. Like disk layout. Assume my existing disks lost their minds and had nothing, I need to partition them as before. Then, if I am doing a network restore, I need to know the IP address, mask & router(s) before I can proceed.

Having performed a real disaster recovery at work, I had to think of these things. So at home I take special care to capture what I think is going to be enough for a complete full restore.

 

[dalnew]

Hmm not sure how I would automate capturing zfs pool/dataset information in a way that allows for automatic reconfiguration later. Let me investigate, thanks.

Also another thing I realized that currently there's no guarantee that what I'm pushing change-wise from the snapshots isn't being modified while the backup is happening. When using zfs directly it's only pushing the file versions from the snapshot when you zfs send a snapshot. When I do a snapshot diff to get the filenames it is possible, however unlikely, that those files could be updated while the backup process is running. Obviously I can stop dockers, turn off smb shares, etc to restrict access and hopefully prevent changing the data while the backup is going on but it would have been better if there were some way to send the files from the latest snapshot instead of just sending the filenames which may or may not actually represent that snapshot.

Know of any way to do that?

 

[NugentS]

Any reason you aren't using Active Backup for Business on Synology?
[Having an older Synology that doesn't support Active Backup for Business is a very good reason]

 

[dalnew]

I've used synology's HyperBackup in the past, however my main issue with it is that it results in a bunch of binary blobs that have no correlation to the actual files, eg it saves everything in some proprietary way that means the data is inaccessible until you restore it back to the original server. Is Active Backup the same? I'd prefer a method of backup that leaves the data accessible while the restore is going on.

Another reason I don't use the synology side backup tools is that I need/want to be able to control the services running on the truenas while the backup is happening to prevent access, which would mean I'd have to stop everything on the truenas side, kickoff hyper/active backup on the syno side, and then restart things back on the truenas side.

In the end it felt easier to just control everything from the truenas side, but maybe I'm missing something? I would love to use some automated tool to do all this, but between wanting the backups to be fast (hence using the snapshot diffs to only backup actual changes without having to checksum all the files to determine diffs), and having to control access to the files while being backed up, it just seemed simpler to automate this from TrueNAS itself. I would love for someone to show me an easier way and tell me that I am way overcomplicating this :)

In an ideal world there would be some way for me to actually send the versions of files from a specific snapshot, which would guarantee no one is modifying and would prevent me from having to shutdown services etc. In fact as I was writing this I just stumbled on this link: https://superuser.com/questions/1287578/backup-zfs-pool-using-rsync
which seems to indicate it may be possible to mount a zfs snapshot in a separate directory and then send files directly from that, thus preventing me from having to stop services AND guaranteeing that I'm sending the actual snapshot file versions. It still doesn't address an automated way of backing up and restoring pool/dataset configurations though... unless I'm missing something about the truenas configuration that does that?

Any reason ActiveBackup would be preferable to the above?

More and more I feel like I should just dump the syno machine and built a simple zfs backup machine, however love the fact that my syno box can leverage SHR and can continue to use all my older drives of varying sizes and maximize their usage without having to dump all my older drives for new matching drives.

 

[NugentS]

Active Backup for Business (not Hyperbackup)- the files are not useable from the Synology and will need to be restored somehow. You can get at them from the synology if the source machine vanishes completely
However it does a proper incremental/differential backup. The first backup takes a while, but further backups are reasonably quick. Yes it has to search for changes. Retention can be configured quite well.

I do a ZFS replication job for most stuff to a second TN and a SMB backup for all SMB shares to a Synology