Restore missing zfs intermediate snapshots on NAS from remote backup.

[tnuser9999]

I am testing different disaster recovery scenarios on two truenas core boxes (13.0-u5). For backups, in this example let's say a business requirement would be to keep one month of snapshots on the backup server to restore from. I am pulling snapshots to the backup server with replication retention set as shown below so if by mistake snapshots or datasets are deleted on the server being backed up, they will not be erased.

snapshots on the server being backed up (some have been deleted!) :

snapshots on the backup server:

Oops snapshots deleted! As you can see from the error message "No incremental base on dataset 'zp0' and replication from scratch is not allowed." We have an issue, how do I get those snapshots that are missing restored back to the remote server they were pulled from? Removing the snapshots from the backup (destination) is not an option, let's say business requirement is always have 1 month of restore points.

 

[tnuser9999]

I tried replication in the other direction but received the same error message: "No incremental base on dataset 'zp0' and replication from scratch is not allowed."

 

[tnuser9999]

I have been reading some other post that makes it sound like what I am asking for is a way to "resync" the backup and nas snapshots. I didn't find instructions on doing so though.

 

[sretalla]

You're trying to do time travel... what you're discovering is that much as your timeline from the past is intact on the backup, your "real" timeline has now changed, so it's not a case of just being able to go back to where you started.

Think of a family photo... take one you have from 1980 (or some year that makes sense for you)... now you have a family with more/new people in it and you effectively want to re-take an exact replica of that family photo now, but there are different people in the family and the family members (may) look different, so it won't work.

What you can do is use the .zfs directory in each of the datasets on the backup to rsync back the missing content to the original.

You will need to more-or-less restart your replication from scratch after that though.

 

[tnuser9999]

Ah thank you, the explanation helps. I was misunderstanding snapshots. If I understand correctly, once the snapshots are removed from source, there is no restoring back from the destination. I guess unless you were to delete the source dataset and then replicate push all back, losing the recent snapshots/dataset on source. You can't "insert" old snapshots in a new timeline.

Also if I am understanding correctly, it sounds like a "restored" state would be to compare the source dataset to the destination dataset (maybe via rysnc -navc ? ) and zfs diff on source and destination in snapshots for any changes that may be undesirable and copy back files from from destination pool/dataset/.zfs/snapshot/files to source dataset. After verifying all data, delete all snapshots on destination pull server, and execute a replication again.

This is really good to know before I start replicating. It really shows the dependency the replication destination has on source, visa versa. I guess this is where the difference is from having a backup versus replication. I may should should explore backuppc plugin, I used that for backups in the past. I like the speed of replication however I could see potential for losing backups.

 

[sretalla]

The only real reason you would want to restore back a snapshot over a source that has changed ince that snapshot was replicated to it would be if there's a complete mess on the source due to something like a cryptolocker or a catastrophic loss of the pool.

Otherwise, putting back the damaged parts by using rsync in the .zfs/snapshots path to go back with missing/incorrect content is the best option (which actually does allow replication to continue on as normal after that).

Also if I am understanding correctly, it sounds like a "restored" state would be to compare the source dataset to the destination dataset (maybe via rysnc -navc ? ) and zfs diff on source and destination in snapshots for any changes that may be undesirable and copy back files from from destination pool/dataset/.zfs/snapshot/files to source dataset. After verifying all data, delete all snapshots on destination pull server, and execute a replication again.

That sounds more-or-less right if I understood you correctly, but see my point above for perhaps a better way back to normal after some comparison and rsync to correct issues.