Replication and snapshots lifetime?

[Dunuin]

Hi,

I got a pool "HDDpool" and want to backup it aperiodically every 2 - 6 weeks to the pool "BACKUPpool" using a replication task or a custom backup script using "zfs send | zfs recieve" but I still want daily/weekly snapshots so I'm able to rollback.

I created a recursive daily snapshot task running every day from monday to saturday with a snapshot lifetime of 7 days.
And I created a recursive weekly snapshot task running only every sunday with a snapshot lifetime of 4 weeks.

I'm not sure how some options of the replication task are working in detail.

The "hold pending snapshots" option sounds like it would prevent snapshots from being auto deleted if they are needed by the replication for incremental tasks:

Prevent source system snapshots that have failed replication from being automatically removed by the Snapshot Retention Policy.

1.) What happens if I only add the weekly and not the daily snapshot task to the "periodic snapshot tasks" list of the replication task? Will the daily snapshots be protected too?
2.) What happens if I disable "run automatically" in the replication task to just do the backup manually. Will the snapshots be protected even if the replication doens't "failed" because it is only run once every 2 to 6 weeks?
3.) Is it ok to set the "Snapshot Retention Policy" to "Same as Source" or will this break the incremental replication because daily snapshots will be deleted after a week even if they are needed?

What is the right way to do this so that every snapshot needed to incrementally replicate a pool will be saved by the snapshot retention policy?
Is it possible to do this without setting the snapshot lifetime of daily and weekly snapshot tasks to something very long like 3 months?

 

[Heracles]

What is the right way to do this

Hey @Dunuin,

The proper way to do this is to replicate all your snapshots (including your daily or more frequent ones) to your backup. There is no gain not doing it.

Snapshots point to each others for the content. If you do daily snapshots and compare them to a weekly one, the size of the 7 daily ones will be pretty close to the size used by the weekly one. The only difference would be a file that has been created after the first weekly snapshot and deleted after a second daily snapshot. Ex: Your weekly snapshots are taken on Sundays. You take one, as well as a daily one. You create a file on Monday. That one is deleted on Tuesday. This file will be part of the daily snapshots but would have been missed by the weekly one.

The file created or modified on Monday but untouched since will not be re-added to your other snapshots.

So the 7 daily snapshots will not add perfectly to the size of a weekly snapshot, but in most cases, the sum will be very close.

By transferring all your snapshots, your backup will be more precise, so you risk loosing less data in case of a disaster / restore from your backups.

 

[Dunuin]

So I should just create daily snapshots and keep them for 4 weeks instead of creating weekly and daily snapshots with different snapshot lifetimes?

How are you replicating your offline backups so the snapshot aren't deleted if not replicated already?

 

[Heracles]

Hi,

I suggest you design a multiple generation snapshot strategy that will fit your needs. Here, mine is :
Snapshot every 15 minutes, kept for 3 days
Snapshot every hour, kept for 3 weeks
Snapshot every day, kept for 3 months
Snapshot every week, kept for 18 months
Snapshot every 4 weeks, kept for 4 years

That way, I have a very precise snapshot to recover from an incident short time after it happened. The longer the time goes, the less precise the recovery is but still, I can recover something up to 4 years after the fact if that resources existed for at least 4 weeks or have been there at the right time when a 4 week snapshot was taken.

The total number of snapshot is not too bad and to find what you are looking for is not too difficult. Should you have too many snapshots, like one every 15 minutes over 4 years, it would be a pain to search anything in them.

As for my offline backup, indeed, many snapshots will not make it there and I do not mind. That offline copy is the last resort and is updated about twice per month. As such, I do not expect a more precise recovery from it.

 

[Dunuin]

As for my offline backup, indeed, many snapshots will not make it there and I do not mind. That offline copy is the last resort and is updated about twice per month. As such, I do not expect a more precise recovery from it.

That is the point I'm unsure about.

If, for example, I use your snapshot strategy and only select the weekly snapshots to replicate because I'm fine with that If I only mount the backup pool every several weeks, then the replication task will also send daily and hourly snapshots to the backup pool if they were created after the last replication and before the creation of the latest weekly snapshot. How will the automated snapshot task handle the clean up in this case? I tried some replications and sometimes the whole pool was send again instead of just the small changes because the source server was missing the snapshots needed by the backup pool to continue an incremental backup.

I'm not sure how and when the deletion of old snapshots will be prohibited if they are needed for a incremental replication but the snapshot lifetime is exceeded already.

If I activate "hold pending snapshots"...
1.) will snapshot deletion be prohibited only for the snapshot tasks selected by the "periodic snapshot tasks" dropdown or for all snapshots sent to the backup pool and the periodic snapshot task that created these snapshot doesn't matter?
2.) will snapshot deletion be prohibited if the replication doesn't fail but simly doens't run for a long time? I would like to deactivate "run automatically" and start the replication manually.

 

[Heracles]

How will the automated snapshot task handle the clean up in this case?

Snapshots have an expiration date. They are deleted automatically after that date. Should part of one to-be-deleted daily snapshot is required for another, that content will be "moved" to the next snapshot referring to it. Here, my 4 weeks snapshots are all created about the size all others are (only a few hundreds of megs) but they grow each time a 15min, 1hour, 1 day, 1 week snapshot gets deleted. The content that is still needed for them is moved and added until that one will expire itself a few years later.

 

[Dunuin]

Snapshots have an expiration date. They are deleted automatically after that date. Should part of one to-be-deleted daily snapshot is required for another, that content will be "moved" to the next snapshot referring to it. Here, my 4 weeks snapshots are all created about the size all others are (only a few hundreds of megs) but they grow each time a 15min, 1hour, 1 day, 1 week snapshot gets deleted. The content that is still needed for them is moved and added until that one will expire itself a few years later.

But is not important that identical snapshots are found on the backup host and the source host for incremental updates of the replication task? Or are the snapshots unimportant as long as there is a snapshot that contains the same data?

 

[Heracles]

Hi again Dunuin,

As mentioned, when the content of an expiring snapshot is needed for another one to be complete, that content is "moved" from the to-be-deleted snapshot to the next one needing it. That is true for both the source and destination. That way, the destination indeed contains all the same snapshots as the source, as long as it is up-to-date. Should I boot up my offline server, it will have some expired snapshots to remove, and so will do it by itself. It will merge the content that is required in another snapshot and will then reach the same state as the source, except for the newest snapshots that were not sync yet.

Once the source detects that the destination is available, it will update it from wherever the last sync point was to now. That may well involve snapshots that were taken separately at first and were merged after the fact. In that case, this merged snapshot is what will be transfered as is.

That way, bot servers stay in sync.