Setting 2 different snapshots (lifetime, periode) for one pool?

[pcace]

Hi,

i am very new to TrueNAS and ZFS, so please excuse my probably kind of stupid question:

i have a ZFS Pool here, wich i want to backup using snapshots to another pool. My goal is to keep lets say two weeks of daily snapshots, and 3 years of monthly snapshots of this pool. this must be possible (?), but i am just not sure how to do that.
do i need two different replication tasks? or do i need one replication task with two selected (and different) periodic snapshot tasks?
Right now it looks like this: one Replication Task with two different periodic Snapshot tasks. But: What do i do with the naming? Should montly and daily have different names, or are they all the same but have different lifetimes?

The Replication Task:

And its settings, with both selected snapshot tasks:

And here are the snapshot tasks:

Both look like this:
(yes the times are now set that close to test the behaviour)

Is this how i would set this up? How about naming of the snapshots. Right now both snapshot tasks produce the same names. Is that a problem?
I am not quite confident that this is what i want to achieve ;)

Any suggestions?

THank you so much!!

 

[Heracles]

this must be possible (?)

It sure is. Here, I use a 5 generations plan (15 minutes for 3 days, 1 hour for 30 days, 1 day for 90 days, 1 week for 1 year, 4 weeks for 4 years).

but i am just not sure how to do that

You just create as many snapshot tasks as you wish, one per frequency. As such, here, I have 5 snapshot tasks.

do i need two different replication tasks?

No. You need multiple snapshot tasks but a single replication task.

To be sure it works, once you are configured, go on your target pool and see if snapshots from both plans are indeed present. If they are, you are in business.

 

[pcace]

Thank you for your superfast reply!
So it seems that i am on the right track ;)
How do i figure out wich snapshot is from wich snapshot task since the naming is the same (i do see Snapshots being created on the target pool)
and what i dont understand, how does TrueNAS (or is this a ZFS feature?) decide wich snapshot needs to be deleted after lifetime ended? Is the lifetime something each snapshot knows by itself?
Or maybe asked in another way: as far as i understand each snapshot copies only the delta to the next older one. if i have now two snapshot tasks running and making snapshots with different lifetime - but the same name, how can i be sure that not one importand (needed) snapshot in the chain gets deleted because it ran out of lifetime? (or am i overthinking it?)

Cheers and thanks!!

 

[winnielinnie]

Beware that you can accidentally destroy your longterm snapshots since you're using the same exact naming convention for multiple snapshot tasks.

Simply "unchecking", "renaming", or disabling/removing the longterm snapshot task will destroy all your longterm snapshots the next time your "daily" snapshot task runs.

I explain it in more detail in this post:

Staged snapshot schedule

Hi! How would I best go about creating a schedule that creates snapshots of a dataset like this: 48 hourly 31 daily 48 monthly 12 yearly I think that I should create a separate schedule for each "frequency", but is there a better way?

www.truenas.com

 

[pcace]

Hi,

thank you so much for your reply!
I think i do understand the Problem, but i have 2 more questions to that:

- if i have 3 snapshot tasks, does that mean, that i do need 3 times the size of the source pool to actually store them?

and one more not really related to the problem:
what happens if for example there is a snapshot of 100GB woth of data. then i delete all the data (via SMB share) and copy the same data on the share again. (sounds stupid, but just for understanding). then a day later a new snapshot will be created. Will ZFS realize that it is the same data then on the old snapshot, or will it not (as far as i understand it would then make the snapshot twice as big).
Maybe someone could clarify this a littlebit?

Thank you so much!!

 

[winnielinnie]

- if i have 3 snapshot tasks, does that mean, that i do need 3 times the size of the source pool to actually store them?

No. Snapshots are just that: a snapshot of an entire dataset (all records and metadata) in a particular moment in time. Such records and metadata can exist in multiple snapshots. The amount of space that a snapshot "uses" is determined by the unique records that only that specific snapshot contains (if such records do not exist in any other snapshots or the live filesysem). If a dataset contains 10TB of data, and you then create a new snapshot named @snap001, then later you save a new 512KB file and create a new snapshot named @snap002. Both snapshots refer to a total of 10TB of data. (So if you revert the dataset to either of those snapshots, it will still be 10TB in size. However, @snap002 refers to an additional 512KB of records unique to itself.) If you delete the 512KB file, nothing will change in "total used space" for the dataset, as @snap002 still exists and has pointers to those extra 512KB of records. But if you then destroy @snap002, you will only "free up" an extra 512KB, since that is what is unique to @snap002 and nothing else.

So while @snap002 refers to 10TB of data, only 512KB of data was unique to it.

Will ZFS realize that it is the same data then on the old snapshot, or will it not (as far as i understand it would then make the snapshot twice as big).

ZFS is copy-on-write block-based, not file-based. So those will be seen as new records (even if the content is exactly the same that was previously deleted.)

This is why it's imperative to use "in-place" edit/write operations for the greatest efficiency. Much software makes a copy of the file being modified (for legacy and safety reasons, such as archivers and editors), which is unfortunate for ZFS, as this creates new records, which will consume more space with snapshots as time moves on.

 

[pcace]

Thank you so much for your great explanation!
Still something to ask:

This is why it's imperative to use "in-place" edit/write operations, for the greatest efficiency. Much software makes a copy of the file being modified (for legacy and safety reasons, such as archivers and editors), which is unfortunate for ZFS, as it creates new records, which will consume more space with snapshots as time moves on.

I understand this that you mean it will consume twice (as my example above) the amount of space even the files are the same.
Is there any way to remove / delete snapshots? As far as i understand the concept removing snapshots would destroy the whole structure (?).

so how would i lets say save space if there is nothing left for the snapshots?
in my small test here i have a 4TB Pool (one disk) and a 12TB Pool (4 Disks). The 4TB Pool will replicate its snapshots to the 12TB Pool. If i now do that:
Add 4TB --> make snapshot --> delete 4TB --> make snapsh. --> Add 4TB (...and this 3 TImes) the 12TB Pool would be full right?
What then? I cannot delete any old snapshots because it would destroy the whole chain, but i also cannot add more. What happens then?

Cheers and thank you for your patience!

 

[winnielinnie]

Add 4TB --> make snapshot --> delete 4TB --> make snapsh. --> Add 4TB (...and this 3 TImes) the 12TB Pool would be full right?

Why would anyone do that? This is "a solution to a problem that doesn't really exist." The rest of your question after this is moot. I mean that sincerely, not trying to sound rude.

Worst-case scenario, you can prune older snapshots on the source dataset, while making sure that the source and destination share a common "base" snapshot with which to be able to keep doing incremental replications. (Without such a common base snapshot, an incremental replication is not possible.)

The Replication Task GUI has an option to safeguard against expired snapshots being removed from the source before they had a chance to make it to the destination. Not sure how well it works, never tried it myself.

UPDATE: To get around the concern of unnecessarily creating redundant records, you can use SMB/NFS to delete/rename/move/edit files directly on the server, and hopefully use "friendly" software that does in-place modification. And if using something like "rsync" to sync files to the server, make sure to use its "--inplace" and "--no-whole-file" options.

UPDATE 2: If you really want to get advanced, you can leverage ZFS's "bookmarks" feature. However, I don't find it very useful, it's not exposed in TrueNAS's GUI, and there are better ways to manage replications and use space wisely.

A bookmark allows you to create a "placeholder" on the source, which will always take up zero space, that references an actual snapshot on the destination. This means that the source can save space by constantly removing snapshots, and rather than use an actual snapshot as the common "base" snapshot for incremental replications, it will use a bookmark instead. This allows for incremental replications, without requiring the source to hold on to old (and possibly large) snapshots, in which the destination has greater capacity to hold more snapshots. The cost is that the incremental replications are slower, since it must calculate (from the within the destination itself, against its own snapshots) what records need to be sent over. It also means that if something were to happen to the destination, the source does not contain any data from the snapshots (since bookmarks are just placeholders), and thus you lose any unique data found only in those snapshots.