Dataset Deletion During Replication

[Johnny Fartpants]

Hi All,

I have one FreeNAS 9.10 system replicating to another system using ZFS send and recv and if I choose to delete a dataset on the primary system it also deletes that dataset on the replica system which 99% of the time is great. However, what if I accidentally deleted a dataset on the primary then it would be deleted on the secondary and be gone forever. Also what if someone gained access to my primary system and deleted the whole pool then although I have a replica this would also be deleted.

Is there anyway of putting something in place that would prevent this from happening? I imagine if there is it would cause me more work maintaining the replica but it currently feels like a vulnerability.

Thanks

 

[hugovsky]

I think that if you have a determined schedule to make the sync(instead of sync immediately), it should only sync at that time, giving you some space to recover if needed.

EDIT: didn't test this. Can't tell for sure.

 

[Johnny Fartpants]

Thanks for your response. Thats not a bad idea. As default I snap every hour but replicate always so as soon as I delete its gone. Only problem is you have to set a time from and too with replication so you can't say do it every hour, you can only say do it between this time and that. So thinking about it that wouldn't work. Thanks anyway for the input.

 

[styno]

What you are looking for is snapshot retentions. For example: you setup an hourly recursive snapshot schedule of your pool with a two weeks retention period. Then you replicate for example from source:source pool to target:target pool/sourcepool-bkp.
This would not only allow you to restore your (accidental) deletes on the source pool from snapshot but as snapshots are replicated as well you could also restore them from target.

 

[hugovsky]

What you are looking for is snapshot retentions. For example: you setup an hourly recursive snapshot schedule of your pool with a two weeks retention period. Then you replicate for example from source:source pool to target:target pool/sourcepool-bkp.
This would not only allow you to restore your (accidental) deletes on the source pool from snapshot but as snapshots are replicated as well you could also restore them from target.

Something like this?

 

[toadman]

What you are looking for is snapshot retentions. For example: you setup an hourly recursive snapshot schedule of your pool with a two weeks retention period. Then you replicate for example from source:source pool to target:target pool/sourcepool-bkp.
This would not only allow you to restore your (accidental) deletes on the source pool from snapshot but as snapshots are replicated as well you could also restore them from target.

I think his use case is testing whether what you said is true. He's clearly replicating snapshots, though we don't know to what time the retention is set. But he is then deleting the source dataset, which means he necessarily deleted all of the source snapshots. It sounds like the replication script detects all snapshots are gone and therefore deletes all snapshots on the target system. Which makes sense to me as it's supposed to replicate one way and I can confirm it does delete target snapshots when source snapshots are deleted.

Possible solution: Uncheck "Delete stale snapshots on remote system" in the replication task. Then either let the target snapshots expire with the snapshot lifetime (which I think is what @styno is suggesting), or run a script on the target that pares down old snapshots. For the latter see this: https://forums.freenas.org/index.ph...napshots-similar-to-apples-timemachine.10304/

I have used @fracai's script for years and it works well.

 

[toadman]

Something like this?

View attachment 18668

That is two snapshot tasks. One is one per hour with a retention of 1 day, the other is snap once per day with a retention of 2 weeks. Which is of course fine if that's what you want to do. I think @styno was suggesting one snapshot task snapping once per hour but with a retention of 2 weeks.

I think your issue is in the replication task. You won't want the replication to delete stale snapshots on the target. Uncheck the box I suggested in my previous post.

 

[hugovsky]

@toadman Understood. Thanks.

 

[styno]

@toadman : exactly! Thanks for pointing that out! I am on my phone with no access to the GUI so this is from the top of my head.
Also without double-checking: nothing should stop you to add a separate long-term snapshot policy on the backup system.

 

[Johnny Fartpants]

Thanks for all the responses. I'm afraid these suggestions don't work however in prticukar the 'delete stale snapshots'. It is still the case that if you delete the dataset on the primary this automatically deletes the dataset and all its snaps on the replica. Thanks for the suggestions however. My only thought is i currently have replication running all the time however I might change this to run at 6pm for example. That would give me approx a 24 hour window to recover deleted datasets on the secondary in the event of this issue arising.

 

[hugovsky]

I've tested it and it works. If I remove test dataset and snapshot, remote dataset and snapshot remained. Or I might not doing it right....:)

 

[Johnny Fartpants]

That's strange. I'll test again but when I tried before it deleted the remote dataset too. Out of interest what snapshot schedule were you replicating?

 

[Johnny Fartpants]

Ok I think I've figured out what you did. You created a dataset, setup a snapshot schedule and replicated it. Therefore when you deleted the dataset it in-turn deleted the snapshot schedule which in-turn deleted the replication hence why you didn't see the dataset in the replica disappear. Although this is interesting it would not work for me as I snapshot the parent and my concern is if I delete a child dataset which upon testing again does delete it from the replica regardless of the retention policy and regardless of if delete stale snapshots is ticked or not. Thanks again for your feedback however its very much appreciated.

 

[styno]

Also without double-checking: nothing should stop you to add a separate long-term snapshot policy on the backup system.

Did you try this?

 

[Johnny Fartpants]

Did you try this?

Im not sure how you would do this? I can't see a way of the replica having a different retention policy to the source.

 

[hugovsky]

Ok I think I've figured out what you did. You created a dataset, setup a snapshot schedule and replicated it. Therefore when you deleted the dataset it in-turn deleted the snapshot schedule which in-turn deleted the replication hence why you didn't see the dataset in the replica disappear. Although this is interesting it would not work for me as I snapshot the parent and my concern is if I delete a child dataset which upon testing again does delete it from the replica regardless of the retention policy and regardless of if delete stale snapshots is ticked or not. Thanks again for your feedback however its very much appreciated.

Yes, that's what I did.

 

[toadman]

Im not sure how you would do this? I can't see a way of the replica having a different retention policy to the source.

You would set a periodic snapshot on the target (backup) system for the particular dataset of interest. So then you'd have two things causing snapshots on the target, (1) those sent over from the source during replication and (2) the periodic snapshot task on the target.

I have no idea whether that would work or not. Depends on what the replication script actually does when it deletes the target dataset and potentially on what user you have running the scripts (i.e. enough privleges). But it's worth a try I suppose.

 

[Johnny Fartpants]

You would set a periodic snapshot on the target (backup) system for the particular dataset of interest. So then you'd have two things causing snapshots on the target, (1) those sent over from the source during replication and (2) the periodic snapshot task on the target.

I have no idea whether that would work or not. Depends on what the replication script actually does when it deletes the target dataset and potentially on what user you have running the scripts (i.e. enough privleges). But it's worth a try I suppose.

Good idea. I'll give it a go and report back.
Thanks

 

[Johnny Fartpants]

You would set a periodic snapshot on the target (backup) system for the particular dataset of interest. So then you'd have two things causing snapshots on the target, (1) those sent over from the source during replication and (2) the periodic snapshot task on the target.

I have no idea whether that would work or not. Depends on what the replication script actually does when it deletes the target dataset and potentially on what user you have running the scripts (i.e. enough privleges). But it's worth a try I suppose.

So the answer is it gets its knickers in a twist. Regardless of what the schedule is set on the replica if you try to delete the dataset from the primary is won't let you. Which in-turn does prevent the accidental deletion that was my main concern but not quite what I had in mind. I think I'll stick with what I've done for now which is tell the replication only to run at the end of everyday. This gives me a window of time to recover data before it is lost forever by the replication. Still think there is a potential feature in there somewhere.

 

[Stux]

You would set a periodic snapshot on the target (backup) system for the particular dataset of interest. So then you'd have two things causing snapshots on the target, (1) those sent over from the source during replication and (2) the periodic snapshot task on the target.

I have no idea whether that would work or not. Depends on what the replication script actually does when it deletes the target dataset and potentially on what user you have running the scripts (i.e. enough privleges). But it's worth a try I suppose.

When i played around with replications, if you make a snapshot on the target system, the target system has diverged. There is a flag you specify in the ZFS recv command which causes a rollback to the last common snapshot. And that means the backup will either succeed and you lose your snapshot, or fail, and you retain it.