Autodelete issue in Periodic Snapshot Tasks

[CSylvain]

Hello,
I am running TrueNAS SCALE 23.10.0.1, and I use Snapshots with one weekend and one month retentions.
Although it is well defined:
Retention: Will be automatically destroyed at 2023-12-10 22:00:00 by the periodic snapshot task.

Absolutely nothing happens, I have to intervene manually each time to delete this Snapshot.

It seems this problem is general, as I have this on several installations including local.

Do you have the same problem on your side ?

THANKS !

 

[winnielinnie]

What is the actual schedule / frequency of your Periodic Snapshot Task?

(Not the expiration. The actual schedule. Such as "once per day" or "once every hour" etc.)

Sharing a screenshot will help.

 

[CSylvain]

What is the actual schedule / frequency of your Periodic Snapshot Task?

(Not the expiration. The actual schedule. Such as "once per day" or "once every hour" etc.)

Sharing a screenshot will help.

Sorry for this long delay !

You would see two, one every week, and another every month.
They are both set to a value of 1, and therefore correspond to the automatic deletion message, but which does not work.

 

[winnielinnie]

So you have it set to take a snapshot once per week, and expire after a week? What's likely happening is that it won't prune your "expired" snapshot until the next schedule. Such a Periodic Snapshot Task lands too "close" for the schedule and expiration.

Have you tried waiting longer than two weeks?

 

[CSylvain]

This is exactly it, and the first image with the red arrow confirms (in French) that it will be automatically destroyed on the date indicated by the periodic snapshot task.
I noticed this problem when I found myself with my disks full, and several Snapshots that should have been deleted weeks ago.

I can wait another week without doing anything manually, and tell you if it got deleted.
The next one will be today, I will come back to you in the next few weeks.

 

[CSylvain]

So you have it set to take a snapshot once per week, and expire after a week? What's likely happening is that it won't prune your "expired" snapshot until the next schedule. Such a Periodic Snapshot Task lands too "close" for the schedule and expiration.

Have you tried waiting longer than two weeks?

I couldn't wait any longer because I'm being harassed by emails with this quota exceeding 80%.
After two days of waiting, it still didn't delete itself automatically, so I had to do it manually again.

Does anyone use this section of Snapshot and auto-delete ?
Otherwise I would have to do this under a script that purges automatically, but that's one more hack, even too much.

 

[ABain]

I did a test this morning, using a 1 hour retention, I found that the retention was longer than 1 hour but they did get deleted, can you try a very short retention time and see if the delete happens at all for you? If not I would recommend filing a bug ticket using the link at the top of the forum. I have raised a ticket to for the extended retention.

 

[winnielinnie]

I couldn't wait any longer because I'm being harassed by emails with this quota exceeding 80%.

If you waited, you would likely have seen them getting pruned.

It's from what I mentioned earlier: Even if a snapshot is beyond its expiration, it will not get pruned until the periodic snapshot task runs again. (There are not two different processes that run independently of each other. There is only one process that combines creation and pruning.)

 

[ABain]

@winnielinnie this is exactly the case, the UI could be clearer as today it implies the deletion will happen at a specific date/time, rather than the next period task after this date/time.

 

[NugentS]

I just checked my 1 week retention snapshots, which get taken at much more frequent intervals (every 1 hour in some cases). They are being deleted correctly (I don't have any hanging around with older dates). Even the replicated snapshots on another server are being deleted correctly

 

[winnielinnie]

Wait, wait, wait. Not to go off topic, but: Why do you have snapshots that individually hold onto nearly 2 TiB of data? I just noticed that. This means you're deleting/modifying ~2TiB worth of data between snapshots?

 

[CSylvain]

Hello,
Yes, because these are one-week and one-month backups of web and mail servers in archives. This is why you have this rate in Tb.

By deleting manually, the operation does not even take 2 to 3 minutes to make space available.

I just checked my 1 week retention snapshots, which get taken at much more frequent intervals (every 1 hour in some cases). They are being deleted correctly (I don't have any hanging around with older dates). Even the replicated snapshots on another server are being deleted correctly

So you advise me to take Snapshots in several parts while keeping them for a week ?
I would like to try the experience.

At worst, if this is not done according to a fixed schedule, I can resume what I was doing with Hetzner Storage, use the TrueNAS API and take my Snapshots as well as clean up myself from a Cron which will have a schedule fixed to do this task.

### EDIT ###

If you waited, you would likely have seen them getting pruned.

It's from what I mentioned earlier: Even if a snapshot is beyond its expiration, it will not get pruned until the periodic snapshot task runs again. (There are not two different processes that run independently of each other. There is only one process that combines creation and pruning.)

I hadn't seen your post, I think you've put your finger on the problem, so I'm going to take Snapshots with shorter intervals so that the previous one can be deleted, I won't fail to warn you of the result.

 

[winnielinnie]

Yes, because these are one-week and one-month backups of web and mail servers in archives. This is why you have this rate in Tb.

How are you backing these up? cp? rsync? third-party software?

 

[CSylvain]

How are you backing these up? cp? rsync? third-party software?

Different solutions under Virtualmin (SCP), Zimbra in CIFS (Samba) with the Zextras solution therefore complete and incremental, all this on night shifts.

 

[winnielinnie]

Are large files being "updated" with each backup? (Not simply adding/removing a bunch of files in folders; but actually "updating" very large files themselves.)

 

[CSylvain]

It’s a little bit of everything and enters into the private domain of my clients.
This concerns several servers that I manage and which automatically back up to a TrueNAS account.
To understand clearly, TruenNAS is installed on a dedicated server with 4x 6TB disks and a 500Gb NVMe as an OpenZFS cache, I do not use it at home.

Perhaps this is what makes you wonder about this rate in Tb ?

 

[winnielinnie]

Perhaps this is what makes you wonder about this rate in Tb ?

It's not the size, per se. It's that so much is being deleted within a week, which the snapshot still "holds onto". It doesn't seem like an efficient way to backup to a NAS server.

What if only a few new emails were downloaded? An entire zipped archive file with all its gigabytes has to be sent and written to the server every time?

To share an alternative example: I backup my entire Thunderbird inbox every day. Because I use rsync with the "--inplace" option, the only additional storage needed with each backup is only a few megabytes or less. I have snapshots that go back a year, with daily backups of my email inbox.

I'm not adding a ton of storage with each daily backup. If I were, my NAS would surpass its capacity very quickly.

 

[CSylvain]

As I mentioned above, there is a complete backup once a week and the others are incremented, this is the perfect recommendation because restoring it from incrementing can become a horror, especially under Virtualmin which does not does not have a perfect Rollback.
But that's another debate !

Thank you very much for the details regarding why my Snapshots are not deleted.