TrueNAS 12 U1 creates automatic (kind of) empty snapshots

[tim64]

Hi,

yesterday afternoon I activated the shown task for automatic snapshots of my main Pool "datapool" (yes, silly name. ). Its the only snapshot-task.

So, the system should take snapshots every three hours (from the Pool itself and the child datasets), but only if there are changes.

But when I checked today I saw a lot of snapshots that show some data change, but are taken at a time (this night) when definitely no one changed any data (the datasets are shared via SMB).

Example for the "Media"- dataset that really did not change at all since yesterday.

Code:

zfs list -t snapshot|grep Media
datapool/Media@auto-2020-12-11_15-00                                              341K      -     6.03G  -
datapool/Media@auto-2020-12-12_00-00                                              114K      -     6.03G  -
datapool/Media@auto-2020-12-12_03-00                                              114K      -     6.03G  -

The last snapshot was at 03:00 this night (no snapshots are created at 06:00, 09:00, .... so, this happens not always)

And when I try to show the diff with
zfs diff datapool/Media@auto-2020-12-12_00-00 datapool/Media@auto-2020-12-12_03-00
nothing is shown (no output at all).

Why are those snapshots taken when there was no change in the dataset and what am I doing wrong, or is this a bug in TrueNAS-12.0-U1?

Thanks alot.

P.S. When I do a "zfs diff" from snapshots where the data in the dataset was changed, the zfs diff shows the changes. So the "zfs diff" itself works.

 

[tim64]

I have now tested this behaviour further during a time when there was definitely no write access to the SMB-shares.
The "Allow Taking Empty Snapshots"-option in the snapshot-task is not set.

I read-accessed some files via SMB (txt,pdf,png,jpg,... the file type doesn't matter) and then an automatic-snapshot was not only created at the next snapshot-time but also at the snapshot-time after the next one.

Two examples:

I accessed a file at 14.12.2020 01:00 and two snapshots (14.12.2020 03:00 and 14.12.2020 06:00) are created.
I accessed a file at 17.12.2020 23:00 and two snapshots (18.12.2020 00:00 and 18.12.2020 03:00) are created.

And the "zfs diff" of these snapshots shows nothing, not even a header line.

Here are the screenshots of this examples:

Is this a "normal" behaviour that can be explained (and changed) or is this a bug?

thanks.

 

[tony95]

I would also be interested in knowing the answer to this

 

[karlfife]

We had a system with the same behavior. TLDR; Solved. Not a bug. An issue with one datasets resulted in multiple empty snapshots.

The facts:
Observation 1. With an automatic snapshot task configured, and with "Allow Taking Empty Snapshots" disabled, we found that empty snapshots taken on certain datasets and certain children which were seemingly empty (e.g. according to #zfs diff pool/dataset@snap1 pool/dataset@snap2). That is to say, unexpected empty snapshots occurred, but not to all datasets, and when they did happen, they didn't happen to all children--only to certain children in certain datasets, and their parents.
Observation 2. Normally, if zfs sees a change to a child dateset, change also accrues to the child's parent dataset. This is expected behavior. Thus, a recursive snapshot triggered by a change to pool/grandpa/dad/son will result in seemingly empty snapshots of pool/grandpa/dad and in pool/grandpa. This is normal. However the 'son' dataset was unchanged in our case. Indeed 'son' was NOT being snapshotted by the task (as requested), however its parents and grandparents WERE! Something's fishy! In this case, experimentally excluding the 'son' dataset from the recursive snapshot task (in the TrueNAS task definition GUI) resulted in no more 'empty' datasets being takenof son's parents and grandparent! Whats going on here? Son was a problem child.

The cause: Even though 'son' dataset it was completely unchanged, the 'son' dataset was different than the other datasets in the pool because it had been 'zfs sent' from a different pool. Notably, it the send had started, but it was interrupted part-way through the send. Thus 'son' was an incomplete dataset. No ongoing changes were being made to the 'son' dataset. Indeed, the remote sending system was offline, and local snapshot task was correctly honoring the 'do not take empty snapshots' directive'. However for some reason, the incompleteness of the son WAS being interpreted by ZFS as change to the parents and grandparents!

Again, in our case the remedy was to exclude the dataset from the recursive snapshot task, either by using the exclude feature, or by moving it out of the recursive hirearchy, or simply by deleting the dataset, or completing the send. With these details in mind, let me know what you find in your caase!

 

[winnielinnie]

Are you using native encryption?

Besides that, any small modification of metadata is considered a "change".