Time Machine Not Deleting Old Backups

[buzzle]

Hello-

This is my first post to this site so I hope I'm posting to the correct forum.

I'm somewhat new to TrueNAS and I have been using TrueNAS SMB share to backup several Mac laptops. Each laptop has its own TimeMachine SMB share and I've had things configured and working with TimeMachine for awhile now. The TimeMachine backups are configured on each Mac device and the scheduled backups work just fine. The issue that I keep having, and have attempted to solve, is when the TimeMachine backups reach the dataset quota. The quota itself works as I would expect, that is I get warnings that the dataset is almost full. The issue that I haven't been able to solve for is having TimeMachine be able to remove older images in order to make room for a new backup, at least this is my understanding of how TimeMachine works.

The dataset and its childern have a quota set for 256 GiB. The storage capacity on the Macbook is 256 GB with 121 GB available, so it would seem that I have enough storage space for several backups, but it seems that TimeMachine cannot manage the space allocated for TimeMachine on the SMB share.

I'm running TrueNAS Core 13.0-U6 on TRUENAS-MINI-3.0-XL+ and the SMB share has the following properties enabled

Time Machine
Use Apple-style Character Encoding
Enable Alternate Data Streams
Enable SMB2/3 Durable Handles
Auxiliary Parameters: tmprotect:retention = 7

Has anyone had this issue recently and if so, is my understanding correct in that TimeMachine *should* be able to manage remote SMB TimeMachine backup space? I've searched these forums for a similar issue but havent found a resolution for anyone recently.

What are some configurations that I can check on the share or dataset or SMB share?

How can I troubleshoot this issue further so that I can see if its something else like permissions on the share, etc that may be the cause?

 

[buzzle]

So...no one's had this issue? Ever? Did I ask the wrong forum? I've been trying to solve this issue for awhile now and would appreciate *any* direction. Thanks.

 

[anodos]

Time machine is supposed to limit its own space usage. There isn't anything we can really do server-side to fix an issue of it not pruning its own snapshots. Newer versions of MacOS allow specifying setting up a disk usage limit within the app while setting up the time machine target. You can also make sure you are fully up-to-date on MacOS client.

 

[Constantin]

Another thing to keep in mind is snapshots. You may have a disk limited to 256GB of backup space and Time Machine will prune against that. But TrueNAS also counts the snapshot space required to deal with the target ... and that may be a lot more than the space you allocated for Time Machine per disk quota. So the the dataset space you allocate to Time Machine may have to be several times larger than the quota that dataset presents to Time Machine. Make sense?

 

[buzzle]

Time machine is supposed to limit its own space usage. There isn't anything we can really do server-side to fix an issue of it not pruning its own snapshots. Newer versions of MacOS allow specifying setting up a disk usage limit within the app while setting up the time machine target. You can also make sure you are fully up-to-date on MacOS client.

Thank you anodos for your response. If I remember correctly, the space available to TimeMachine when I configured TimeMachine was the full 256 GB and that's what I had configured. I'm running latest version of MacOS Sonoma.

I've attempted to troubleshoot this issue in the past by looking at SMB share configurations and permissions, but nothing really jumps out. I don't know if there are ways to set log levels higher on either the TrueNAS or MacOS side to get better information as to why it seems TimeMachine cannot prune backups on the SMB share.

 

[buzzle]

Another thing to keep in mind is snapshots. You may have a disk limited to 256GB of backup space and Time Machine will prune against that. But TrueNAS also counts the snapshot space required to deal with the target ... and that may be a lot more than the space you allocated for Time Machine per disk quota. So the the dataset space you allocate to Time Machine may have to be several times larger than the quota that dataset presents to Time Machine. Make sense?

Thank you Constantin for your response.

That is a good point and one I hadn't considered. I am running daily snapshot tasks on those datasets. I'm not sure how much space zfs snapshots use typically use, but the TimeMachine backups have been running fine for months until it runs out of space on the SMB share to perform the needed backup. I've seen TimeMachine backups fail with and error message that TimeMachine was only able to complete 0% fo the backup because there is insufficient free space on backup disk.

Looking at my allocated space for dataset for one of my macs that is failing TimeMachine backups i have a quota of 256 GiB, Used: 252.16 GiB and Available 3.84 GiB. I executed the command

zfs list -ro space tank1/ds1/timemachine/lapmacjm01 and got the following output

NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHILD

tank1/ds1/backups/timemachine/lapmacjm01 3.84G 252G 256K 352K 0B 252G

tank1/ds1/backups/timemachine/lapmacjm01/tmgolf 3.84G 252G 163G 89.6G 0B 0B

The 'tmgolf' dataset was created by TimeMachine and I believe that is the sparsebundle file? If I understand the output correctly it shows that the 'tmgolf' dataset has 163 GB used by snapshots and 89 GB used by the dataset which adds up to 252 G used space.

Then I looked at the tmgolf dataset snapshots in the TrueNAS webui and can see my scheduled snapshots 'auto-2023-12-13_01-45' for example and also see other snapshots that appear to be made by TimeMachine such as 'aapltm-1684183035'.

This explains what is making up that 163 G of USEDSNAP. I havent figured out how to break this out between how much the aapltm snapshot are taking and how much the 'auto' snapshot are taking.

I still think there may be something preventing TimeMachine from being able to manage those 'appltm' snapshots tho. The auto snapshots may be taking away from TimeMachine's available space, but I would think that TimeMachine would be able to prune backups unless there's something preventing that. This is where I'm wondering is I have a dataset configuration issue or a SMB permission issue. I wish I could get more information on the TimeMachine side of this equation too, like bump log level on MacOS.

 

[j4cbo]

This confused me too...

The "aapltm" snapshots are not created or managed by Time Machine. They're created by TrueNAS itself, which has a module called "tmprotect" that automatically takes a ZFS snapshot when a Time Machine backup completes.

This is, IMO, a fantastic feature - just severely under-documented. About all I've been able to find is the forum post by the engineer that introduced it. As discussed there, the reason for it is that Time Machine has an unfortunate tendency to corrupt its sparsebundle file sometimes, which breaks the entire backup target and not just that specific backup. Using a ZFS snapshot allows a second level of restore in case that happens. It can similarly protect against user errors or ransomware. The Mac is able (through accident or malice) to corrupt or delete the sparseimage that it's backing up to, but it won't be able to tamper with the ZFS snapshots.

tmprotect is always enabled on "Multi-User Time Machine" shares (formerly "Enhanced Time Machine") and can only be enabled on other shares through the CLI. There's a feature request to make it more configurable.

 

[Stux]

I’ve been having issues with TM backups approaching the assigned limits too. I haven’t finished troubleshooting yet.

At first I didn’t really realize there was an issue, and just increased the quota.

Next I started increasing the percentage of quota utilization before warning (which is where I am at the moment)

I also found @anodos post about the “aapltm” snaps when investigating them the other day, which got me thinking that maybe I’m using the wrong form of quota.

I believe for TM to work properly the quota needs to be the current dataset usage not including snaps, if that makes sense.

Anyway, I haven’t solved the issue yet, but it’s something I’m working on at a very slow rate as my primary system has a very large disk ;)