SOLVED TimeMachine backups *EXTREMELY* Slow to SMB share (12.0U6)

[lrosenman]

I have 2 macs (one iMac, one MacBook Pro) that back up to my TrueNAS 12.0-U6 server, connected by 1G ethernet, and it's taking HOURS to back up the data,

Attached is a picture of the current status, after having been running since 10ish AM today (UTC-5), and it's now 6ish PM.

Ideas on what I need to tweak/show/fix/etc?

 

[Samuel Tai]

How do you have your Time Machine share configured?

 

[lrosenman]

attached.

 

[Samuel Tai]

How is your SMB service set up, and the underlying dataset, and the permissions? The only thing lacking I can see is the Apple-style character encoding.

 

[lrosenman]

Here ya go

 

[Samuel Tai]

Finally, the options for your smb-tm dataset, please.

 

[lrosenman]

I *THINK* this is what you want?

 

[Samuel Tai]

Your settings look OK on the TrueNAS side, so it's likely something going on at the client side.
According to this post on the Synology forum, there's a couple of things you can try on the MacOS side:

• Disable AFP
• sudo sysctl debug.lowpri_throttle_enabled=0 in Terminal
• Exclude your share from Spotlight indexing via sudo mdutil -d /Volumes/smb-tm

 

[lrosenman]

Thank You for the confirmation that the TrueNAS stuff looks reasonable. I've made those changes and I'll reboot the MBP at some point and see if it speeds stuff up.

Again, thank you.

 

[lrosenman]

It's still slow, so I called Apple Support, and they had me send a bunch of diagnostic stuff and will get back to me next week. I'll post here when I get a solution/answer.

 

[socrates324]

Hi!

It's still slow, so I called Apple Support, and they had me send a bunch of diagnostic stuff and will get back to me next week. I'll post here when I get a solution/answer.

just out of curiosity since I'm facing the same / a similar problem: Did Apple Support provide any insights on your issue?

Thanks in advance!

Cheers,

 

[lrosenman]

I missed their callback, but Monterey fixed the issue for me, FWIW.

 

[mberi]

I'm new to TrueNAS, but I've been using Time Machine Forman years now and here are my observations:

• Time Machine is a heavily throttled process on macOS. Even if your computer is completely idle, it doesn't run even remotely as fast as it should. This is especially noticeable on the first time machine backup (where it's not unheard of to take more than 24 hours). It gets much better on subsequent backups.
  
  
• The command
  sudo sysctl debug.lowpri_throttle_enabled=0
  in Terminal (as mentioned by Samuel above) basically removes this throttling and will make your first Time Machine a lot faster. (It comes down to around 2-3 hours for me)
  
  
• IMPORTANT NOTE: The above command gets RESET when you reboot your Mac.
  
  
• I usually run that command before I start my first Time Machine backup, and then just leave it in the default state after reboot as subsequent hourly backups only take a few minutes at most

 

[socrates324]

For me it was actually hardware related...

I knew about the throttling in MacOS but even with that deactivated, the Backup took more than a night!

Bottom line is: My NAS was equipped with a Realtek NIC. I did not notice any problems when copying files around before, but suddenly playback of Video from the NAS got stuck every now and then.

Checking the FreeNAS logs led me to this: https://www.truenas.com/community/t...o-1-96-to-prevent-re0-watchdog-timeout.88806/

I set the mentioned tuneable and the Video playback worked afterwards. But taking into account the massive problems I've had with Realtek in the early 2000's - I bought an INTEL NIC.

Now TimeMachine without throttling finishes within 2 hours. With throttling enabled it is still very slow. But quite fast in comparison with the speed before using the Realtek NIC.

Just in case someone else has a similar setup / set of problems :)

Cheers,

 

[hybra]

For anyone using Adobe software with large media libraries (Premiere, Lightroom,etc) I found out that the Adobe Media Cache folders were what was deadly slowing my Time Machine (first) backup process.
Stop your current stuck backup, go under your Library folder and delete the three folders () in the Common as per this article:

The Definitive Guide to Adobe Premiere Pro's Media Cache — Video Review & Approval

There are probably files on your system right now, devouring precious hard drive space and you have no idea they’re there! ::cue dramatic music:: Welcome to Premiere’s Media Cache!

screenlight.tv

then restart Time Machine first backup.

I found out that Time Machine was copying thousands of tiny cache files thanks to this command that shows the current Time Machine activity:

sudo fs_usage -wf filesys backupd | grep HFS_update

 

[anodos]

For anyone using Adobe software with large media libraries (Premiere, Lightroom,etc) I found out that the Adobe Media Cache folders were what was deadly slowing my Time Machine (first) backup process.
Stop your current stuck backup, go under your Library folder and delete the three folders () in the Common as per this article:

The Definitive Guide to Adobe Premiere Pro's Media Cache — Video Review & Approval

There are probably files on your system right now, devouring precious hard drive space and you have no idea they’re there! ::cue dramatic music:: Welcome to Premiere’s Media Cache!

screenlight.tv

then restart Time Machine first backup.

I found out that Time Machine was copying thousands of tiny cache files thanks to this command that shows the current Time Machine activity:

sudo fs_usage -wf filesys backupd | grep HFS_update

You can configure time machine to exclude certain paths as well. Then you don't have to delete anything.

 

[hybra]

To dramatically improve performance I also excluded the SMB share folder where the sparsebundle is located from the SPOTLIGHT indexing.

 

[awasb]

TimeMachine issues a sync after every single file write to the server, which can cause painfully low performance with many small files. To dramatically improve performance add a fast (speaking of latency) SLOG device, preferably nvme/Optane. An Optane P1600X (118GB) or a Kingston DC1500M Data Center Series Mixed-Use SSD (960GB) e.g.; both offer a reasonable price/performance ratio. Even for SOHO use.

Ever since I added a slog, the backup times with TimeMachine went down from hours to minutes.

 

[anodos]

TimeMachine issues a sync after every single file write to the server, which can cause painfully low performance with many small files. To dramatically improve performance add a fast (speaking of latency) SLOG device, preferably nvme/Optane. An Optane P1600X (118GB) or a Kingston DC1500M Data Center Series Mixed-Use SSD (960GB) e.g.; both offer a reasonable price/performance ratio. Even for SOHO use.

Ever since I added a slog, the backup times with TimeMachine went down from hours to minutes.

Right, that's another thing. We simulate having F_FULLFSYNC by issuing (fsync() / aio_fsync()) for every (pwrite() / aio_write()) on a file handle that has been opened with the corresponding SMB2/3 Apple flag. This triggers zil_commit() on datasets where sync is not disabled. So slog helps here.

I could maybe optimize and avoid fsync calls on datasets where sync = ALWAYS, but that's maybe a future enhancement internally in samba.

 

[driveburner]

Right, that's another thing. We simulate having F_FULLFSYNC by issuing (fsync() / aio_fsync()) for every (pwrite() / aio_write()) on a file handle that has been opened with the corresponding SMB2/3 Apple flag. This triggers zil_commit() on datasets where sync is not disabled. So slog helps here.

I could maybe optimize and avoid fsync calls on datasets where sync = ALWAYS, but that's maybe a future enhancement internally in samba.

I set sync to disabled on the time machine dataset. Since a zfs snapshot is taken after every backup I can always restore to that if I need to after an unclean shutdown.