Finder-browsing gets SMB stuck at 100% cpu - in otherwise almost perfect film post-production setup with Mac clients

simonj

Dabbler
Joined
Feb 28, 2022
Messages
32
Happy new year everyone

We are a small post-production house and switched to TrueNAS around 6 months ago.

Everything runs well but we could not get rid of an annoying issue when browsing a folder containing a large number of files stalls both the client and the server. While the stall happens there is one smbd process on the server stuck at 100%. This can last up to hours. Most of the time we have to kill Finder or the application accessing files. Image sequences are an essential part of our daily business.

See a short demo how the server gets stuck for more than 20 minutes after accessing a directory with 16K files:

It becomes most annoying when after rendering an image sequence in Davinci Resolve, pressing the <Browse> button stalls Resolve and it's either force-quit and potentially losing some work or waiting for an undefined time.

We have two fairly powerful servers connected to Mac workstations over 10gbe.

I'll concentrate on the <Workhorse> server which holds recent projects. Hardware is mostly the same on both apart from disk configurations. The issue is on both servers.

TrueNAS-13.0-U3.1
Supermicro SC-848 24bay with X9QRI-F mainboard
4 x Intel Xeon E5-4650, 8x 2.70GHz - total 32 cores
512GB ECC RAM

Pool Layout:
2 x RAIDZ2 vdevs containing of 8 x 10 TB Seagate Ironwolf Harddisks each (40% full)
2 x 2 TB NVME SSD L2ARC
2 x 2 TB NVME SSD mirrored special vdev

After a lot of research here on the forums and much back-and-forth we came to a less-is-more approach with tunables and additional settings.

Just lately @anodos kindly helped me to get rid of a bad configuration that was supposed to help with our problem but led to strange behavior with moving files. https://www.truenas.com/community/t...e-files-with-apple-clients.105109/post-731230

Tunables:
1672596686009.png

SMB settings:
1672597107723.png

Share settings:
1672596836885.png


Additional details:
  • After the folder contents have been displayed once, it doesn't stall anymore on the same folder for some time.
  • Everything else on the server becomes painfully slow while SMB is stuck. Didn't test if it also affects connections from other clients
  • It happens most of the time (I'd say 90%) but not all the time
  • It happens with both our main Mac clients (Mac Pro2019 / Mac Studio)
  • A short test from a Linux workstation did not show the problem
  • It does not happen when accessing direct attached storage on a Mac from another Mac over SMB
  • We did not see that problem on our previous QNAP server
  • listing files via <ls> on Terminal doesn't show the problem
  • .DS_Stores and Thumbnail creation are disabled on the Mac
All this points in the direction that there is a SMB-communications issue between Mac clients (specifically Finder) and TrueNAS that seems to be unique to TrueNAS - which very hopefully could be solved through some configuration change.

Otherwise the speeds are really good and close to 10G line level for single files and acceptable for image sequences.
See some systematic tests here (Fatboy is a direct attached TB3 RAID on the Mac Studio):
1672598128741.png


It would be great to get some advice on this. The servers are great but this bug has raised so much annoyance in our team that we are considering to move away from TrueNAS.

Thanks everyone.
 
Last edited:
Top