SMB shares dismount from OSX when opening large SMB folders

[John45622]

Hi there,

I think I need some help diagnosing an issue I am having when using a folder synchroniser called Chronosync for OSX. It syncs two folders and when I'm using that on folders containing many items (5000-6000) at some point the SMB mounted volume just disappears and hard-unmounts from the Mac (Mojave).
I am trying to narrow down if TN(U6) is somehow getting "overwhelmed" and decides to somehow crash or SMB locks up or if the cause is on the Mac or that particular software. What I can say is that it happens on every work station in our facility.

Is there a particular log file on TN that I can look at to get some pointers on what's going on when the volume is unmounted and if there's an error in that moment?

Thanks!

 

[anodos]

Can you PM me a debug?

 

[John45622]

Sent.

Thanks!

 

[John45622]

Sorry to report that I'm still seeing this under current TN13 (U2) and Monterey OSX 12.6. Also happens when opening folders containing lots of items. Since I can reproduce this 100% of the time I wonder if there's a log I could look at to see what happens on TN in the moment when the volume dis-mounts. CPU is pretty low on TN when this happens also nothing weird with memory.
I tried to narrow down what the maximum items are that work without the hard dis-mount. 2000 .WAV files in a folder work fine. Takes a bit of time (about 1-2 Minutes) till they are are displayed but the volume stays mounted.
Opening a folder with 2600 or more items makes the volume dismount after a 1-2 Minute wait for the items to list. Same thing happens when going into that folder via terminal and doing a simple "ls" command. System comes back with a "Socket is not connected" message as the volume has then dismounted.

Is this a weakness of TN or OSX?

 

[John45622]

Ok, found something. I can open a folder on a different shared SMB volume containing 17000+ items just fine. This volume is on a raid Z2 pool. The one that shows the strange behaviour is a single drive pool with encryption (non-legacy) (yes, I know but in this case it's intentional). Could this be a limitation of single-drive pools?
I copied that folder to a different 1-drive-pool (GELI encrypted) and again no problem to open at all.
Then I copied the same folder from to the initial drive I can't open it. So it must be something specific to that ONE pool.

 

[John45622]

Couple of days later I can report that what I suspected earlier was pure coincidence. SMB disconnects are really random but ALWAYS and ONLY with folders with a lot of files in them. 2k+ seems to be a margin. It's really random but makes a lot of syncing workflows fail when a mounted volume just pops away randomly. Some days I can open a folder with 10k+ items on other days opening that same folder make SMB disconnect. When that happens not only that specific volume disconnects but the entire server gets disconnected from the Mac.
Any way I can narrow this down? Nothing pops up in the TN13 log when this happens and nothing obvious in the OS X Console either. Happens on every OS X between Mojave and the latest Monterey 12.6 and on different client hardware.

Thanks!

 

[John45622]

Did some more testing now. I setup an NFS share to a new dataset and copied a bunch of very large folders over. When I mount those on OS X via NFS they won't dismount when opened. Even folders with 10k+ items work fine. The same folder on an SMB share can not be opened. So maybe SMB is just rubbish under OS X or TN has some issue? NFS seems to work fine but unfortunately write speed is not on par with SMB (max 60MB/s vs almost full GBit speed with SMB).
I guess now one else is having this problem. We have it on every single Mac from Mojave to Monterey.
Problem is NFS doesn't have the granular permissions that SMB offers (unless I'm missing something) and 60% write speed is a bit lame.

 

[ChrisRJ]

SMB in MacOS is also not the best for me. I have a DMS (document management system) runinng on Linux here that scans a directory, which is shared via Samba. Putting files into there from Mac does not work reliably, it will eventually lock up. I know this doesn't solve your problem, but wanted to share nevertheless.

 

[John45622]

Thanks for chiming in. This is for a media production setup. AVID MediaComposer and ProTools so folders containing thousands or 10k+ .wav or MXF files. SMB mount just hard disconnect randomly with no error message. I can reproduce on 5 connected rigs running flavours of Mojave, Catalina and Monterey. All the same behaviour. Drives me crazy as there's nothing being logged.

 

[ChrisRJ]

Regardless of anything else (SMB/Samba, Mac, file size) it is always a good idea to limit the number of files per directory. I have no recent experience on exact numbers. But I remember that in the early 2000s we split up an inbound directory. Originally up to 30k PDF files were placed into a single directory and it caused performance issues. We then created sub-directories and had about 5k files in each of them. Would something in that direction be an option?

 

[anodos]

I have recently seen finder behave erratically (and sometimes spontaneously disconnect) due to following reasons:
1 presence of file names with decomposed unicode diacritical marks
2 the "fruit:zero_file_id" parameter

(1) can be unpredictable and is IMHO a bug in finder / MacOS SMB client (which forces pre-composed unicode when converting file names to network names). This causes various SMB ops to fail with STATUS_NOT_FOUND.

(2) is an issue in BlueFin RC1 where upstream samba changed default (we're switching back for release). It's also of course an issue where users place auxiliary parameters to enable it. Big Sur and earlier Finder tends to crash / forcibly close, Ventura just behaves somewhat erratically.

This is not exhaustive of course.

 

[John45622]

All my aux parameters are default. (no entries in TN under AUX).
What's also weird that bulk-deleting often takes foreeeeever via SMB. Often it take 1sec/file and deleting a folder with 500 items takes literally 500seconds.
Might have to abandon SMB entirely in favour of NFS but the latter has no user/groups management which is a show-stopper on my network. I need granular permissions.
This is for media production so 5000+ files inside a single folder is totally everyday routine and unavoidable unless you hand-maintain and manually sort media file backup into split folders constantly every day. Not an option unfortunately for the users.

 

[anodos]

All my aux parameters are default. (no entries in TN under AUX).
What's also weird that bulk-deleting often takes foreeeeever via SMB. Often it take 1sec/file and deleting a folder with 500 items takes literally 500seconds.
Might have to abandon SMB entirely in favour of NFS but the latter has no user/groups management which is a show-stopper on my network. I need granular permissions.
This is for media production so 5000+ files inside a single folder is totally everyday routine and unavoidable unless you hand-maintain and manually sort media file backup into split folders constantly every day. Not an option unfortunately for the users.

Was the SMB share created with the SMB dataset preset?

 

[simonj]

I would like to chime in here as we have the same issues: Just lately adding folders with thousands of files (Arri Alexa RAW image sequences) to Davinci Resolve always made the SMB share unmount. In the end the workaround was to switch to AFP temporarily until the media files were added to the project.

We also see extremely long times to delete files and sometimes - but not always - for other Finder operations when eg. listing a directory with many files can lock the server for 20+ minutes.

I started a thread here: https://www.truenas.com/community/t...ost-production-setup-with-mac-clients.106503/

A good idea is to remove all the SMB aux parameters. I think @anodos meant us about the "fruit:zero_file_id" parameter, which was supposed to help but added a bug for moving files with Finder. We left only "strict sync = no" and " case sensitive = yes"

Otherwise we haven't found a solution to make Finder behave better with TrueNAS SMB and I would be extremely glad if some solutions were found in this thread.

 

[anodos]

I would like to chime in here as we have the same issues: Just lately adding folders with thousands of files (Arri Alexa RAW image sequences) to Davinci Resolve always made the SMB share unmount. In the end the workaround was to switch to AFP temporarily until the media files were added to the project.

We also see extremely long times to delete files and sometimes - but not always - for other Finder operations when eg. listing a directory with many files can lock the server for 20+ minutes.

I started a thread here: https://www.truenas.com/community/t...ost-production-setup-with-mac-clients.106503/

A good idea is to remove all the SMB aux parameters. I think @anodos meant us about the "fruit:zero_file_id" parameter, which was supposed to help but added a bug for moving files with Finder. We left only "strict sync = no" and " case sensitive = yes"

Otherwise we haven't found a solution to make Finder behave better with TrueNAS SMB and I would be extremely glad if some solutions were found in this thread.

Did you create the dataset with the SMB preset? If you don't set auxiliary parameters, the case sensitivity settings are auto-detected.

 

[simonj]

Did you create the dataset with the SMB preset? If you don't set auxiliary parameters, the case sensitivity settings are auto-detected.

I'm not 100% sure anymore if datasets were created with SMB preset. Is there a way to check retroactively? Both our main datasets have case sensitivity = insensitive.

Archive Server:
"ec_backup" dataset: SMB preset - on this dataset we had the unmount problem when adding image sequences in Resolve. Directory listing is also slow.

Workhorse Server
"work_mirrored" dataset: probably created with SMB preset. Case Sensitivity = Insensitive - This is the one I made the demo video in other post of getting stuck for 20 minutes on directory listing. We didn't have unmount problems with that yet.

Should "case sensitive = yes" be removed from aux parameters?

 

[anodos]

I'm not 100% sure anymore if it datasets were created with SMB preset. Is there a way to check retroactively? Both our main datasets have case sensitivity = insensitive.

Archive Server:
"ec_backup" dataset: SMB preset - on this dataset we had the unmount problem when adding image sequences in Resolve. Directory listing is also slow.

Workhorse Server
"work_mirrored" dataset: probably created with SMB preset. Case Sensitivity = Insensitive - This is the one I made the demo video in other post of getting stuck for 20 minutes on directory listing. We didn't have unmount problems with that yet.

Should "case sensitive = yes" be removed from aux parameters?

Yes. In general auxiliary parameters should never be needed.

 

[simonj]

Yes. In general auxiliary parameters should never be needed.

Removed everything but "strict sync = no". I still see slow directory listing. Didn't test if the unmount still happens.

Do you have any theory what might cause the Mac (Finder) <> TrueNAS not liking each other?