SMB: Deleting large folders fails from macOS Finder

seanm

Guru
Joined
Jun 11, 2018
Messages
570
I'm using FreeNAS 11.2U2.1 (newest) and macOS 10.13.6. If I have a folder on my SMB share that contains many many files, and then try to delete it from macOS Finder, it'll often fail with the message:

"The operation can’t be completed because the item “MacOS” is in use."

"MacOS" is indeed the name of a folder inside the folder I'm trying to delete. Only one user is connected to the share (from one computer).

Someone else is seeing similar: https://www.ixsystems.com/community...11-1-u6-11-2-release.71746/page-2#post-519631

I've tried increasing the log level, but I'm not sure it's working, because every time I come back to the screen, it no longer shows the log level I previously selected. Is that a bug in itself?

The last few lines of /var/log/samba4/log.smbd are:

Code:
[2019/03/18 19:49:01.725229,  3] ../source3/smbd/dir.c:662(dptr_create)
  creating new dirptr 0 for path big/ooeuoeuo/build_results_rev47168/Debug/Brainsight-2.3.x/NIRSTestApp.app, expect_close = 0
[2019/03/18 19:49:01.725428,  3] ../source3/smbd/dir.c:1225(smbd_dirptr_get_entry)
  smbd_dirptr_get_entry mask=[*] found big/ooeuoeuo/build_results_rev47168/Debug/Brainsight-2.3.x/NIRSTestApp.app/. fname=. (.)
[2019/03/18 19:49:01.725855,  3] ../source3/smbd/dir.c:1225(smbd_dirptr_get_entry)
  smbd_dirptr_get_entry mask=[*] found big/ooeuoeuo/build_results_rev47168/Debug/Brainsight-2.3.x/NIRSTestApp.app/.. fname=.. (..)
[2019/03/18 19:49:01.726350,  3] ../source3/smbd/dir.c:1225(smbd_dirptr_get_entry)
  smbd_dirptr_get_entry mask=[*] found big/ooeuoeuo/build_results_rev47168/Debug/Brainsight-2.3.x/NIRSTestApp.app/Contents fname=Contents (Contents)
[2019/03/18 19:49:01.727536,  3] ../source3/smbd/smb2_server.c:3195(smbd_smb2_request_error_ex)
  smbd_smb2_request_error_ex: smbd_smb2_request_error_ex: idx[1] status[STATUS_NO_MORE_FILES] || at ../source3/smbd/smb2_query_directory.c:158
[2019/03/18 19:49:01.730469,  3] ../source3/smbd/dir.c:662(dptr_create)
  creating new dirptr 0 for path big/ooeuoeuo/build_results_rev47168/Debug/Brainsight-2.3.x/NIRSTestApp.app/Contents, expect_close = 0
[2019/03/18 19:49:01.730688,  3] ../source3/smbd/dir.c:1225(smbd_dirptr_get_entry)
  smbd_dirptr_get_entry mask=[*] found big/ooeuoeuo/build_results_rev47168/Debug/Brainsight-2.3.x/NIRSTestApp.app/Contents/. fname=. (.)
[2019/03/18 19:49:01.731219,  3] ../source3/smbd/dir.c:1225(smbd_dirptr_get_entry)
  smbd_dirptr_get_entry mask=[*] found big/ooeuoeuo/build_results_rev47168/Debug/Brainsight-2.3.x/NIRSTestApp.app/Contents/.. fname=.. (..)
[2019/03/18 19:49:01.731737,  3] ../source3/smbd/dir.c:1225(smbd_dirptr_get_entry)
  smbd_dirptr_get_entry mask=[*] found big/ooeuoeuo/build_results_rev47168/Debug/Brainsight-2.3.x/NIRSTestApp.app/Contents/MacOS fname=MacOS (MacOS)
[2019/03/18 19:49:01.732276,  3] ../source3/smbd/dir.c:1225(smbd_dirptr_get_entry)
  smbd_dirptr_get_entry mask=[*] found big/ooeuoeuo/build_results_rev47168/Debug/Brainsight-2.3.x/NIRSTestApp.app/Contents/Resources fname=Resources (Resources)
[2019/03/18 19:49:01.732770,  3] ../source3/smbd/dir.c:1225(smbd_dirptr_get_entry)
  smbd_dirptr_get_entry mask=[*] found big/ooeuoeuo/build_results_rev47168/Debug/Brainsight-2.3.x/NIRSTestApp.app/Contents/PkgInfo fname=PkgInfo (PkgInfo)
[2019/03/18 19:49:01.733302,  3] ../source3/smbd/dir.c:1225(smbd_dirptr_get_entry)
  smbd_dirptr_get_entry mask=[*] found big/ooeuoeuo/build_results_rev47168/Debug/Brainsight-2.3.x/NIRSTestApp.app/Contents/Frameworks fname=Frameworks (Frameworks)
[2019/03/18 19:49:01.733834,  3] ../source3/smbd/dir.c:1225(smbd_dirptr_get_entry)
  smbd_dirptr_get_entry mask=[*] found big/ooeuoeuo/build_results_rev47168/Debug/Brainsight-2.3.x/NIRSTestApp.app/Contents/_CodeSignature fname=_CodeSignature (_CodeSignature)
[2019/03/18 19:49:01.735574,  3] ../source3/smbd/smb2_server.c:3195(smbd_smb2_request_error_ex)
  smbd_smb2_request_error_ex: smbd_smb2_request_error_ex: idx[1] status[STATUS_NO_MORE_FILES] || at ../source3/smbd/smb2_query_directory.c:158
[2019/03/18 19:49:01.738707,  2] ../source3/smbd/open.c:1447(open_file)
  builder opened file big/ooeuoeuo/build_results_rev47168/Debug/Brainsight-2.3.x/NIRSTestApp.app/Contents/PkgInfo read=Yes write=No (numopen=4)
[2019/03/18 19:49:01.740309,  3] ../source3/smbd/smb2_read.c:421(smb2_read_complete)
  smbd_smb2_read: fnum 2721744304, file big/ooeuoeuo/build_results_rev47168/Debug/Brainsight-2.3.x/NIRSTestApp.app/Contents/PkgInfo, length=8 offset=0 read=8
[2019/03/18 19:49:01.741313,  2] ../source3/smbd/close.c:802(close_normal_file)
  builder closed file big/ooeuoeuo/build_results_rev47168/Debug/Brainsight-2.3.x/NIRSTestApp.app/Contents/PkgInfo (numopen=3) NT_STATUS_OK
[2019/03/18 19:49:01.776684,  3] ../source3/smbd/dir.c:662(dptr_create)
  creating new dirptr 0 for path big/ooeuoeuo/build_results_rev47168/Debug/Brainsight-2.3.x/NIRSTestApp.app/Contents/MacOS, expect_close = 0
[2019/03/18 19:49:01.776817,  3] ../source3/smbd/smb2_server.c:3195(smbd_smb2_request_error_ex)
  smbd_smb2_request_error_ex: smbd_smb2_request_error_ex: idx[5] status[NT_STATUS_NO_SUCH_FILE] || at ../source3/smbd/smb2_query_directory.c:158
[2019/03/18 19:49:04.952019,  3] ../source3/smbd/trans2.c:3465(smbd_do_qfsinfo)
  smbd_do_qfsinfo: level = 1003
[2019/03/18 19:49:04.952740,  3] ../source3/lib/sysquotas.c:488(sys_get_quota)
  sys_get_vfs_quota() failed for mntpath[.] bdev[(null)] qtype[1] id[-1]: Operation not supported
[2019/03/18 19:49:04.952774,  3] ../source3/lib/sysquotas.c:488(sys_get_quota)
  sys_get_vfs_quota() failed for mntpath[.] bdev[(null)] qtype[3] id[-1]: Operation not supported


Anyone else having this problem? Do the logs above give any hint?
 

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
9,543
Try turning off Samba's AIO for the share.
Code:
aio write size = 0
aio read size = 0

This shouldn't affect unlink operations, but if other operations are already taxing VM, then this might help. Otherwise, I have some optimizations to the unlink code in samba that you can test.
 

seanm

Guru
Joined
Jun 11, 2018
Messages
570
That didn't help I'm afraid.

This machine isn't taxed at all. It's just-purchased, and going through testing before put into production. Deleting this big folder is literally the only thing the box is doing now.

I can test anything you've got! debug builds, bleeding edge, etc.
 

JaimieV

Guru
Joined
Oct 12, 2012
Messages
742
I was seeing this too. FreeNAS 11.2U2.1, macOS 10.14.4beta and 10.14.3. Switching off AIO for the share does appear to have fixed it for me, make of that what you will. I had previously added fruit,catia to the options as per other recommendations.
 

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
9,543
I still haven't had a chance to look at this (I'm working on rewriting the Active Directory service). Today I'm planning to port Samba 4.9.5 and 4.10.0. (FreeNAS 11.3 will be based on 4.10). While MacOS has a particularly inefficient method of deleting large numbers of files over SMB. This potentially exacerbates situations where the VM subsystem is already taxed by samba's AIO. While I'm working on the new builds, I'll shoot Sean a modified version of samba that might alleviate some of the resource contention / locking issues.
 

seanm

Guru
Joined
Jun 11, 2018
Messages
570
While MacOS has a particularly inefficient method of deleting large numbers of files over SMB.

Do you file bugs with Apple about these things? I could if you'd help me describe the problem to them. Worst case I could time deleting identical large folders from Win10 vs macOS and let them figure it out, but I guess a clue would always help them.

Looking forward to your test samba.
 

overshoot

Explorer
Joined
Jul 16, 2019
Messages
80
Hi,

Sorry for re-opening an old thread but my client running on an iMac Pro with Mojave 10.13.6 is having the same issue with the Finder locking up some dng files. We are using FreeNAS 11.2 U5 on an HP Proliant Gen8 Server with a 10GB Ethernet interface so he's getting a 10GB link to the server with a direct cable. We are achieving 500MB/s write and 250MB/s read average through the SMB shares.
Previously I was getting extremely poor write performances (15MB/s) but then added "strict sync = no" and got to 500MB/s.

However we can't remove some folders as it will always say that an app is using some of the files. Using lsof command, only the finder is. So relaunching the Finder and then removing the folder with the rm command through the terminal works. Obviously this is not an option for my client...

I've tried the aio read/write options from @anodos but it will completely freeze the finder or even terminal to the point I have to unmount the volume using diskutil unmount through the terminal. Basically the volume access drops and I have to eject the drive and remount it.

I've added some SMB parameters to the auxiliary option so please let me know if some could mess up with file locking:

min protocol = SMB2
ea support = yes
vfs objects = catia fruit streams_xattr
fruit:aapl = yes
readdir_attr:aapl_rsize = yes
readdir_attr:aapl_finder_info = yes
readdir_attr:aapl_max_access = yes
fruit:nfs_aces = yes
fruit:copyfile= yes
fruit:metadata = netatalk
fruit:resource = file
fruit:locking = none
fruit:encoding = private
unix extensions = yes
fruit:model = MacSamba
spotlight = no
smb2 max read = 8388608
smb2 max write = 8388608
smb2 max trans = 8388608
smb2 leases = yes
aio read size = 1
aio write size = 1
kernel oplocks = no
use sendfile = yes
strict sync = yes
sync always = no
delete veto files = true
fruit:posix_rename = yes
fruit:veto_appledouble = yes
fruit:zero_file_id = yes
fruit:wipe_intentionally_left_blank_rfork = yes
fruit:delete_empty_adfiles = yes

Thanks!
 

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
9,543
Try the following
1) create new dataset (case insensitive)
2) create new SMB share
3) remove [zfs_space, zfsacl] from share VFS object, and add [ixnas]
4) add share auxiliary parameter "case sensitive = true"
5) remove [fruit] from all existing shares
6) restart samba
7) repeat test on new share.
 

overshoot

Explorer
Joined
Jul 16, 2019
Messages
80
Thanks Anodos, I'll try these next week and report back. I do appreciate your quick reply.
 

overshoot

Explorer
Joined
Jul 16, 2019
Messages
80
Try the following
1) create new dataset (case insensitive)
2) create new SMB share
3) remove [zfs_space, zfsacl] from share VFS object, and add [ixnas]
4) add share auxiliary parameter "case sensitive = true"
5) remove [fruit] from all existing shares
6) restart samba
7) repeat test on new share.

Sorry... by reading you I am getting a bit confused so please clarify:

You're saying to create a new dataset+share and test... are you saying that I have to move one of the folder that keeps being locked by the finder and try to remove it from that new share and see if it fixes the issue?

Also, you're saying to remove the "Fruit" vfs object from all existing shares... is that supposed to possibly fix the issue as well so should I try that first?

Or does all of that in order fixes the issue?
Thanks!
 
Last edited:

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
9,543
Sorry. I misread the post. There's a separate issue that causes deletes to be extremely slow (Mac clients rescan directories after each delete causing performance issues).
 

JaimieV

Guru
Joined
Oct 12, 2012
Messages
742
@overshoot You may be short of luck on this one - the Finder (or I think more accurately Quicklook running under Finder) regularly goes off on a traversal spree when you hit a new location, whether hosted by a FreeNAS or local or whatever. If you're seeing lsof from the Mac side show Finder having files open, that's a purely local hold and nothing to do with the FreeNAS system. You can see the same thing on locally-attached or internal Mac disks sometimes too.

It's immensely annoying and I've been filing Apple bugs about it for several years now. On the plus side they did fix Spotlight indexing doing similar a while back, so it's possible it'll be solved sometime...

On the other hand if you're seeing "The operation can’t be completed because the item “.smbdelete(randomstring)” is in use." then I think that is a FreeNAS side issue, and if I'm right on that then I know the team are trying to nail it down from prior posts here. In this case lsof does not show the .smbdelete file as being open, but it is visible with ls -la. If you try to delete it with rm you'll get "resource busy".
 

overshoot

Explorer
Joined
Jul 16, 2019
Messages
80
@JaimieV Thanks for your feedback

So there is no workaround or tweak that we can apply either on the FreeNAS side or Client side to mitigate the issue from your experience?
 

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
9,543
@JaimieV Thanks for your feedback

So there is no workaround or tweak that we can apply either on the FreeNAS side or Client side to mitigate the issue from your experience?
Try setting smb2 leases = no as an auxiliary parameter under Services->SMB. In the past I've seen situations before where MacOS will set leases on resource forks and subsequently not release them. You can check for this using smbstatus. Also avoid using Column view in finder.
 

JaimieV

Guru
Joined
Oct 12, 2012
Messages
742
@anodos Do you have a pointer to the currently recommended way to set up SMB shares for Mac users? I set mine up following suggestions from iX staff here back in June or so, but I know there's been a bunch of work on this lately.

Mine are mostly working but occasionally I'll get the .smbdelete(randomstring) stuck problem during a multilevel delete, and often Finder won't see a change to the filesystem made by another client (eg Transmission completes a download and moves it to the completed folder, Finder's contents don't change even if I go elsewhere then come back - an excess of caching?).

I'm in Catalina (but Mojave saw the same)
VFS Objects are: catia, fruit, streams_xattr, zfs_space, zfsacl.
Auxiliary parms are aio write size = 0, aio read size = 0

I do use columns view mostly... not noticed that warned about before!
 

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
9,543
@JaimieV I have that link taken from the Samba wiki but not sure if applying them all works well in FreeNAS: https://wiki.samba.org/index.php/Configure_Samba_to_Work_Better_with_Mac_OS_X

Though I would also love to hear from the FreeNAS community for which recommended settings to apply for macOS clients only or a mixed environment.
Thanks!
It's generally better to understand the parameters rather than copying-pasting from a wiki. Specifically, removing zfsacl is a bad idea. Some of those items on the list are defaults for example:
Code:
smb2 max read = 8388608
smb2 max write = 8388608
smb2 max trans = 8388608

You don't want to use this one:
Code:
use sendfile = yes

You can find the defaults for parameters in the relevant manpages.
 

overshoot

Explorer
Joined
Jul 16, 2019
Messages
80
I did some more testing...

Connecting the same volume using AFP does the same thing. Finder still complains about some used files.
Doing an lsof on the filename gives no result as if the file was not in use but simply used when the finder is browsing the folder to list its content in order to delete it... like a snake biting its own tail. Good job Apple!

If I manually remove one file using "rm" in terminal from the Mac and then try again removing the main root folder, finder will still find another folder used.
If I manually go inside that folder and remove the files in it, I will then be able to remove that folder. It is doable from the Finder but really time consuming.

Finally I've tried from a different Mac and same thing is also happening.

Both Macs are running 10.14.6 which the latest updated version of Mojave.

Shall I try to change the dataset options or permissions, for example the share type from Windows to Mac?
 

overshoot

Explorer
Joined
Jul 16, 2019
Messages
80
I've tried and I am still getting the same issue.

I can't believe there is not more users having this same problem pushing Apple to do something about it.
They are not supporting macOS Server anymore and leave us with the SMB protocol has the main protocol to connect to a Server and still you are getting these annoying behavior. So tired of this!

Please tell me there is a workaround so we are not forced to buy this Acronis Files Connect?!
 
Top