Deleting a bulk of files taking for ever over OSX 10.14.6 SMB

[John45622]

Hi,

when I delete a bunch of files from a folder on my SMB share it take about a second or two per file to delete. Is this normal? Tried to delete 200 small files and it took 7 Minutes. I'm on U4 on my TN.

Thanks!

John

 

[anodos]

No. It should't take that long, but probably much depends on configuration details.

 

[c77dk]

Please list the system specifications, and how TrueNAS has been setup. It might give a clue as where to look.

 

[John45622]

Thank you for chiming in! The share is SMB with these aux parameters:

These are the settings:

 

[c77dk]

What's the hardware used? And how are the pool+dataset setup?

 

[anodos]

Try commenting out the auxiliary parameter as well. Might be the mixture of fruit, catia, etc triggered a migration to using "apple-style character encoding". If you didn't explicitly set that, then unset it.

 

[John45622]

What's the hardware used? And how are the pool+dataset setup?

Intel(R) Core(TM) i3-9100F CPU @ 3.60GHz
32GB RAM
8 slot Supermicro Server (have to look up the exact model if you need it)
Uplink to switch is 4x1Gb LAG

Dataset on an encrypted (migrated legacy encrypted from FreeNAS) Z2 pool which works absolutely flawless and fast otherwise. Just deleting is extremely slow via OSX Mojave

Thanks!

 

[John45622]

Try commenting out the auxiliary parameter as well. Might be the mixture of fruit, catia, etc triggered a migration to using "apple-style character encoding". If you didn't explicitly set that, then unset it.

OK, great thank you! Will try that. Do I not need that? All client machines are Mac OSX machines. The warning presented when trying to unset it reads kinda scary (mangled characters in file names etc.)

J.

 

[John45622]

So I tired removing the aux parameter but it still takes 2 seconds per file to delete. This is when deleting 300 files from a folder containing around 4000 files. When I copy those 300 files to a new folder and delete those there is goes very fast. So I suspect deleting from folders that contain a lot of items gets extremely slow?

 

[anodos]

So I tired removing the aux parameter but it still takes 2 seconds per file to delete. This is when deleting 300 files from a folder containing around 4000 files. When I copy those 300 files to a new folder and delete those there is goes very fast. So I suspect deleting from folders that contain a lot of items gets extremely slow?

Depends on how dataset was configured when it was created. Try creating a new dataset with the "SMB" share type, copy 4000 files to it, then delete.

 

[John45622]

Sorry, no change.

 

[John45622]

I also noticed that CPU usage goes up to 40-45% during the deletion process. Never really seen it that high.

 

[anodos]

What's output of testparm -s? With some settings, listdir ends up being O(N^2) or worse based on items in dir. MacOS clients tend to perform a listdir after each file deletion.

 

[John45622]

This is the output for that share:

 

[anodos]

That's probably incomplete output. If you enable SSH and use putty your should be able to copy full output from it.

 

[John45622]

OK thanks, will investigate when I get a moment.

 

[John45622]

Here's the what's output (same as TN GUI shell, I just added the global parameters)

under Global parameters:

aio max threads = 2
bind interfaces only = Yes
disable spoolss = Yes
dns proxy = No
enable web service discovery = Yes
interfaces = 127.0.0.1 10.10.20.200
kernel change notify = No
load printers = No
logging = file
max log size = 5120
nsupdate command = /usr/local/bin/samba-nsupdate -g
registry shares = Yes
restrict anonymous = 2
server role = standalone server
server string = TrueNAS server
unix extensions = No
idmap config *: range = 90000001-100000000
fruit:nfs_aces = No
idmap config * : backend = tdb
directory name cache size = 0
dos filemode = Yes
strict sync = No

and the test-share:

[TEST]
comment = test
ea support = No
kernel share modes = No
path = /mnt/Yoda/TEST
posix locking = No
read only = No
vfs objects = fruit streams_xattr shadow_copy_zfs ixnas aio_fbsd
fruit:resource = stream
fruit:metadata = stream
nfs4:chown = true

and the share that I actually want to use:

access based share enum = Yes
ea support = No
mangled names = no
path = /mnt/Yoda/YYY
read only = No
vfs objects = catia fruit streams_xattr shadow_copy_zfs ixnas aio_fbsd
fruit:resource = stream
fruit:metadata = stream
fruit:encoding = native
nfs4:chown = true

 

[anodos]

If this is a test environment, try the following:
net conf setparm TEST "vfs objects" "streams_xattr shadow_copy_zfs ixnas aio_fbsd"
Then unmount all SMB shares from MacOS side.
Then service samba_server onerestart
Then in Finder "Go->Connect to Server" and input "smb://<ip of server>/TEST"
Then verify in smbutil statshares -a output in MacOS that Server connection isn't an OSX server.
Then repeat test on your TEST share.

 

[John45622]

If this is a test environment, try the following:
net conf setparm TEST "vfs objects" "streams_xattr shadow_copy_zfs ixnas aio_fbsd"
Then unmount all SMB shares from MacOS side.
Then service samba_server onerestart
Then in Finder "Go->Connect to Server" and input "smb://<ip of server>/TEST"
Then verify in smbutil statshares -a output in MacOS that Server connection isn't an OSX server.
Then repeat test on your TEST share.

After doing this OSX showed the red "no access" logo on the folder containing the TEST share as if I had lost access to them.
statsshares under OSX now shows:
OS_X_SERVER VALUE TRUE:

Code:

 SMB_NEGOTIATE                 SMBV_NEG_SMB1_ENABLED
                              SMB_NEGOTIATE                 SMBV_NEG_SMB2_ENABLED
                              SMB_NEGOTIATE                 SMBV_NEG_SMB3_ENABLED
                              SMB_VERSION                   SMB_3.02
                              SMB_SHARE_TYPE                DISK
                              SIGNING_SUPPORTED             TRUE
                              EXTENDED_SECURITY_SUPPORTED   TRUE
                              UNIX_SUPPORT                  TRUE
                              LARGE_FILE_SUPPORTED          TRUE
                              OS_X_SERVER                   TRUE
                              FILE_IDS_SUPPORTED            TRUE
                              DFS_SUPPORTED                 TRUE
                              FILE_LEASING_SUPPORTED        TRUE
                              MULTI_CREDIT_SUPPORTED        TRUE
                              ENCRYPTION_SUPPORTED          TRUE

Under Big Sur (11.3) I get:

Code:

SERVER_NAME                   xxxxxx
                              USER_ID                       502
                              SMB_NEGOTIATE                 SMBV_NEG_SMB1_ENABLED
                              SMB_NEGOTIATE                 SMBV_NEG_SMB2_ENABLED
                              SMB_NEGOTIATE                 SMBV_NEG_SMB3_ENABLED
                              SMB_VERSION                   SMB_3.1.1
                              SMB_SHARE_TYPE                DISK
                              SIGNING_SUPPORTED             TRUE
                              EXTENDED_SECURITY_SUPPORTED   TRUE
                              LARGE_FILE_SUPPORTED          TRUE
                              FILE_IDS_SUPPORTED            TRUE
                              DFS_SUPPORTED                 TRUE
                              FILE_LEASING_SUPPORTED        TRUE
                              MULTI_CREDIT_SUPPORTED        TRUE

Checking file system ACL on TN showed that the share ACL group had gone back to "wheel". So I re-applied the group I had set originally. I restartet my OSX (10.14.6) machine and now the folders can be accessed and sharestats show:

Code:

 SERVER_NAME                   XXX._smb._tcp.local
                              USER_ID                       501
                              SMB_NEGOTIATE                 SMBV_NEG_SMB1_ENABLED
                              SMB_NEGOTIATE                 SMBV_NEG_SMB2_ENABLED
                              SMB_NEGOTIATE                 SMBV_NEG_SMB3_ENABLED
                              SMB_VERSION                   SMB_3.02
                              SMB_SHARE_TYPE                DISK
                              SIGNING_SUPPORTED             TRUE
                              EXTENDED_SECURITY_SUPPORTED   TRUE
                              LARGE_FILE_SUPPORTED          TRUE
                              FILE_IDS_SUPPORTED            TRUE
                              DFS_SUPPORTED                 TRUE
                              FILE_LEASING_SUPPORTED        TRUE
                              MULTI_CREDIT_SUPPORTED        TRUE
                              ENCRYPTION_SUPPORTED          TRUE

I then did the deletion test and it's very slow deleting every file one by one about 1-1.5 seconds per item.

 

[John45622]

I guess I'll have to ask my users to work around this my moving the items to a temp folder first and then delete the folder. This seems to work a million times faster than deleting the selected individual items.
I do think that wasn't a issue on U3 but I'm not 100% sure.