SMB Performance on MacOS

[TravisT]

In an effort to more concisely capture the problem at hand, I'm beginning a new thread instead of continuing on this one. I'm also posting in general as I'm not sure if this is storage, networking or some other unknown problem.

Problem: Perceived slow transfers over SMB sharing to Mac clients, specifically when using Adobe Lightroom.

Setup: New Mac Mini with hardwired 1GB ethernet connection to a Cisco 3750 gigabit switch to the FreeNAS box in my signature. Pool is a 4-disk RaidZ pool with 4TB disks. There is 3.84TB (38%) of free space remaining in the pool as reported by FreeNAS. I have a 2-interface LAGG setup on the FreeNAS box. The GUI reports 5.1GB RAM utilized for services and 25.4GB used (of 32GB total) for ZFS caching.

Goal: Get the most out of the system and identify bottlenecks. I want to get more performance and am willing to spend more money, but want to make sure it's spent wisely.

I think this could be a mixture of problems, but am having trouble pinpointing the bottlenecks. Based on my limited storage knowledge, everything seems to checkout individually, but something seems to be drastically reducing performance. A "real world" example is an import of about 8K photos into Adobe Lightroom (only adding them to the library without moving image files) took almost all day. Blackmagic Disk Speed Test app is showing about 18MB/s write and read to the network drive as of today. In theory, I should be seeing in the 90-100MB/s rate across a gigabit link while taking network overhead into account.

How/what can I test to help identify my bottleneck(s)?

 

[Samuel Tai]

If Your Lightroom Is Running Slow…It's Probably One Of These 7 Reasons - Lightroom Killer Tips

Well, I guess it should read, “one or MORE of these seven reasons: 1) Your Lightroom Catalog (and preview files) are NOT on your computer (you’ve stored them on an external hard drive). It’s fine for photos to be stored on an external hard drive (in fact, I recommend it), but not your Lightroom...

lightroomkillertips.com

 

[Samuel Tai]

Stop Your Mac From Creating .DS_Store Files on Shared Network Drives

Your Mac creates hidden .DS_Store files whenever it browses to a new directory. These files store handy metadata and layout information, but if you're using a network drive shared with Windows users, these files may not be so hidden. Here's how to stop your Mac from creating .DS_Store files on...

www.techjunkie.com

Also, you should delete any existing .DS_Store files via find /mnt/<zpool name> -name .DS_Store -delete.

 

[Samuel Tai]

Disable Spotlight indexing of network volumes · Enterprise Mac

enterprisemac.bruienne.com

 

[Samuel Tai]

Apple macOS and CIFS Performance Troubleshooting Tips | IT@Cornell

MacOS systems sometimes have issues when accessing SMB (“CIFS”) shares. This article is intended to list issues and potential work-arounds. These issues may vary with different OS versions or applications, so these suggested work-arounds are only a starting point. Slow Performance When Accessing...

it.cornell.edu

 

[TravisT]

If Your Lightroom Is Running Slow…It's Probably One Of These 7 Reasons - Lightroom Killer Tips

Well, I guess it should read, “one or MORE of these seven reasons: 1) Your Lightroom Catalog (and preview files) are NOT on your computer (you’ve stored them on an external hard drive). It’s fine for photos to be stored on an external hard drive (in fact, I recommend it), but not your Lightroom...

lightroomkillertips.com

Thanks for the replies. I'll try to address each recommendation individually, and some I want to confirm before I fire off an answer.

As for these 7 tips:
1. Lightroom catalog and preview files are on local SSD storage.
2. Mac has 32GB of RAM installed
3. See #1
4. Version is 9.2.1
5. Computer is brand new.
6. Catalog has been optimized, and a new catalog was built and tested by just adding photos in-place.
7. Automatically write changes to XMP is turned off.

 

[TravisT]

Stop Your Mac From Creating .DS_Store Files on Shared Network Drives

Your Mac creates hidden .DS_Store files whenever it browses to a new directory. These files store handy metadata and layout information, but if you're using a network drive shared with Windows users, these files may not be so hidden. Here's how to stop your Mac from creating .DS_Store files on...

www.techjunkie.com

Also, you should delete any existing .DS_Store files via find /mnt/<zpool name> -name .DS_Store -delete.

Confirmed that no .DS_Store files exist, and no new ones are being created.

What is the best way to get a generally accepted benchmark of read/write performance to start with?

App? DD from command line? Etc.

 

[Samuel Tai]

How to Fix Slow SMB File Transfers on OS X 10.11.5+ and macOS Sierra

Apple broke SMB file transfer speeds by forcing client signing on by default. Here’s how to fix it.

dpron.com

 

[Samuel Tai]

[SOLVED!] Very slow file transfer on Mac Shares (SMB/AFP) | DebugginLife.com - A blog on Technology and Life

After hours of research trying to fix Slow file transfers over wifi, the fix took us from 10MB/s to 500MB/s by changing one text file. You need to change the file /etc/sysctl.conf.

www.debugginlife.com

 

[TravisT]

How to Fix Slow SMB File Transfers on OS X 10.11.5+ and macOS Sierra

Apple broke SMB file transfer speeds by forcing client signing on by default. Here’s how to fix it.

dpron.com

On Catalina, here's my output:

Code:

                              SMB_NEGOTIATE                 SMBV_NEG_SMB1_ENABLED
                              SMB_NEGOTIATE                 SMBV_NEG_SMB2_ENABLED
                              SMB_NEGOTIATE                 SMBV_NEG_SMB3_ENABLED
                              SMB_VERSION                   SMB_3.02
                              SMB_SHARE_TYPE                DISK
                              SIGNING_SUPPORTED             TRUE
                              EXTENDED_SECURITY_SUPPORTED   TRUE
                              UNIX_SUPPORT                  TRUE
                              LARGE_FILE_SUPPORTED          TRUE
                              OS_X_SERVER                   TRUE
                              DFS_SUPPORTED                 TRUE
                              FILE_LEASING_SUPPORTED        TRUE
                              MULTI_CREDIT_SUPPORTED        TRUE
                              ENCRYPTION_SUPPORTED          TRUE

There is no "SIGNING_ON" attribute displayed.

I did however, run the command suggested successfully. I have tried this before, but am not sure how to check to see if it is turned off.

[SOLVED!] Very slow file transfer on Mac Shares (SMB/AFP) | DebugginLife.com - A blog on Technology and Life

After hours of research trying to fix Slow file transfers over wifi, the fix took us from 10MB/s to 500MB/s by changing one text file. You need to change the file /etc/sysctl.conf.

www.debugginlife.com

With default setting (delayed_ack=3)

After changing setting to delayed_ack=0:

 

[HoneyBadger]

Try disabling SMB strict sync; in the SMB auxiliary settings (Services -> SMB -> Little Wrench Icon) set strict sync = no

You likely need to restart the service to see the effects.

 

[TravisT]

Ok, getting somewhere.

although I already had that in six parameters on each of my shares, I had one other entry that was supposed to increase the size of the recycle bin on that share. I removed that and restarted SMB. My speeds returned to what appear to be normal (~100MB/s read and write).

Those setting have been there for a while now (and likely through service restarts or even server restarts), so I’m skeptical, but progress is progress!

 

[Samuel Tai]

Glad you found the supposed culprit. My bag of known MacOS SMB tricks was getting empty.

 

[TravisT]

I was getting worried, because it seemed I had tried everything (including that). Fingers crossed that it's fixed now!

 

[anodos]

Ok, getting somewhere.

although I already had that in six parameters on each of my shares, I had one other entry that was supposed to increase the size of the recycle bin on that share. I removed that and restarted SMB. My speeds returned to what appear to be normal (~100MB/s read and write).

Those setting have been there for a while now (and likely through service restarts or even server restarts), so I’m skeptical, but progress is progress!

What is output of testparm -s? Recycle bin with cross_rename basically trigger a 'cp' on every unlink from a child dataset. If client behavior is particularly obnoxious, this can have a performance impact. I've considered rewriting vfs_recycle to be dataset aware (recycle bin in each dataset), but need definitive confirmation that it's an issue before I actually commit time to it.

 

[TravisT]

FreeNAS Output:

Code:

root@sannas[~]# testparm -s
Load smb config files from /usr/local/etc/smb4.conf
Loaded services file OK.
Server role: ROLE_STANDALONE

# Global parameters
[global]
        aio max threads = 2
        bind interfaces only = Yes
        disable spoolss = Yes
        dns proxy = No
        enable web service discovery = Yes
        kernel change notify = No
        load printers = No
        logging = file
        map to guest = Bad User
        max log size = 51200
        nsupdate command = /usr/local/bin/samba-nsupdate -g
        server min protocol = SMB2_02
        server role = standalone server
        server string = FreeNAS Server
        unix extensions = No
        idmap config *: range = 90000001-100000000
        fruit:nfs_aces = No
        idmap config * : backend = tdb
        allocation roundup size = 0
        directory name cache size = 0
        dos filemode = Yes
        include = /usr/local/etc/smb4_share.conf


[AV Library]
        access based share enum = Yes
        aio write size = 0
        ea support = No
        mangled names = illegal
        path = /mnt/globemaster/AV Library
        read only = No
        strict sync = No
        vfs objects = catia fruit streams_xattr shadow_copy_zfs ixnas
        nfs4:acedup = merge
        nfs4:chown = true
        fruit:resource = stream
        fruit:metadata = stream


[Databases]
        access based share enum = Yes
        aio write size = 0
        ea support = No
        mangled names = illegal
        path = /mnt/globemaster/Databases
        read only = No
        strict sync = No
        vfs objects = catia fruit streams_xattr shadow_copy_zfs ixnas crossrename recycle
        nfs4:acedup = merge
        nfs4:chown = true
        recycle:subdir_mode = 0700
        recycle:directory_mode = 0777
        recycle:touch = yes
        recycle:keepversions = yes
        recycle:keeptree = yes
        recycle:repository = .recycle/%U
        fruit:resource = stream
        fruit:metadata = stream


[Files]
        aio write size = 0
        ea support = No
        guest ok = Yes
        guest only = Yes
        mangled names = illegal
        path = /mnt/globemaster/Files
        read only = No
        strict sync = No
        vfs objects = catia fruit streams_xattr shadow_copy_zfs ixnas crossrename recycle
        nfs4:acedup = merge
        nfs4:chown = true
        recycle:subdir_mode = 0700
        recycle:directory_mode = 0777
        recycle:touch = yes
        recycle:keepversions = yes
        recycle:keeptree = yes
        recycle:repository = .recycle/%U
        fruit:resource = stream
        fruit:metadata = stream


[MacDaddy_TM]
        aio write size = 0
        browseable = No
        ea support = No
        mangled names = illegal
        path = /mnt/globemaster/TimeMachine/MacDaddy_TM
        read only = No
        strict sync = No
        vfs objects = catia fruit streams_xattr ixnas
        nfs4:acedup = merge
        nfs4:chown = true
        fruit:volume_uuid = 782b0b2a-c0c2-4cd6-9392-e1a2cbd003e6
        fruit:time machine = yes
        fruit:resource = stream
        fruit:metadata = stream


[MacShare]
        access based share enum = Yes
        aio write size = 0
        ea support = No
        mangled names = illegal
        path = /mnt/globemaster/MacShare
        read only = No
        strict sync = No
        vfs objects = catia fruit streams_xattr shadow_copy_zfs ixnas crossrename recycle
        nfs4:acedup = merge
        nfs4:chown = true
        recycle:subdir_mode = 0700
        recycle:directory_mode = 0777
        recycle:touch = yes
        recycle:keepversions = yes
        recycle:keeptree = yes
        recycle:repository = .recycle/%U
        fruit:resource = stream
        fruit:metadata = stream


[Media]
        aio write size = 0
        ea support = No
        guest ok = Yes
        guest only = Yes
        mangled names = illegal
        path = /mnt/globemaster/Media
        read only = No
        strict sync = No
        vfs objects = catia fruit streams_xattr shadow_copy_zfs ixnas crossrename recycle
        nfs4:acedup = merge
        nfs4:chown = true
        recycle:subdir_mode = 0700
        recycle:directory_mode = 0777
        recycle:touch = yes
        recycle:keepversions = yes
        recycle:keeptree = yes
        recycle:repository = .recycle/%U
        fruit:resource = stream
        fruit:metadata = stream


[Reference]
        aio write size = 0
        ea support = No
        guest ok = Yes
        guest only = Yes
        mangled names = illegal
        path = /mnt/globemaster/Reference
        read only = No
        strict sync = No
        vfs objects = catia fruit streams_xattr shadow_copy_zfs ixnas crossrename recycle
        nfs4:acedup = merge
        nfs4:chown = true
        recycle:subdir_mode = 0700
        recycle:directory_mode = 0777
        recycle:touch = yes
        recycle:keepversions = yes
        recycle:keeptree = yes
        recycle:repository = .recycle/%U
        fruit:resource = stream
        fruit:metadata = stream


[Software]
        aio write size = 0
        ea support = No
        guest ok = Yes
        guest only = Yes
        mangled names = illegal
        path = /mnt/globemaster/Software
        read only = No
        vfs objects = catia fruit streams_xattr shadow_copy_zfs ixnas crossrename recycle
        nfs4:acedup = merge
        nfs4:chown = true
        recycle:subdir_mode = 0700
        recycle:directory_mode = 0777

 

[TravisT]

I'm still having weird issues in Lightroom, but have some work to do before I can confirm it's FreeNAS related or Lightroom related. I am curious if you see anything out of place in the above output. It would not surprise me if there is configuration on FreeNAS that is incorrect or could be tweaked.

What is output of testparm -s? Recycle bin with cross_rename basically trigger a 'cp' on every unlink from a child dataset. If client behavior is particularly obnoxious, this can have a performance impact. I've considered rewriting vfs_recycle to be dataset aware (recycle bin in each dataset), but need definitive confirmation that it's an issue before I actually commit time to it.

If I understand correctly, it seems that enabling the recycle bin may cause problems. I'll be honest, the recycle bin does not seem to work correctly on any of my datasets, so if disabling that will be beneficial, I'm more than willing to try it out.

 

[TravisT]

Ok, so this problem keeps coming back for some reason. It seems that restarting the SMB service remedies the problem temporarily. As a side note, I did turn autotune on at one point in trying to resolve these problems, but I didn't remove the parameters left in place by it.

Any ideas?

Below are the in-place tunables:

 

[ObiTobi]

I'm not sure if I understand all, but which problems do you have yet? I think your performance problems are gone after you have set "signing to off".
What are your problems with LR?

 

[TravisT]

The performance problems appeared to be gone after setting "strict sync = no", however I did have that set on each dataset and the dataset in question already. I also restarted the SMB service after doing this. The easiest measurable feedback I could get was by using BlackMagic Disk Speed Test on the dataset in question. Just performed a clean run with the below results:

After resetting SMB service on FreeNAS and re-running test:

The only change is restarting the SMB service. This could have been a fluke in the original results and thinking the "strict sync = no" was the fix acton. As an aside, the Lightroom issues were the most noticeable at the time, as I was doing a lot of work within Lightroom. When transfer speeds are showing "normal", my Lightroom performance is what I would expect. When I notice slow Lightroom performance, I'm also seeing slow performance on the BlackMagic results, so I'm using that as the baseline tool at this point, even though it may not be the best tool.