TrueNAS smb issues with Adobe Lightroom

[wdp]

I've been in the process of setting up a new TrueNAS server at home and noticed that I'm getting extremely poor performance when I attempt to ingest RAW photography files into Adobe Lightroom on macOS. Lightroom then fumbles as it crawls through folders it's trying to link to on the networked storage.

I spent some time troubleshooting and just trying to make sense of it, but so far I haven't had any luck. I get about 800 MB/s read/write in tests. If folders are small, they're okay, but over 1000 images the scaffold takes 30+ s to load. That is not normal behavior at all.

First thing I did was disable sync in the gui, that didn't seem to have any impact though.

I have a synology on the same 10gbe network, over SMB everything is near instant in Lightroom with it.

So my thoughts are I either configured my pools wrong? or I botched permissions? or truenas and macOS just suck at talking?

I don't know how to troubleshoot beyond wild uneducated guesses at this point. My knowledge of the SMB protocol and tuning is very limited.

Any ideas what I should swing at next? Thanks.

 

[wdp]

OpenZFS on OS X • View topic - SMB FullSync missing för zfs volumes

Possibly related?

 

[sretalla]

I get about 800 MB/s read/write in tests.

What tests?

So my thoughts are I either configured my pools wrong? or I botched permissions? or truenas and macOS just suck at talking?

Well... I have seen folks saying that SMB can be asked by macOS to do sync writes, so if your process is writing, then maybe slowness would come from that (I see you have no SLOG).

Not sure which of your NAS you're talking about, but I guess the 4x18TB mirrors?

 

[c77dk]

We need to know what hardware you're using (I see 2 servers in your sig), and how it's setup.

The mention of larger number of files in directories, suggests metadata could be an issue - here more RAM / a metadata only l2arc might be helpful

 

[wdp]

This is on a TrueNAS Mini XL+
Atom C3758 @ 2.20GHz
128GB 2400 ECC
Currently configured to a 8x18TB RaidZ2

Client is macOS 13.1, M1 Max, 64Gb, Thunderbolt to PCIE Atto NT-12 10GbE RJ45

I'm trying to bring raw photography files from an SMB share on the TrueNAS into Adobe Lightroom.

Import, and opening folders/scaffolds on imports in Lightroom take 15-30 seconds for 1600 image files, which have XMP sidecars, so 3200 files probably get scanned. Most of the time I never make it to the thumbnails loading and the import basically locks up.

Code:

zpool list
NAME        SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
boot-pool   216G  1.52G   214G        -         -     0%     0%  1.00x    ONLINE  -
hopper     3.62T   624K  3.62T        -         -     0%     0%  1.00x    ONLINE  /mnt
shelf       131T  1.01T   130T        -         -     0%     0%  1.00x    ONLINE  /mnt

Local storage and the SMB mount on the synology (smb, ex4, 6x6TB) are instant, no issue.

Code:

smbutil statshares -a

==================================================================================================
SHARE                         ATTRIBUTE TYPE                VALUE
==================================================================================================
Illmatic
                              SERVER_NAME                   192.168.11.10
                              USER_ID                       501
                              SMB_NEGOTIATE                 SMBV_NEG_SMB1_ENABLED
                              SMB_NEGOTIATE                 SMBV_NEG_SMB2_ENABLED
                              SMB_NEGOTIATE                 SMBV_NEG_SMB3_ENABLED
                              SMB_VERSION                   SMB_3.1.1
                              SMB_ENCRYPT_ALGORITHMS        AES_128_CCM_ENABLED
                              SMB_ENCRYPT_ALGORITHMS        AES_128_GCM_ENABLED
                              SMB_ENCRYPT_ALGORITHMS        AES_256_CCM_ENABLED
                              SMB_ENCRYPT_ALGORITHMS        AES_256_GCM_ENABLED
                              SMB_CURR_ENCRYPT_ALGORITHM    OFF
                              SMB_SHARE_TYPE                DISK
                              SIGNING_SUPPORTED             TRUE
                              EXTENDED_SECURITY_SUPPORTED   TRUE
                              UNIX_SUPPORT                  TRUE
                              LARGE_FILE_SUPPORTED          TRUE
                              OS_X_SERVER                   TRUE
                              FILE_IDS_SUPPORTED            TRUE
                              DFS_SUPPORTED                 TRUE
                              FILE_LEASING_SUPPORTED        TRUE
                              MULTI_CREDIT_SUPPORTED        TRUE

--------------------------------------------------------------------------------------------------
Illmatic2
                              SERVER_NAME                   192.168.10.10
                              USER_ID                       501
                              SMB_NEGOTIATE                 SMBV_NEG_SMB1_ENABLED
                              SMB_NEGOTIATE                 SMBV_NEG_SMB2_ENABLED
                              SMB_NEGOTIATE                 SMBV_NEG_SMB3_ENABLED
                              SMB_VERSION                   SMB_3.1.1
                              SMB_ENCRYPT_ALGORITHMS        AES_128_CCM_ENABLED
                              SMB_ENCRYPT_ALGORITHMS        AES_128_GCM_ENABLED
                              SMB_ENCRYPT_ALGORITHMS        AES_256_CCM_ENABLED
                              SMB_ENCRYPT_ALGORITHMS        AES_256_GCM_ENABLED
                              SMB_CURR_ENCRYPT_ALGORITHM    OFF
                              SMB_SHARE_TYPE                DISK
                              SIGNING_SUPPORTED             TRUE
                              EXTENDED_SECURITY_SUPPORTED   TRUE
                              UNIX_SUPPORT                  TRUE
                              LARGE_FILE_SUPPORTED          TRUE
                              OS_X_SERVER                   TRUE
                              FILE_IDS_SUPPORTED            TRUE
                              DFS_SUPPORTED                 TRUE
                              FILE_LEASING_SUPPORTED        TRUE
                              MULTI_CREDIT_SUPPORTED        TRUE

--------------------------------------------------------------------------------------------------

 

[winnielinnie]

Maybe take a look at this thread. (Discussion)

In particular, this post. (Solution)

You can adjust the value accordingly.

Note: You won't notice any difference until the first crawl or so, since it needs to populate your ARC with metadata.


Since you have 128 GiB of RAM, you are more free to explore even higher values, such as 6442450944 (6 GiB) or even 8589934592 (8 GiB).

I've been so pleased with the results, that even on my system with only 32 GiB of RAM, I went ahead and bumped this value to 6442450944 (6 GiB) just in case the directory tree and file count grows in the near future.

---

I would hold off on introducing an L2ARC drive, since leveraging your RAM will yield better performance anyways, especially if metadata crawls are common in your workflow.

 

[wdp]

Thanks @winnielinnie! I will take a look.

I didn't expect Lightroom to be this slow, even on ZFS/TrueNAS out of the box. It's basically unusable at this stage.

 

[winnielinnie]

Keep in mind the first time you try this, it will be just as slow (since it still needs to retrieve metadata from the spinning HDDs.) The next subsequent attempts should be notably snappier, as it will retrieve the metadata from your ARC (RAM) instead.

 

[wdp]

It's pretty painful warming up the cache. I didn't expect it to cripple the applications like this. I guess with enough time that dust tends to settle, but this is basically crashing Lightroom as it tries to load each folder.

 

[winnielinnie]

You can try to "pre-warm" the ARC with metadata by right-clicking the folder in Windows Explorer and viewing its properties. (It will crawl the entire tree, counting the number of files and total size.)

To really seal the deal, from within the TrueNAS server itself, force a crawl down the relevant directory tree:
ls -lahR /mnt/poolname/path/with/files > /dev/null

Either way, once the metadata is all within the ARC, the performance of browsing via Lightroom "should" notably improve. If it doesn't, even after multiple attempts (and using the above "tuneable"), then it might be another culprit causing the behvaior.