Intermittently slow CIFS share using Lightroom

Status
Not open for further replies.

moon

Dabbler
Joined
Jul 17, 2014
Messages
32
A few weeks ago, after some research, I decided to build a FreeNAS system to replace the QNAP NAS I've used for years.
In doing so I decided to stick as much as possible with the recommendations of the forum experts and thanks to that even a total noob like me was able to plan for and perform the build without encountering any major issue (many thanks especially to cyberjock and jgreco for their posts on hardware selection and system burn-in).
Burn-in is done and I'm now playing a bit with the system to familiarize with it before deploying it as my main storage unit.

My system configuration:
FreeNAS version: 9.2.1.7
Motherboard: Supermicro X10SLL-F
CPU: Intel E3-1230V3
RAM: 32GB - 2 x Crucial CT2KIT102472BD160B 16GB (2 x 8GB)
Boot device: 8GB Sandisk flash drive on internal USB Type-A connector
Hard Drives: 4 x WD4001FFSX in RAIDZ2 (for my data) + 2 X WD40EFRX in MIRROR (for snapshots replications and other backups)
PSU: Seasonic G-450
UPS: Powercom Vanguard 1000 LCD


The main issue I'm facing right now is an extremely slow CIFS share in some specific and repeatable conditions.
The share configuration was pretty straightforward.
I was also able to map the CIFS share as a network drive in my Win7 workstation without trouble.
Read / Write performance using the Win7 standard file manager and several other file transfer utilities is as fast as expected and very consistent.
I've also run speed test utilities and again results are good (write 80MB/s , read 100MB/s).
The problem is related to inconsistent performance with Lightroom 5 (my main application ...)

Lightroom 5.6 is installed on my Win7 workstation.
Photographs files (RAW of 25MB each) are in a dedicated dataset in the RAIDZ2 pool.
The share is pointing at this dataset (but I've tried pointing it at the pool without any noticeable difference).
Lightroom saves all of the changes to local files (catalogs, previews and cache files) , original photographs files remain unchanged.
Lightroom local files are on the Win7 workstation.
The connection between the FreeNAS system and the workstation is GBe through a Cisco router.
Lightroom reads the photographs files during import (into the Lightroom database), previews generation, image changes and other tasks.
(a) In some occasions these tasks are executed at high speed (performing a file transfer and/or speed test in parallel confirm high connection speed) but,
(b) in other occasions (especially during import) the same exact tasks are extremely slow (and performing a file transfer and/or speed test in parallel confirms the low speed of the connection: < 10MB/s).

If I terminate the Lightroom task, after a while, the share goes back to the expected performance.

I'm not able to see any differences both at the FreeNAS and the Win7 sides in (a) and (b), other than the extremely different performance.
At the moment I have no idea about how to troubleshoot the issue. I'm just able to monitor the LAN transfer rate at both the win7 and freenas sides and everything confirms the connection becomes slow in some occasions.

Has anybody experienced the same issue?
Any idea about how to troubleshoot the issue?

Regards

AM
 
Last edited:

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
9,545
These sorts of performance problems can be difficult to troubleshoot, and speed test utilities are not overly helpful. Honestly, performance on any protocol starts to suck when you have lots of smallish files.

To start with, attach your smb4.conf file. It's located at /usr/local/etc/smb4.conf.

On a side note, Your storage configuration is less than ideal. Basically 3 out of your 6 drives are parity drives. You might as well just use mirrors. You'd get better pool performance.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
T
On a side note, Your storage configuration is less than ideal. Basically 3 out of your 6 drives are parity drives. You might as well just use mirrors. You'd get better pool performance.

Quoted for truth.

There are some things in the works. One of the FreeNAS devs is looking at an alternative strategy that *may* help (significantly) in some (very specific) circumstances. Unfortunately unless you can find a repeatable test that causes slow performance outside of Lightroom there is no way to prove that Lightroom isn't the problem somehow. It could be doing something silly like opening and closing the same 20-30 files in quick succession and multiple times because of some poorly optimized code.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
Quoted for truth.

There are some things in the works. One of the FreeNAS devs is looking at an alternative strategy that *may* help (significantly) in some (very specific) circumstances. Unfortunately unless you can find a repeatable test that causes slow performance outside of Lightroom there is no way to prove that Lightroom isn't the problem somehow. It could be doing something silly like opening and closing the same 20-30 files in quick succession and multiple times because of some poorly optimized code.

Some Adobe applications have caused serious, inexplicable issues before. I specifically remember Photoshop CS3 (I believe it was CS3, but am not certain) on OS X deleting files it's supposed to write, but only on CIFS shares.
I don't think that case ever got a true explanation...
 

moon

Dabbler
Joined
Jul 17, 2014
Messages
32
anodos,
cyberjock,

thank you for your replies.

What actually bothers me is not the performance itself, in fact when it works it's better than what I was hoping for. The issue is that in same cases it drops to less than 10%.
Anyway, here is the smb4.conf:

Code:
[global]
  server max protocol = SMB2
  encrypt passwords = yes
  dns proxy = no
  strict locking = no
  oplocks = yes
  deadtime = 15
  max log size = 51200
  max open files = 11070
  load printers = no
  printing = bsd
  printcap name = /dev/null
  disable spoolss = yes
  getwd cache = yes
  guest account = nobody
  map to guest = Bad User
  obey pam restrictions = Yes
  directory name cache size = 0
  kernel change notify = no
  panic action = /usr/local/libexec/samba/samba-backtrace
  server string = FreeNAS Server
  ea support = yes
  store dos attributes = yes
  time server = yes
  acl allow execute always = true
  idmap config *:backend = tdb
  idmap config *:range = 90000000-100000000
  server role = standalone
  netbios name = FREENAS
  workgroup = WORKGROUP
  security = user
  pid directory = /var/run/samba
  smb passwd file = /var/etc/private/smbpasswd
  private dir = /var/etc/private
  create mask = 0666
  directory mask = 0777
  client ntlmv2 auth = yes
  dos charset = CP437
  unix charset = UTF-8
  log level = 1


[zpool_000_Photo]
  path = /mnt/zpool_data/000_Photo
  printable = no
  veto files = /.snap/.windows/.zfs/
  writeable = yes
  browseable = yes
  recycle:repository = .recycle/%U
  recycle:keeptree = yes
  recycle:versions = yes
  recycle:touch = yes
  recycle:directory_mode = 0777
  recycle:subdir_mode = 0700
  shadow:snapdir = .zfs/snapshot
  shadow:sort = desc
  shadow:localtime = yes
  shadow:format = auto-%Y%m%d.%H%M-4w
  vfs objects = shadow_copy2 zfsacl streams_xattr aio_pthread
  hide dot files = yes
  guest ok = no
  nfs4:mode = special
  nfs4:acedup = merge
  nfs4:chown = yes
  zfsacl:acesort = dontcare


[zpool_Multimedia]
  path = /mnt/zpool_data/Multimedia
  printable = no
  veto files = /.snap/.windows/.zfs/
  writeable = yes
  browseable = yes
  recycle:repository = .recycle/%U
  recycle:keeptree = yes
  recycle:versions = yes
  recycle:touch = yes
  recycle:directory_mode = 0777
  recycle:subdir_mode = 0700
  vfs objects = zfsacl streams_xattr aio_pthread
  hide dot files = yes
  guest ok = no
  nfs4:mode = special
  nfs4:acedup = merge
  nfs4:chown = yes
  zfsacl:acesort = dontcare


[zpool_backups]
  path = /mnt/zpool_backups
  printable = no
  veto files = /.snap/.windows/.zfs/
  writeable = yes
  browseable = yes
  recycle:repository = .recycle/%U
  recycle:keeptree = yes
  recycle:versions = yes
  recycle:touch = yes
  recycle:directory_mode = 0777
  recycle:subdir_mode = 0700
  vfs objects = zfsacl streams_xattr aio_pthread
  hide dot files = yes
  guest ok = no
  nfs4:mode = special
  nfs4:acedup = merge
  nfs4:chown = yes
  zfsacl:acesort = dontcare


[zpool_data]
  path = /mnt/zpool_data
  printable = no
  veto files = /.snap/.windows/.zfs/
  writeable = yes
  browseable = yes
  recycle:repository = .recycle/%U
  recycle:keeptree = yes
  recycle:versions = yes
  recycle:touch = yes
  recycle:directory_mode = 0777
  recycle:subdir_mode = 0700
  vfs objects = zfsacl streams_xattr aio_pthread
  hide dot files = yes
  guest ok = no
  nfs4:mode = special
  nfs4:acedup = merge
  nfs4:chown = yes
  zfsacl:acesort = dontcare




Regarding the storage configuration, I made a correction to my initial post, the correct backups zpool description is "2 X WD40EFRX in MIRROR (for snapshots replications and other backups)" and not RAIDZ1 as originally posted.

Let me add that in my own experience the storage configuration is probably the most difficult decision to take in designing a ZFS system for the first time and any recommendation (e.g. more examples) for experienced users would be useful.

In designing the system my priorities were: 1) reliability, 2) performance , 3) cost

I wanted to make use of redundancy, snapshots and replications as a mean to increase data reliability and that's why I created a second zpool (btw initially using 2 old 2TB HDDs, later replaced by the 2 4TB Red).
This is a subject I need to spend some more thoughts on. Having reliability as the first priority which of the following configurations would be preferable ?

1) 6 X 4TB HDD in RAIDZ3
2) 4 X 4TB HDD in RAIDZ2 + 2 X 4TB HDD in MIRROR with replication of RAIDZ2 to MIRROR

I guess 1) (if single HDD reliability is the only factor).

AM
 

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
9,545
Some Adobe applications have caused serious, inexplicable issues before. I specifically remember Photoshop CS3 (I believe it was CS3, but am not certain) on OS X deleting files it's supposed to write, but only on CIFS shares.
I don't think that case ever got a true explanation...
I think that samba feature may be due to the way photoshop uses resource forks on Macs. There's a new VFS module called "VFS_FRUIT" that should help take care of the resource fork stream and provide stabler and better performance. See here: https://bugzilla.samba.org/show_bug.cgi?id=10683
 

moon

Dabbler
Joined
Jul 17, 2014
Messages
32
Quoted for truth.

There are some things in the works. One of the FreeNAS devs is looking at an alternative strategy that *may* help (significantly) in some (very specific) circumstances. Unfortunately unless you can find a repeatable test that causes slow performance outside of Lightroom there is no way to prove that Lightroom isn't the problem somehow. It could be doing something silly like opening and closing the same 20-30 files in quick succession and multiple times because of some poorly optimized code.

So far I've been able to observe the issue only when Lightroom is running and executing import tasks.
I was considering the fact that the issue might be related to Lightroom itself and I was hoping we had a mean to monitor and troubleshoot a behavior e.g. similar to the one you mentioned.
Unless I can show them something specific, I don't think Adobe would spend any time on a totally generic claim.

The main reason why I decided to make an investment in FreeNAS is to have a better place for my photographs and this issue is really painful for me
 

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
9,545
anodos,
cyberjock,

thank you for your replies.

What actually bothers me is not the performance itself, in fact when it works it's better than what I was hoping for. The issue is that in same cases it drops to less than 10%.
Anyway, here is the smb4.conf:
  1. Try setting "max protocol" in your CIFS config to the maximum that Samba supports (3_00). I know that W7 doesn't support SMB3, but "SMB2" is, I think, some early version of SMB2 that's not actually used by W7.
  2. If you have some flexibility with restructuring your pool, you may want to try configuring 3 striped mirrors (each VDEV has the write IOPS performance of a single disk).
  3. You also may want to try (for testing purposes only) setting sync=disabled on your dataset for photos
    Code:
    zfs set sync=disabled tank/dataset 
    re-enable it once you are done. This will tell you whether lightroom is making lots of sync writes (which can really suck if you don't have a SLOG). If it eliminates your performance problem, then you may benefit from a SLOG. Alternatively, you can use zilstat to track sync writes by typing "zilstat 1" in your putty session and compare zil usage during import vs. regular samba browsing.
  4. If it's a bug in an adobe product, I wouldn't hold my breath about them actually fixing it. Their products are like swiss cheese.
 
Last edited:

solarisguy

Guru
Joined
Apr 4, 2014
Messages
1,125
You might need to learn a thing or two, but if you could spare some time, try using NFS instead of CIFS (Samba).

FreeNAS would be happy to server NFS, and since you look like the only user, you do not need to worry about permissions...

Windows 7 has ability to use Client for NFS distributed by Microsoft. You can also try NFSv4.1 Client for Windows available (binaries too) at http://www.citi.umich.edu/projects/nfsv4/windows/
 

RobD68

Cadet
Joined
Jul 1, 2014
Messages
3
I'm going to keep an eye on this thread as I am about to start using my FreeNAS box for the exact same thing. I went with a RAIDZ2 setup on my drives because I wanted the protection from bit-rot that mirroring doesn't provide. I'm just finishing up my setup and about to take the box home to replace my EX490/WHS2011 setup.
 

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
9,545
I'm going to keep an eye on this thread as I am about to start using my FreeNAS box for the exact same thing. I went with a RAIDZ2 setup on my drives because I wanted the protection from bit-rot that mirroring doesn't provide. I'm just finishing up my setup and about to take the box home to replace my EX490/WHS2011 setup.
Scrubs work on mirrors.
 

RobD68

Cadet
Joined
Jul 1, 2014
Messages
3
Scrubs work on mirrors.

Yes, but if the 2nd drive in the mirror fails before the rebuild my entire pool goes down. This way I can lose any two drives and still have a happy family. The alternative was doing a Z2 of 2x4x3TB but that resulted in only a 50% capacity capability to get extra IOPS. I spent plenty of time debating between the 2xZ10-4D, the 2xZ2-4D, the 1xZ1-6D (optimal), and the 1xZ1-8D. It finally came down to just pure storage capacity with an acceptable level of parity protection.

But let's get back to the performance issues of the original topic...
 

moon

Dabbler
Joined
Jul 17, 2014
Messages
32
  1. Try setting "max protocol" in your CIFS config to the maximum that Samba supports (3_00). I know that W7 doesn't support SMB3, but "SMB2" is, I think, some early version of SMB2 that's not actually used by W7.
  2. If you have some flexibility with restructuring your pool, you may want to try configuring 3 striped mirrors (each VDEV has the write IOPS performance of a single disk).
  3. You also may want to try (for testing purposes only) setting sync=disabled on your dataset for photos
    Code:
    zfs set sync=disabled tank/dataset 
    re-enable it once you are done. This will tell you whether lightroom is making lots of sync writes (which can really suck if you don't have a SLOG). If it eliminates your performance problem, then you may benefit from a SLOG. Alternatively, you can use zilstat to track sync writes by typing "zilstat 1" in your putty session and compare zil usage during import vs. regular samba browsing.
  4. If it's a bug in an adobe product, I wouldn't hold my breath about them actually fixing it. Their products are like swiss cheese.


First I run "zilstat" (3) to check sync writes and got mostly zeroes. This is what I was expecting since Lightroom uses non destructive editing and this means that all writes are done to local files (catalog=database, images previews, cache, etc) and local files are on my workstation, not on the freenas system. Lightroom does not write to the pool, just reads from it.

Then I changed Server maximum protocol from "SMB2" to "SMB3_00" (1). It seems it works fine but I need to test it for a while before saying that the problem is solved (it's intermittent ...). I'll provide feedback after a few days of tests.

Regarding storage configuration, I might be missing something here but I thought that striped mirrors were for performance with minimal redundancy (1 disk). In a 3 striped mirrors pool, should 2 disks in the same mirrors fail, the pool would fail.
I my case I want to maximize reliability so a RAIDZ2 looks a better choice to me. Am I worng ?
In my current configuration (4 x 4TB in RAIDZ2 for data + 2 x 4TB in MIRROR for replication) 4 disks can fail in any of the 2 pools and my data are still safe (5 disks could fail even though not in any of the 2 pools). In addition to that having 2 pools gives me the option to located the jails and system datasets in the mirror pool so that the raidz2 is dedicated to photos without any other process interfering.
 

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
9,545
First I run "zilstat" (3) to check sync writes and got mostly zeroes. This is what I was expecting since Lightroom uses non destructive editing and this means that all writes are done to local files (catalog=database, images previews, cache, etc) and local files are on my workstation, not on the freenas system. Lightroom does not write to the pool, just reads from it.

Then I changed Server maximum protocol from "SMB2" to "SMB3_00" (1). It seems it works fine but I need to test it for a while before saying that the problem is solved (it's intermittent ...). I'll provide feedback after a few days of tests.

Regarding storage configuration, I might be missing something here but I thought that striped mirrors were for performance with minimal redundancy (1 disk). In a 3 striped mirrors pool, should 2 disks in the same mirrors fail, the pool would fail.
I my case I want to maximize reliability so a RAIDZ2 looks a better choice to me. Am I worng ?
In my current configuration (4 x 4TB in RAIDZ2 for data + 2 x 4TB in MIRROR for replication) 4 disks can fail in any of the 2 pools and my data are still safe (5 disks could fail even though not in any of the 2 pools). In addition to that having 2 pools gives me the option to located the jails and system datasets in the mirror pool so that the raidz2 is dedicated to photos without any other process interfering.
Yes. If the second drive in a mirror fails then you lose your pool. The suggestion is related to performance rather than redundancy. That being said, resilvering a mirror is very fast and you should be careful about backups anyways (even on RAIDZ2) because hard drive failure isn't the only way to lose your data.
 

moon

Dabbler
Joined
Jul 17, 2014
Messages
32
You might need to learn a thing or two, but if you could spare some time, try using NFS instead of CIFS (Samba).

FreeNAS would be happy to server NFS, and since you look like the only user, you do not need to worry about permissions...

Windows 7 has ability to use Client for NFS distributed by Microsoft. You can also try NFSv4.1 Client for Windows available (binaries too) at http://www.citi.umich.edu/projects/nfsv4/windows/

I'll try to spare the time for NFS on Windows.
I've already experimented a bit with it when I started noticing the issue with CIFS and wanted to compare the two shares. I was able to start the client in win and map the share but could not access the dataset due to permissions issues. Having to manage UNIX permissions was a bit scaring and gave up.... You say that as a single users I should not have issues with that so ..... I'll have to work on my homework and see if I can sort it out.
 

moon

Dabbler
Joined
Jul 17, 2014
Messages
32
First I run "zilstat" (3) to check sync writes and got mostly zeroes. This is what I was expecting since Lightroom uses non destructive editing and this means that all writes are done to local files (catalog=database, images previews, cache, etc) and local files are on my workstation, not on the freenas system. Lightroom does not write to the pool, just reads from it.

Then I changed Server maximum protocol from "SMB2" to "SMB3_00" (1). It seems it works fine but I need to test it for a while before saying that the problem is solved (it's intermittent ...). I'll provide feedback after a few days of tests.

Regarding storage configuration, I might be missing something here but I thought that striped mirrors were for performance with minimal redundancy (1 disk). In a 3 striped mirrors pool, should 2 disks in the same mirrors fail, the pool would fail.
I my case I want to maximize reliability so a RAIDZ2 looks a better choice to me. Am I worng ?
In my current configuration (4 x 4TB in RAIDZ2 for data + 2 x 4TB in MIRROR for replication) 4 disks can fail in any of the 2 pools and my data are still safe (5 disks could fail even though not in any of the 2 pools). In addition to that having 2 pools gives me the option to located the jails and system datasets in the mirror pool so that the raidz2 is dedicated to photos without any other process interfering.

Bad news. Unfortunately it seems that setting Server maximum protocol from "SMB2" to "SMB3_00" does not harm but does not provide benefits as well.

Let me describe the sequence of events once again:
File transfer speed is good (monitored with a file transfer reading from the pool and writing to a desktop and windows monitor on the desktop)
A Lightroom import is started and, if the import is slow, then the file transfer speed drops to few Mbps (and the windows monitor show very low network traffic )
Cancel import and close Lightroom and the speed goes back to normal
The above happens
with Lightroom working on files in RAIDZ2/datasetphotos and share datasetphotos
with the file transfer either from RAIDZ2/datasetphotos and share datasetphotos OR RAIDZ2/datasetotherdata and share datasetotherdata OR MIRROR/datasetmittor and share datasetmirror

It looks like Lightroom has an effect not only on the share it's using but on all of them.
 

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
9,545
During import lightroom might be generating huge amounts of thumbnails.

You can view zpool activity by running "zpool iostat 1" and comparing the zpool activity during import vs zpool activity during normal file transfers.

You can also increase samba's logging level, capture a sample during import and post here.
For the sake of thoroughness, try to recreate the problem with your FreeNAS and workstation directly connected via cat6.

Note that rebooting between tests can help to reproduce problems (removes all data from arc). One of your goals should be to narrow down the exact circumstances in which performance tanks (turn it from an intermittent problem to one that you can reproduce at will).
 
Last edited:

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
9,545
I have a hunch. Through the console, go to an image you've imported. Run lsextattr and post output.

To explain: lightroom may be using alternate data streams to store metadata. I believe the samba vfs module streams_xattr takes this metadata and stores it as an extended attribute.
 
Last edited:

moon

Dabbler
Joined
Jul 17, 2014
Messages
32
I have a hunch. Through the console, go to an image you've imported. Run lsextattr and post output.

To explain: lightroom may be using alternate data streams to store metadata. I believe the samba vfs module streams_xattr takes this metadata and stores it as an extended attribute.


Not sure I fully understand what I'm doing and how to judge results, however:

Code:
[root@freenas] /mnt/zpool_data/000_Photo/20140607# lsextattr system DXB_4366.CR2
DXB_4366.CR2
[root@freenas] /mnt/zpool_data/000_Photo/20140607# lsextattr user DXB_4366.CR2
DXB_4366.CR2  DOSATTRIB


To my knowledge Lightroom does not modify the original photo files in any way. All metadata and changes are saved in the database files that are saved locally on the workstation.
Lightroom read the dataset, does not write to it.

In 15 minutes I'll be on my way to the airport for a 10 days trip. I won't be able to access the freenas until I'll be back (but I might be able to read the forum).
Any suggestion received in the meantime will be actioned upon my return.
 

Pasquale61

Explorer
Joined
Oct 8, 2014
Messages
62
I'm a total noob but I'm very interested in seeing how this turns out as I am a heavy Lightroom user and am currently trying to decide between FreeNAS and Synology. (My current rig is a Win7 machine with internal RAID5 that I have outgrown.)

My background is primarily networks so I have some questions/suggestions in case you haven't already tried them:

You mention you're using a Cisco switch. (What model?) Have you been able to rule out the network? Is it just your PC and the NAS on the same switch?

Regarding Lightroom, have you tried testing this with a new empty catalog, just to see if it has anything to do with it? Some of my catalogs are very large and I have noticed inconsistent performance problems, especially on imports. I know you said you did a parallel copy outside of Lightroom and it was slow as well, but I'm assuming you mean from the same PC. If at all possible, I would try the parallel copy from another PC while you're experiencing the slowness, just to rule out your Win7 PC and/or Lightroom. I would also try opening Resource Monitor on your Win7 PC to monitor network, disk and CPU activity while you're doing your imports. This is a great tool to help troubleshoot bottlenecks and may help you see what Lightroom is doing behind the scenes. One last comment about LR. There's an option that autowrites a metadata sidecar file (XMP?) for each of your photos, as well as the usual metadata in the catalog. I think those get written out to the same folder as the RAW files, which can result in random performance problems according to Adobe. Just a thought in case that's what you're doing.
 
Status
Not open for further replies.
Top