CIFS directory browsing slow? Try this

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
This has become a larger and larger problem that has affected more and more users. If your CIFS directory browsing is slow there are multiple ways to resolve the issue.

#1: More hardware-

You can always have more ARC and/or add an L2ARC (you can never have too much ARC). This caches data in RAM (or the L2ARC as appropriate) and should improve performance... after its been read the first time. The catch is the last part of the sentence. If it isn't cached already it will still be slow. If it isn't accessed frequently, then it may be expired from the ARC or L2ARC so it will be slow (again). So clearly not the best short or long-term option, but may be necessary in situations where your ARC and/or L2ARC are stressed and there is a need to relieve some of that stress.

#2: Increase the amount of ARC that stores metadata-


This is controlled by setting the tunable vfs.zfs.arc_meta_limit. Note that this does not force ZFS to use more ARC for metadata, but it makes ZFS more favorable to using more ARC for metadata. Also note that this value is not strictly enforced. If ZFS feels that it is more favorable to use more ARC for metadata than the vfs.zfs.arc_meta_limit then it will exceed the limit, but the higher it goes above the limit the more pressure ZFS has placed on it to not add more metadata to the ARC. ZFS typically uses about 25% of ARC for metadata. Be careful about setting this value too high. If ZFS caches too much metadata and not enough file data, server performance should suffer. There's generally better options if you have this problem.

You can check how many bytes of ARC are currently storing metadata with the command:

# sysctl vfs.zfs.arc_meta_used
vfs.zfs.arc_meta_used: 13967879256


You can check (and adjust) the ARC metadata limit with the following commands:

# sysctl vfs.zfs.arc_meta_limit
vfs.zfs.arc_meta_limit: 12000000000
# sysctl vfs.zfs.arc_meta_limit=16000000000
vfs.zfs.arc_meta_limit: 1200000000 -> 16000000000

#3: Make your L2ARC store metadata exclusively-


This is controlled by the ZFS property secondarycache.

You can set the property for your pool (or dataset) with the command:

# zfs set secondarycache=metadata <poolname or poolname/dataset>

To change it back to default use the command:

# zfs set secondarycache=all tank <poolname or poolname/dataset>

Note that this can have serious consequences for your system if the pool or dataset relies on your L2ARC to store file data. As this is often not desired this isn't generally the best option.

#4 Disable DOS attributes-

Samba imitates DOS attributes (read-only, hidden, system, archive) and ZFS stores them as extended attributes. These are really a throwback to the 80s and not too many people use them anymore. ZFS stores them as extended attributes (EAs) and there is a pretty serious performance penalty with doing so. If you don't need them the best thing you can do is disable them outright. Here's how:

Under the CIFS Settings page you can add the following settings as shown below:
upload_2015-2-19_10-24-32.png


Simply add the following to the Auxiliary Parameters of CIFS settings and save the changes. They will immediately apply themselves.

ea support = no
store dos attributes = no
map archive = no
map hidden = no
map readonly = no
map system = no

This does several things:

1. Disables Samba from using EAs.
2. Tells Samba not to store the DOS attributes (any existing bits that are set are simply abandoned in place in the EAs).
3. Forces all 4 parameters to off/no/unchecked/disabled.

Note: The read-only bit I am mentioning, disabling, then forcing to off is not the standard read-only permissions users may be given to files or folders.

If this is your problem, simply applying the above settings should result in an instantaneous increase in performance. If you have directories with 100k+ files it will still take a few seconds to populate, but you'll find it is MUCH more expedient than it was previously.

If the changes do not help, or you wish to undo the changes simply remove the settings. The old bits will immediately reassert themselves as they were (the abandoned bits are never overwritten with these changes).

Problems with this approach:

If you, the software you use, hardware that uses the CIFS shares directly, or your end-users rely on these settings this will break things for them.

By "software you use" I generally mean backup software. Some old-fashioned backup software uses the old "archive" bit to determine what does or doesn't need backed up. This shouldn't be used by anything that is even remotely recent, so unless you've explicitly enabled the backup software to use the archive bit or you are using backup software that is pre-2005 or so, you shouldn't have a problem.

By "hardware that uses the CIFS shares directly" I'm refering to things like all-in-one scan/copy/fax/print machines that let you scan documents to a CIFS share. Those will sometimes want to force one or more bits to on or off, and those operations will fail.

From my own personal experience, anyone that has had slow Samba directory listings have seen amazing performance increases by using these parameters. Nobody I've worked with directly has required additional changes above and beyond the above 6 parameters.

Generally its safe to try these settings and if there isn't any problems for a week its safe to assume that nothing required those bits.

Edit (4/24/2017): Directory listing performance should be much improved as of 9.10.2-U3 (released 4/21/2017).
 
Last edited:

JJT211

Patron
Joined
Jul 4, 2014
Messages
323

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
oh sweet....just seeing this thread.

I tired this tweak from jpaetzel in the performance forum and I went from 20-30 MB/s to 100 MB/s
https://forums.freenas.org/index.php?threads/lan-performance.8089/

Ill have a looksy when IK get home later from class
Yeah the default for that value for more than 2 years has been the exact value that the thread tells you to change. That thread is from 2012, and the default changed shortly after that post.

So unless you are going to tell me you are using 8.0.4 or something older than 2012, that "tweak" didn't actually change any setting at all, so I really doubt you'd see performance change from that setting.

Code:
[root@mini] ~# sysctl -a | grep delayed
net.inet.tcp.delayed_ack: 0
 

JJT211

Patron
Joined
Jul 4, 2014
Messages
323
Well somehow, someway, my CIFS transfers speed did jump immediately after setting value, no other changes. I dunno.
 

JJT211

Patron
Joined
Jul 4, 2014
Messages
323
I was curious as to why it seems nobody is using it anymore, which is why I posted it here in the first place. Ill try duplicating it by disabling the tweak and testing CIFS speeds again. You're probably right, its placebo.

EDIT:

Just saw the command you posted and you're right. After deleting sysctl tweak, I typed command and same thing.

I judged the difference a bit too quickily as CIFS speeds mostly seem to start really fast but slow down considerably when they settle.
 
Last edited:

JJT211

Patron
Joined
Jul 4, 2014
Messages
323
I have a question regarding #2: Increase the amount of ARC that stores metadata-

I noticed, like your example above, that the metadata used was above the metadata limit.


[root@freenas] ~# sysctl vfs.zfs.arc_meta_limit
vfs.zfs.arc_meta_limit: 3874161664
[root@freenas] ~# sysctl vfs.zfs.arc_meta_used
vfs.zfs.arc_meta_used: 446137736

So I set it slightly higher than what im currently using.

[root@freenas] ~# sysctl vfs.zfs.arc_meta_limit=500000000
vfs.zfs.arc_meta_limit: 3874161664 -> 5000000000
[root@freenas] ~# sysctl vfs.zfs.arc_meta_used
vfs.zfs.arc_meta_used: 446105464
[root@freenas] ~# sysctl vfs.zfs.arc_meta_limit
vfs.zfs.arc_meta_limit: 5000000000
[root@freenas] ~#

Then after I realized that the ARC Size value is actually displayed in the Reporting tab, I looked back at the history and noticed that my system normally settles at about 10GB, which is about half my current limit. much smaller. I recently rebooted my server, which is why I think its (ARC size = 5GB) currently kind of low. I noticed you have a bit higher spec'd machine so it prob wouldnt be a good idea to use your limit. So my question is, considering the specs in my sig, where do you think I should set my limit. Slightly above 10G, maybe around 11?

EDIT: I miss counted 0's at first and had it set to 500MB instead of 5GB! And I was misreading my used meta. Thats 446MB used. My old limit was already plenty above at around 3.8 GB. So nevermind then, Ill leave this here as a warning to others.

Moral of the story, just go with:

#4 Disable DOS attributes
 
Last edited:

AlainD

Contributor
Joined
Apr 7, 2013
Messages
145
This has become a larger and larger problem that has affected more and more users. If your CIFS directory browsing is slow there are multiple ways to resolve the issue.

#4 Disable DOS attributes-

Samba imitates DOS attributes (read-only, hidden, system, archive) and ZFS stores them as extended attributes. These are really a throwback to the 80s and not too many people use them anymore. ZFS stores them as extended attributes (EAs) and there is a pretty serious performance penalty with doing so. If you don't need them the best thing you can do is disable them outright. Here's how:

Under the CIFS Settings page you can add the following settings as shown below:
View attachment 6748

Simply add the following to the Auxiliary Parameters of CIFS settings and save the changes. They will immediately apply themselves.

ea support = no
store dos attributes = no
map archive = no
map hidden = no
map readonly = no
map system = no

This does several things:

1. Disables Samba from using EAs.
2. Tells Samba not to store the DOS attributes (any existing bits that are set are simply abandoned in place in the EAs).
3. Forces all 4 parameters to off/no/unchecked/disabled.

Note: The read-only bit I am mentioning, disabling, then forcing to off is not the standard read-only permissions users may be given to files or folders.

If this is your problem, simply applying the above settings should result in an instantaneous increase in performance. If you have directories with 100k+ files it will still take a few seconds to populate, but you'll find it is MUCH more expedient than it was previously.

If the changes do not help, or you wish to undo the changes simply remove the settings. The old bits will immediately reassert themselves as they were (the abandoned bits are never overwritten with these changes).

Problems with this approach:

If you, the software you use, hardware that uses the CIFS shares directly, or your end-users rely on these settings this will break things for them.

By "software you use" I generally mean backup software. Some old-fashioned backup software uses the old "archive" bit to determine what does or doesn't need backed up. This shouldn't be used by anything that is even remotely recent, so unless you've explicitly enabled the backup software to use the archive bit or you are using backup software that is pre-2005 or so, you shouldn't have a problem.

By "hardware that uses the CIFS shares directly" I'm refering to things like all-in-one scan/copy/fax/print machines that let you scan documents to a CIFS share. Those will sometimes want to force one or more bits to on or off, and those operations will fail.

From my own personal experience, anyone that has had slow Samba directory listings have seen amazing performance increases by using these parameters. Nobody I've worked with directly has required additional changes above and beyond the above 6 parameters.

Generally its safe to try these settings and if there isn't any problems for a week its safe to assume that nothing required those bits.

Thanks directory browsing is a lot faster after that small adjustment, makes CIFS on FreeNAS and thus FreeNAS itself, usable again.
 

Mike77

Contributor
Joined
Nov 15, 2014
Messages
193
This has become a larger and larger problem that has affected more and more users. If your CIFS directory browsing is slow there are multiple ways to resolve the issue.

#4 Disable DOS attributes-

Samba imitates DOS attributes (read-only, hidden, system, archive) and ZFS stores them as extended attributes. These are really a throwback to the 80s and not too many people use them anymore. ZFS stores them as extended attributes (EAs) and there is a pretty serious performance penalty with doing so. If you don't need them the best thing you can do is disable them outright. Here's how:

Under the CIFS Settings page you can add the following settings as shown below:
View attachment 6748

Simply add the following to the Auxiliary Parameters of CIFS settings and save the changes. They will immediately apply themselves.

ea support = no
store dos attributes = no
map archive = no
map hidden = no
map readonly = no
map system = no

Did it and it works like a charm! Amazing :smile:
 

neckbeard

Cadet
Joined
Mar 5, 2014
Messages
3
Disabling the DOS attributes also fixed my issue where file transfers would go from 80-100MB/s, stall to 0, and then go back up to 80-100 in a wave formation. That behavior started about a month ago from the updates.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Disabling the DOS attributes also fixed my issue where file transfers would go from 80-100MB/s, stall to 0, and then go back up to 80-100 in a wave formation. That behavior started about a month ago from the updates.

If file transfers themselves are fluctuating this badly you've probably under-resourced your system in some fashion.
 

neckbeard

Cadet
Joined
Mar 5, 2014
Messages
3
If file transfers themselves are fluctuating this badly you've probably under-resourced your system in some fashion.

I've been running the same config for about two years and this only started happening in the last few weeks. Currently, I have a FreeNAS virtual machine on ESXi 6.0 with 8GB of ram and pass through to an LSI 9211i in IT mode with 6 WD Red 3TB in RAIDZ2 and 32GB of L2ARC. I can push 120MB/s+ across the VMware environment where everything is 10gb and 100MB/s across a gigabit LAN.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
I've been running the same config for about two years and this only started happening in the last few weeks.

Honestly, that means absolutely nothing in terms of diagnosing the problem (or validating one argument over another). ZFS needs more resources the more it is used. Fragmentation, disk usage, etc all only get worse as time goes on. So the fact that it suddenly started happening lately (even if it started happening clear out of the blue) doesn't mean anything. In fact, if you haven't made any changes, your comment tends to give me the advantage on the likely status of the situation. ;)

Currently, I have a FreeNAS virtual machine on ESXi 6.0 with 8GB of ram and pass through to an LSI 9211i in IT mode with 6 WD Red 3TB in RAIDZ2 and 32GB of L2ARC. I can push 120MB/s+ across the VMware environment where everything is 10gb and 100MB/s across a gigabit LAN.

That sums it up. 8GB of RAM is not appropriate for 18TB of disk space. You also shouldn't have *any* L2ARC with just 8GB of RAM. 8GB of RAM is the minimum. It is NOT sufficient for a 32GB L2ARC. In fact, even an 8GB L2ARC would probably oversize and starve your system.

So yeah, you pretty much validated that you starved your system and fixed it by removing some of the load. No doubt you are overloading the system with it's limited resources and no doubt you'll someday have the system's performance come crashing down. As the sole user of my system I went to bed one night with speeds in excess of 300MB/sec. I woke up the next morning and couldn't even do 5MB/sec. Why? I hit the cliff where my system was under-resourced enough that there was no going back. I had to double my RAM to correct the issue.
 
Last edited:

Aberu

Cadet
Joined
Jan 30, 2015
Messages
6
Your #4 method really helped me a while ago. I also have helped a few other people on /r/freenas with the same solution by redirecting them here. Thanks!
 

rogerh

Guru
Joined
Apr 18, 2014
Messages
1,111
I added the #4 dos attribute removal parameters to the CIFS service auxiliary parameters. This was a few weeks ago but a I am not sure when. I am now on the latest stable FreeNAS 9.3.1 I noticed two things today: in the CIFS service dialogue box had appeared the entry "browseable = ea support = no", which is clearly an error of some kind. I replaced it with "browseable = yes" and after a newline "ea support = no. This seems to be accepted and neither parameter appears in testparm output, which is probably good as both my values appear to be defaults (at least in some versions of Samba).

However my "/usr/local/etc/smb4.conf" has two entries for "ea support", one set to "no" and one set to "yes". Judging by testparm I shouldn't worry about this? What do people think?
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
I wouldn't worry about it.
 

rogerh

Guru
Joined
Apr 18, 2014
Messages
1,111

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
9,543
Simply add the following to the Auxiliary Parameters of CIFS settings and save the changes. They will immediately apply themselves.

ea support = no
store dos attributes = no
map archive = no
map hidden = no
map readonly = no
map system = no

I haven't had reason to test it myself, but the samba documentation lists these parameters with an (S) meaning that they can be applied at a share level as well. This means users can fine-tune how dos attributes are stored on a per-share basis depending on whether they need the archive bit.
 

webpilot

Dabbler
Joined
Nov 9, 2015
Messages
27
Does one need to reboot the server or just restart CIFS service for the benefit to be seen? I am not seeing an improvement with #4.

upload_2015-12-13_8-28-48.png

Image shows one disk with a file of the same size of file before and after on the right. I canceled the 2nd transfer when it kept dropping in transfer speeds.

I was copying from one FreeNAS to another. The server build is in the header and identical for both units. Win7 box acting as CIFS client.

Any ideas?
 

webpilot

Dabbler
Joined
Nov 9, 2015
Messages
27
Here is the write view on the receiving 9.3.1 stable box. B was before and A is after the Auxiliary Parameters changes were saved.
upload_2015-12-13_8-49-36.png


Both CIFS options were setup with the six Auxiliary Parameters.

This started out as a means to mass copy files and stop an nvalid MS-DOS popp-up. I was getting an Invalid MS-DOS issue reading files from the freenas server to the win7 client from files copied over from an external drive and suspected an extended attributes issues. If it is an EA issue, are all six entries needed in the CIFS parameters to work correctly?

Thanks
 
Top