SOLVED Browsing directories slow

[Deaks2]

So further investigation has found that all file operations are slow, until, the connection to the client and server has been active for a while...

e.g.: Today I came home, powered up my PC and copied a set of files from one folder on the NAS to another. Transfer speed = 194 KB/s, however, half-way through the transfer it shoots up to the speeds it should be at (i.e.: 100+ MB/s)

During this time the disk access light on the NAS is solid and Samba is taking up 150 MB of RAM.

 

[ProtoSD]

I think I recall a cached directories variable for samba? I think? Sorry I can't check mine now, seems like it should have been one of the settings I listed, but I don't remember. ;)

 

[cyberjock]

It sounds to me like your hard drives might be waking up or one of your hard drives might be going bad.

 

[Deaks2]

Further testing has revealed that I can run "max xmit" at up to around 50000, however, 65000+ causes samba to exit on signal 6.

 

[Deaks2]

Performing the tweaks listed here: www.sysprobs.com/windows-7-network-slow, and excluding my mapped NAS drive from my virus scanner (MS Security Essentials) has fixed my issue.

It appears it was all client side. Funny thing is, my Win7 notebook over Wifi loads the file listing a few seconds quicker than my Win7 desktop over GigE.

 

[cyberjock]

Further testing has revealed that I can run "max xmit" at up to around 50000, however, 65000+ causes samba to exit on signal 6.

My max xmit is set to 262144 and I have zero issues.

 

[BillyTheFish]

Performing the tweaks listed here: www.sysprobs.com/windows-7-network-slow, and excluding my mapped NAS drive from my virus scanner (MS Security Essentials) has fixed my issue.

It appears it was all client side. Funny thing is, my Win7 notebook over Wifi loads the file listing a few seconds quicker than my Win7 desktop over GigE.

Excluding my mapped NAS drive from my virus scanner. Why didn't I think of that?!? Now I feel silly :p

Great tip! It sped things up even more and, with the tunables from post #7, I'm almost getting real-time directory browsing in Windows 7 now.


Protosd - hope the extended holiday is going well and I can't wait for that Serviio plugin. I've got a couple of Pure Flow radios and can't wait to be able to stream to them. :)

By the way, which is the latest list of tunables? Post #7 http://forums.freenas.org/showthread.php?5338-Browsing-directories-slow&p=26269&viewfull=1#post26269 or this one http://forums.freenas.org/showthrea...ase!!-Super-slow&p=28793&viewfull=1#post28793 or are they the same?

 

[qyqgpower]

The tuning option stated in post #7, kern.ipc.nsfbufs, is not needed by 64bit version of FreeNAS.

According to http://www.freebsd.org/cgi/man.cgi?query=sendfile&sektion=2

If a value of zero is reported for kern.ipc.nsfbufs, your architecture does not need to use sendfile() buffers because their task can be efficiently performed by the generic virtual memory structures.

And this is just the case for AMD64 Architecture.

Also, even if you add kern.ipc.nsfbufs to tunable or sysctl, you won't see any changes in sysctl -a | grep kern.ipc.nsfbufs.

 

[toddos]

As long as we're pointing out things that are wrong, the sysprobs link in post #45 is pretty much wrong. It's full of misinformation and half-truths at best, and is a great example of how a seemingly innocuous piece of information (a KB article detailing a valid issue in pre-SP1 Vista that was fixed in a patch and does not affect Windows 7 happened to contain a last resort option that suggested turning off TCP auto tuning if you didn't want to install the fix or try any of the other three workarounds) can be misinterpreted and propagate across the web as if it's valid (google "windows auto tuning" and almost every hit is about how to shut it off, with no more justification about why than that this KB article from 2009 suggested it as a workaround to a bug that's been fixed for years).

And before anybody says, "But the fixes worked for me!" I ask: Did you do them one at a time and check if the problem had gone away, so that you know exactly which fix made a difference? Did you re-enable the setting (if possible -- if clearing DNS cached worked, you can't exactly force that to break again) and verify the problem came back? If not, then you don't know what the problem was, you don't know what fixed it, and while it may be gone now that doesn't mean it won't come back or you didn't break something else in the process. I suspect that the real solution was this:

and excluding my mapped NAS drive from my virus scanner (MS Security Essentials)

I can state for a fact that I have TCP auto tuning enabled, RDC enabled, IPv6 enabled, wireless enabled, etc and everything works just fine. I did do some samba tuning because that's backed up by actual facts and results:

Code:

socket options = TCP_NODELAY IPTOS_LOWDELAY SO_KEEPALIVE SO_RCVBUF=98304 SO_SNDBUF=98304
getwd cache = yes

(obviously buffers should be tweaked for your specific network)

The only time I ever see slowness in directory listings now is when the hard drives in my ZFS pool have to spin up from sleep after not being accessed for a while. And there's nothing that can be done about that except not ever letting the disks go idle.

 

[qyqgpower]

After some research, I suggest SMB2 shoud be enabled if your clients are Windows Vista or later (in case of Mac OSX, 10.7 or later) AND using Samba 3.6 or later.
SMB2 is introduced in Windows Vista and it's performace will be a lot better than old school SMB.

Since FreeNAS 8.2 is shipped with Samba 3.6.5 which is including production level of SMB2 support, just add following line to CIFS's auxiliary parameters:
max protocol = SMB2

 

[Stephens]

Since FreeNAS 8.2 is shipped with Samba 3.6.5 which is including production level of SMB2 support, just add following line to CIFS's auxiliary parameters:
max protocol = SMB2

Nice tip. Thanks.

 

[giox069]

Hi all, I'm new to this thread, but I have the same problem: directory bowsing of a big directory is very slow when accessed via CIFS (via windows client or via commandline smbclient from ubuntu). Here is my configuration:

• The directory is composed of 19300 subdirectories
• Freenas is 8.2.0-RELEASE-p1-x64
• Hardware: HP Proliant Microserver N40L with 8GB RAM

adding "max protocol = SMB2" did not solve the problem.
Browsing the directory takes 5 minutes via CIFS, but it takes only few seconds with ls -la lauchend internally in a ssh console.

I played around with top, truss and zpool iostat.
The only process in freenas doing some activity during a smbclient "dir" is smbd (for a maximum of 2 or 3% of cpu load!). The disk activity LED is always on.

truss -D -p _pid_of_smbd_ reports the following

Code:

0.019411824 stat("Lavori/gamatec 40+15x34 men 80 gns D 01-03-2010",{ mode=drwxrwxr-x ,inode=84164,size=5,blksize=4096 }) = 0 (0x0)
0.128582195 extattr_get_file(0x803cbee50,0x1,0x1692a7b,0x0,0x0,0x0) = 56 (0x38)
0.000044977 extattr_get_file(0x803cbee50,0x1,0x1692a7b,0x7fffffffdae0,0x100,0x0) = 56 (0x38)
0.000011245 fcntl(13,F_SETLKW,0x7fffffffd9f0)	 = 0 (0x0)
0.000010755 fcntl(13,F_SETLKW,0x7fffffffda50)	 = 0 (0x0)
0.305608883 stat("Lavori/lani 32+13x39 jnx milano 20-03-2012",{ mode=drwxrwxrwx ,inode=233551,size=3,blksize=4096 }) = 0 (0x0)
1.183616283 extattr_get_file(0x803cbee50,0x1,0x1692a7b,0x0,0x0,0x0) = 56 (0x38)
0.000062089 extattr_get_file(0x803cbee50,0x1,0x1692a7b,0x7fffffffdae0,0x100,0x0) = 56 (0x38)
0.000020534 fcntl(13,F_SETLKW,0x7fffffffd9f0)	 = 0 (0x0)
0.000011733 fcntl(13,F_SETLKW,0x7fffffffda50)	 = 0 (0x0)
...
0.012985868 stat("Lavori/tavns & co. 25+11x31 kissa 18-06-2007",{ mode=drwxrwxr-x ,inode=46253,size=6,blksize=4096 }) = 0 (0x0)
0.000053778 extattr_get_file(0x803cb0050,0x1,0x1692a7b,0x0,0x0,0x0) = 56 (0x38)
0.000082622 extattr_get_file(0x803cb0050,0x1,0x1692a7b,0x7fffffffdae0,0x100,0x0) = 56 (0x38)
0.000017111 fcntl(13,F_SETLKW,0x7fffffffd9f0)	 = 0 (0x0)
0.000013689 fcntl(13,F_SETLKW,0x7fffffffda50)	 = 0 (0x0)
3.287608551 stat("Lavori/winnen-pa- 41+12x35 jnx maruneo PA 08-03-2010",{ mode=drwxrwxr-x ,inode=2801,size=3,blksize=4096 }) = 0 (0x0)
0.000050355 extattr_get_file(0x803c0a910,0x1,0x1692a7b,0x0,0x0,0x1) = 56 (0x38)
0.000062578 extattr_get_file(0x803c0a910,0x1,0x1692a7b,0x7fffffffdae0,0x100,0x1) = 56 (0x38)
...
0.000088978 stat("Lavori/herzannma 20+10x27 raxele 20-11-2009",{ mode=drwxrwxr-x ,inode=94037,size=4,blksize=4096 }) = 0 (0x0)
0.010090668 extattr_get_file(0x803c9cfd0,0x1,0x1692a7b,0x0,0x0,0x1) = 56 (0x38)
2.044919815 extattr_get_file(0x803c9cfd0,0x1,0x1692a7b,0x7fffffffdae0,0x100,0x1) = 56 (0x38)
0.000035200 fcntl(13,F_SETLKW,0x7fffffffd9f0)	 = 0 (0x0)
0.000019555 fcntl(13,F_SETLKW,0x7fffffffda50)	 = 0 (0x0)

As you can see, there are simple syscall (like "stat" or extattr_get_file) that sometimes takes up to 2-3 seconds for some subdirectories. These subdirs only contains 1 to 5 files, usually large 5 to 10MB each one.
But the most weird thing is that... the system is WRITING to the filesystem at 3MBytes/sec ??? This test is done without any client connected to CIFS or other protocols, and zfs writings starts exactly when I type "dir" on smbclient and stops when dir is done.

Code:

[root@pyprel] ~# zpool iostat 1
               capacity     operations    bandwidth
pool         used  avail   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
Volume1     3.06T  2.37T     42     82   193K   632K
Volume1     3.06T  2.37T      0    525      0  4.09M
Volume1     3.06T  2.37T      7    287  63.9K  1.84M
Volume1     3.06T  2.37T      0    409      0  3.09M
Volume1     3.06T  2.37T      3    480  20.0K  3.71M
Volume1     3.06T  2.37T      0    324      0  2.54M
Volume1     3.06T  2.37T      0    313      0  2.40M
Volume1     3.06T  2.37T      4    321  40.0K  2.11M
Volume1     3.06T  2.37T      0    505      0  3.90M
Volume1     3.06T  2.37T      0    478      0  3.69M
Volume1     3.06T  2.37T      0    294      0  2.25M
Volume1     3.06T  2.37T      0    261      0  2.04M
Volume1     3.06T  2.37T      0    328      0  2.17M
Volume1     3.06T  2.37T      0    322  7.99K  2.36M
Volume1     3.06T  2.37T      0    420      0  3.28M
Volume1     3.06T  2.37T      0    593      0  4.64M

Any idea ?

I hope this could also help in solve other people's directory browsing problems :)


Thank you
Giovanni

EDIT and edit again:
I disabled "Support DOS File Attributes" into the CIFS options and the directory browsing is now VERY FAST. Note that "EA support" was already disabled.
After disabilng "Support DOS File Attributes", I lost all the icons that were associated to some directories: icons are configured in a Desktop.ini file that needs READONLY attribute on the parent dir to be used by explorer.
So the real question are now a bit different
a) why is freenas so slow when "Support DOS File Attributes" is enabled
b) Why zpool iostat reports write activity ?

Thank you again

EDIT/UPDATE:
I removed "atime" from zfs, and now browsing from CIFS is faster again with "Support DOS File Attributes".
So... should I think zfs is not caching atime updates ?

 

[elrelojerociego]

Atime off + DOS attributes = slow listing

I'm trying to figure the same out: slow folder listing when "store DOS attributes" enabled. I saw your last edit and tried to disable atime from either the Pool and the Dataset but I keep having slow listings if I enable the DOS attributes. Did you do anything else appart from disabling acces time? I would like being able to keep the DOS attributes but I don't know where else to look for advice!! (Googling now for hours literally!)

Thanks.

Regards,
Gonzalo.

Hi all, I'm new to this thread, but I have the same problem: directory bowsing of a big directory is very slow when accessed via CIFS (via windows client or via commandline smbclient from ubuntu). Here is my configuration:

• The directory is composed of 19300 subdirectories
• Freenas is 8.2.0-RELEASE-p1-x64
• Hardware: HP Proliant Microserver N40L with 8GB RAM

adding "max protocol = SMB2" did not solve the problem.
Browsing the directory takes 5 minutes via CIFS, but it takes only few seconds with ls -la lauchend internally in a ssh console.

I played around with top, truss and zpool iostat.
The only process in freenas doing some activity during a smbclient "dir" is smbd (for a maximum of 2 or 3% of cpu load!). The disk activity LED is always on.

truss -D -p _pid_of_smbd_ reports the following

Code:

0.019411824 stat("Lavori/gamatec 40+15x34 men 80 gns D 01-03-2010",{ mode=drwxrwxr-x ,inode=84164,size=5,blksize=4096 }) = 0 (0x0)
0.128582195 extattr_get_file(0x803cbee50,0x1,0x1692a7b,0x0,0x0,0x0) = 56 (0x38)
0.000044977 extattr_get_file(0x803cbee50,0x1,0x1692a7b,0x7fffffffdae0,0x100,0x0) = 56 (0x38)
0.000011245 fcntl(13,F_SETLKW,0x7fffffffd9f0)	 = 0 (0x0)
0.000010755 fcntl(13,F_SETLKW,0x7fffffffda50)	 = 0 (0x0)
0.305608883 stat("Lavori/lani 32+13x39 jnx milano 20-03-2012",{ mode=drwxrwxrwx ,inode=233551,size=3,blksize=4096 }) = 0 (0x0)
1.183616283 extattr_get_file(0x803cbee50,0x1,0x1692a7b,0x0,0x0,0x0) = 56 (0x38)
0.000062089 extattr_get_file(0x803cbee50,0x1,0x1692a7b,0x7fffffffdae0,0x100,0x0) = 56 (0x38)
0.000020534 fcntl(13,F_SETLKW,0x7fffffffd9f0)	 = 0 (0x0)
0.000011733 fcntl(13,F_SETLKW,0x7fffffffda50)	 = 0 (0x0)
...
0.012985868 stat("Lavori/tavns & co. 25+11x31 kissa 18-06-2007",{ mode=drwxrwxr-x ,inode=46253,size=6,blksize=4096 }) = 0 (0x0)
0.000053778 extattr_get_file(0x803cb0050,0x1,0x1692a7b,0x0,0x0,0x0) = 56 (0x38)
0.000082622 extattr_get_file(0x803cb0050,0x1,0x1692a7b,0x7fffffffdae0,0x100,0x0) = 56 (0x38)
0.000017111 fcntl(13,F_SETLKW,0x7fffffffd9f0)	 = 0 (0x0)
0.000013689 fcntl(13,F_SETLKW,0x7fffffffda50)	 = 0 (0x0)
3.287608551 stat("Lavori/winnen-pa- 41+12x35 jnx maruneo PA 08-03-2010",{ mode=drwxrwxr-x ,inode=2801,size=3,blksize=4096 }) = 0 (0x0)
0.000050355 extattr_get_file(0x803c0a910,0x1,0x1692a7b,0x0,0x0,0x1) = 56 (0x38)
0.000062578 extattr_get_file(0x803c0a910,0x1,0x1692a7b,0x7fffffffdae0,0x100,0x1) = 56 (0x38)
...
0.000088978 stat("Lavori/herzannma 20+10x27 raxele 20-11-2009",{ mode=drwxrwxr-x ,inode=94037,size=4,blksize=4096 }) = 0 (0x0)
0.010090668 extattr_get_file(0x803c9cfd0,0x1,0x1692a7b,0x0,0x0,0x1) = 56 (0x38)
2.044919815 extattr_get_file(0x803c9cfd0,0x1,0x1692a7b,0x7fffffffdae0,0x100,0x1) = 56 (0x38)
0.000035200 fcntl(13,F_SETLKW,0x7fffffffd9f0)	 = 0 (0x0)
0.000019555 fcntl(13,F_SETLKW,0x7fffffffda50)	 = 0 (0x0)

As you can see, there are simple syscall (like "stat" or extattr_get_file) that sometimes takes up to 2-3 seconds for some subdirectories. These subdirs only contains 1 to 5 files, usually large 5 to 10MB each one.
But the most weird thing is that... the system is WRITING to the filesystem at 3MBytes/sec ??? This test is done without any client connected to CIFS or other protocols, and zfs writings starts exactly when I type "dir" on smbclient and stops when dir is done.

Code:

[root@pyprel] ~# zpool iostat 1
               capacity     operations    bandwidth
pool         used  avail   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
Volume1     3.06T  2.37T     42     82   193K   632K
Volume1     3.06T  2.37T      0    525      0  4.09M
Volume1     3.06T  2.37T      7    287  63.9K  1.84M
Volume1     3.06T  2.37T      0    409      0  3.09M
Volume1     3.06T  2.37T      3    480  20.0K  3.71M
Volume1     3.06T  2.37T      0    324      0  2.54M
Volume1     3.06T  2.37T      0    313      0  2.40M
Volume1     3.06T  2.37T      4    321  40.0K  2.11M
Volume1     3.06T  2.37T      0    505      0  3.90M
Volume1     3.06T  2.37T      0    478      0  3.69M
Volume1     3.06T  2.37T      0    294      0  2.25M
Volume1     3.06T  2.37T      0    261      0  2.04M
Volume1     3.06T  2.37T      0    328      0  2.17M
Volume1     3.06T  2.37T      0    322  7.99K  2.36M
Volume1     3.06T  2.37T      0    420      0  3.28M
Volume1     3.06T  2.37T      0    593      0  4.64M

Any idea ?

I hope this could also help in solve other people's directory browsing problems :)


Thank you
Giovanni

EDIT and edit again:
I disabled "Support DOS File Attributes" into the CIFS options and the directory browsing is now VERY FAST. Note that "EA support" was already disabled.
After disabilng "Support DOS File Attributes", I lost all the icons that were associated to some directories: icons are configured in a Desktop.ini file that needs READONLY attribute on the parent dir to be used by explorer.
So the real question are now a bit different
a) why is freenas so slow when "Support DOS File Attributes" is enabled
b) Why zpool iostat reports write activity ?

Thank you again

EDIT/UPDATE:
I removed "atime" from zfs, and now browsing from CIFS is faster again with "Support DOS File Attributes".
So... should I think zfs is not caching atime updates ?

 

[giox069]

I'm trying to figure the same out: slow folder listing when "store DOS attributes" enabled. I saw your last edit and tried to disable atime from either the Pool and the Dataset but I keep having slow listings if I enable the DOS attributes. Did you do anything else appart from disabling acces time? I would like being able to keep the DOS attributes but I don't know where else to look for advice!! (Googling now for hours literally!)

Hi Gonzalo, I really have no idea of what other parameter may affect your speed. Please note that now my FreeNAS is also running with 6GB of ram. ZFS performs better with more than 4GB.
Here are the settings for "ZFS options" on my dataset:

• Compression Level: off,
• enable atime: Off,
• all the other 4 values are '0'.

On the CIFS global options:

• Auth Model: Local user,
• DOS charset: CP850,
• Unix Charset: UTF-8,
• Log Level: minimum,
• Local Master: Yes,
• Time Server for domain: Yes,
• Guest account: nobody,
• File mask: empty,
• Directory mask: empty,
• Large RW support: off,
• Send files with sendfile(2): Yes,
• EA support: No,
• Support DOS files attributes: Yes,
• Allow empty password: no,
• The rest is empty/unchecked up to:
• Unix Extensions: yes, Enable
• AIO: yes,
• Min AIO read sz: 1
• Max AIO read sz:1,
• Zeroconf share dir: No
• On the CIFS share: only "Browseable to Network Clients" is enabled.

Giovanni

 

[jasonlitka]

I've been fighting this issue all day. When DOS Attributes are enabled directory browsing performance is miserable and someone right-clicking a folder in Windows with a ton of stuff in it, then properties, will result in no one else being able to do anything at any reasonable speed for the hours that task can take to finish. Disabling DOS attributes makes the system as fast as the Windows box it's replacing, but since I need DOS attributes (hidden files/folders) that's not an option. ZFS's atime option is already off and I've done the relevant tweaks from post 7 of this thread.

Any ideas?

EDIT #1: Actually, I may have spoken too soon. It looks like the above problem still exists, even with DOS Attributes off.

EDIT #2: More info. This has nothing to do with CIFS as it happens over SSH as well with a simple ls. If I run "ls -aR" from the root of a dataset I'll see it sit there paused for 2-3 seconds for each folder before displaying the list of 100-200 files in each.

EDIT #3: I put everything back to the defaults for now. I've got an open ticket with iX since this hits both the system I bought from them and the one I built in-house. Until it gets worked out I've got a workaround. I've setup a cron job for every 30 minutes that runs "ls -aR" on the root of the pool to force all the metadata into the ARC. The first run after a startup takes about 25 minutes for a few hundred-thousand files, the others only take about 6 seconds and are just there to keep the data in ARC cached as MFU.

 

[Ravefiend]

Got any luck with iX on this case jasonlitka ? Did they found the root cause?

 

[jasonlitka]

Got any luck with iX on this case jasonlitka ? Did they found the root cause?

I called them off because it happens over SSH as well and may just be a ZFS issue.

My "fix" to the issue is a cron job that runs every 30 minutes to cache all the necessary metadata and keeps it there as frequently used. The first run takes forever because there are 2,150,000 files on that box but each run after that only takes a few seconds because everything is cached.

 

[mikesm]

Hi. I applied the changes from protosd's post #7 in this thread, and it has improved performance, particularly copy performance quite a bit. However, I still sometimes see long delays (20-30 secs) when attempting to browse directories. I don't think this problem is a ZFS problem at all. I logged into the server with the shell, and issued the following command:

time ls -l -R /mnt/* > /dev/null

It took about 5 secs to traverse all the directories with tons of files in them, but I don't see this kind of speed via CIFS browsing. So it appears the issue is somehow related to how samba serves up this directory information to the windows clients, and not the core file system itself.

Given I have applied protosd's tweaks to CIFS already, what more can I do to track down the slowness in browsing large directories?

Thanks!
Mike

 

[ProtoSD]

Hi. I applied the changes from protosd's post #7 in this thread, and it has improved performance, particularly copy performance quite a bit. However, I still sometimes see long delays (20-30 secs) when attempting to browse directories. I don't think this problem is a ZFS problem at all. I logged into the server with the shell, and issued the following command:

time ls -l -R /mnt/* > /dev/null

It took about 5 secs to traverse all the directories with tons of files in them, but I don't see this kind of speed via CIFS browsing. So it appears the issue is somehow related to how samba serves up this directory information to the windows clients, and not the core file system itself.

Given I have applied protosd's tweaks to CIFS already, what more can I do to track down the slowness in browsing large directories?

Thanks!
Mike

Mike, did read through the rest of this thread and try any of the other things people mentioned? Like atime and/or DOS attributes.

I haven't had time to play around with this stuff since I posted it awhile back. It seems like it only happens with certain configurations.

There's also the Performance section of the forums that has some benchmarks and testing that people have done that might give some insight.

 

[cyberjock]

Mikesm,

Try disabling the firewall software on your desktop and see if the delay goes away...