SOLVED Browsing directories slow

[cyberjock]

That registry entry won't do anything if you have Vista or newer.

 

[jasonlitka]

Yeah, doesn't seem to have done so.

Any other suggestions?

 

[cyberjock]

Have you tried Linux to see if it still performs poorly there? You could use a LiveCD just to rule out your client OSes.

 

[AlainD]

Hi

I did some tests :

ran
ls -alR /mnt >/dev/null
several times from the gui shell, after the first time I didn't hear the disks from my freenas box seeking. When I opened a large directory via CIFS from win7 I immediately heard the disks seeking and the count inside win7 explorer goes up, but slowly.
If I change dir inside the explorer to a smaller one, the seeking stops. If I reopen the "big" dir inside explorer it fast jumps to where it was and then start seeking again on the freenas box.

It seems that my win7 wants some attribute that's not cached with "ls -alR".
Has anybody looked at this behaviour?

 

[AlainD]

Hi

See the tread for slow nfs browsing for a "nice" solution to the slow directory browsing :
http://forums.freenas.org/threads/nfs-slow-directory-listing.13653/#post-92029

Alain

 

[mstang1988]

I'm having issues with CIFS directory browsing. I've done a lot of investigation and have the tunables all set, atime off, etc. I'm still having horrible load times of 1 folder. It's around 30 seconds which is unreasonable. I can do a wireshark trace from my windows client if that's helpful.

The folder has 4.9TB with 17500 total nested files and most of the data (4.8TB+ worth) being in 750 of those files. The base directory only has folders in it. I created another dataset to troubleshoot what the issues could be. I created duplicate content except the larger files (.m4v and .mkv files) but did have files named .m4v and .mkv with 0 data (i.e. touch {FILENAME}). The directory listing is not slow in this instance.

The clients I'm listing/browsing it on are XBMC (Linux, I believe Debian based), Windows 7, and my Mac Pro. The Mac is the only wireless one but it is slow on all the clients.

NFS is instantaneous
ls -la > {filename} is instantaneous

I know CIFS should be slower but really, not this slow. Especially as it's only sub folders.

Example of the data dir:
/FOLDER/FOLDER1/Files.Files
/FOLDER/FOLDER2/
/FOLDER/FOLDER3/
/FOLDER/FOLDER4/
/FOLDER/FOLDER5/
...
...
/FOLDER/FOLDER764/

Tunables that are set:
SAMBA COMPLETELY
socket options = TCP_NODELAY IPTOS_LOWDELAY SO_KEEPALIVE
aio write behind=false
read raw = yes
write raw = yes
max xmit = 65536
getwd cache = yes
write cache size = 5124288

INDIVIDUAL SHARE:
locking = no
strict locking = no
oplocks = no
kernel oplocks = no
level2 oplocks = no
fake oplocks = yes

 

[AlainD]

I'm having issues with CIFS directory browsing. I've done a lot of investigation and have the tunables all set, atime off, etc. I'm still having horrible load times of 1 folder. It's around 30 seconds which is unreasonable. I can do a wireshark trace from my windows client if that's helpful.

The folder has 4.9TB with 17500 total nested files and most of the data (4.8TB+ worth) being in 750 of those files. The base directory only has folders in it. I created another dataset to troubleshoot what the issues could be. I created duplicate content except the larger files (.m4v and .mkv files) but did have files named .m4v and .mkv with 0 data (i.e. touch {FILENAME}). The directory listing is not slow in this instance.

The clients I'm listing/browsing it on are XBMC (Linux, I believe Debian based), Windows 7, and my Mac Pro. The Mac is the only wireless one but it is slow on all the clients.
...

Hi

Thanks for trying it on 3 client OS'es. This makes it clear that it's not windows related.

Did you test the "solution" from the link I posted above?

I'm not surprised that the browsing is fast with zero length files, I have the impression that some cifs needed attributes are stored with the file and not in a directory like structure. With zero length files those files are probably "close" to each other on the disks which gives a lot less seek times.

Alain

 

[cyberjock]

I'm not sure what your exact problem is, but I've been in and out of this thread over the years.. Few tidbits I've learned from dealing with paying customers that wanted this problem solved:

1. They often messed with settings they didn't understand and the cause of their problems was their custom settings.
2. They changed CIFS settings without a *deep* level understanding. Hint: All of those settings you are using are not exactly applicable to Samba4(and if not all then at least 90%). So I'd recommend you delete all of your settings and use the defaults that FreeNAS uses.
3. The defaults really do work great. I keep trying to tell people don't mess with stuff if it isn't broken, but people still keep trying to optimize what isn't broken, then are shocked when things end up broken.
4. If your hardware isn't appropriate for your server you will have serious performance problems with how Samba and ZFS relate to each other.

Additionally, you made no comment as to what FreeNAS version you are using nor did you include the hardware you are using, so I'm not much more help than that.

Good luck!

 

[mstang1988]

I'm not sure what your exact problem is, but I've been in and out of this thread over the years.. Few tidbits I've learned from dealing with paying customers that wanted this problem solved:

1. They often messed with settings they didn't understand and the cause of their problems was their custom settings.
2. They changed CIFS settings without a *deep* level understanding. Hint: All of those settings you are using are not exactly applicable to Samba4(and if not all then at least 90%). So I'd recommend you delete all of your settings and use the defaults that FreeNAS uses.
3. The defaults really do work great. I keep trying to tell people don't mess with stuff if it isn't broken, but people still keep trying to optimize what isn't broken, then are shocked when things end up broken.
4. If your hardware isn't appropriate for your server you will have serious performance problems with how Samba and ZFS relate to each other.

Additionally, you made no comment as to what FreeNAS version you are using nor did you include the hardware you are using, so I'm not much more help than that.

Good luck!

cyberjock
1. I understand the settings and the problem persists with/without the settings.
2/3. I'm using 9.2.0 so it's Samba 3.X and those settings are applicable. I tried 9.2.1.4 as well with Samba 4.X and the issue persists on a clean install without the settings.
4. 4 x 3TB Seagate 7200.14 drives with 16GB RAM, Intel CT NIC, i5-2400 (yes, no ECC RAM and I'm working on fixing this with a new build). I am not hardware bound and in a home setting I'm following the rule of >1GB per 1TB Raw

What seems odd to me and what I've never quite understood is that it seems Samba recursively populates directories or potentially the CIFS client is recursively requesting all children's children. I'll have to look at the Samba source to understand why it does this or take wireshark traces and see if it's client/server requesting the recursion. Stopping it from traversing the recursive dir's would solve the problem as I can populate a single Max 50GB subdir (one of the 750ish) quickly. The fact that 1 folder with only 762 folder children (NO FILES) takes 30 seconds to populate with those 762 folders have large children points to this. With small children it's instantaneous to the user.

I would agree with AlainD that Samba must be using attributes from each file or perhaps all the clients are. In either case it does bog down the browsing.

Just for your FYI, I do have an extensive background in Unix (A former programmer on a commercial *IX that I left about 1 month ago after 10 years).

 

[marian78]

Hi, today i switched in "CIFS settings" - "Server maximum protocol" from SMB3 to SMB2, for testing only. But after that, i cant stop "samba_server" from freenas UI. CIFS service buton i still on (blue), with report "The service could not be stopped". But in SSH i can stop it and restart it without problem. Dont helped switch again to SMB3...... Have any ideas whats wrong?

 

[glipman]

Hi, today i switched in "CIFS settings" - "Server maximum protocol" from SMB3 to SMB2, for testing only. But after that, i cant stop "samba_server" from freenas UI. CIFS service buton i still on (blue), with report "The service could not be stopped". But in SSH i can stop it and restart it without problem. Dont helped switch again to SMB3...... Have any ideas whats wrong?

This is a known bug, see https://bugs.freenas.org/issues/4874

 

[marian78]

Thx for your fast reply, now i know, that im not alone. :(

edit: hot patch is working for me, but need restart manualy server.

 

[cyberjock]

marian78,

Don't take this as a personal attack, but I've found that programmers are the worst for breaking some of the FreeNAS taboos. They always seem convinced that tweaking software is the way to go. I'm not a programmer so I don't try to understand why. Maybe it's related to the average programmers thought process, I don't know.

Most of those settings mean almost nothing on all FreeNAS installs with 9.x. Samba4 just makes some of them obsolete.

SMB3 is only supported in Samba4 afaik, so enabling it is going to be..... bizaare. (This kind of goes with what I said that the settings you are using aren't really applicable for your version of Samba). For example, the following parameters shouldn't be set with any modern version of Samba...

socket options = TCP_NODELAY IPTOS_LOWDELAY SO_KEEPALIVE
read raw = yes
write raw = yes
max xmit = 65536
write cache size = 5124288

As for the others, I'm not sure on them. I'm pretty sure getwd cache gains you nothing.

Anyway, sorry I'm not much more help. Some users have installed software on their client machines that have gone searching through subdirectories for thumbnails and such, and that may be related to your problem. I don't know. There's so many ways for the problem to be on the server, the client, or both that it's hard to narrow down without troubleshooting.

 

[mstang1988]

marian78,

Don't take this as a personal attack, but I've found that programmers are the worst for breaking some of the FreeNAS taboos. They always seem convinced that tweaking software is the way to go. I'm not a programmer so I don't try to understand why. Maybe it's related to the average programmers thought process, I don't know.

Most of those settings mean almost nothing on all FreeNAS installs with 9.x. Samba4 just makes some of them obsolete.

SMB3 is only supported in Samba4 afaik, so enabling it is going to be..... bizaare. (This kind of goes with what I said that the settings you are using aren't really applicable for your version of Samba). For example, the following parameters shouldn't be set with any modern version of Samba...

socket options = TCP_NODELAY IPTOS_LOWDELAY SO_KEEPALIVE
read raw = yes
write raw = yes
max xmit = 65536
write cache size = 5124288

As for the others, I'm not sure on them. I'm pretty sure getwd cache gains you nothing.

Anyway, sorry I'm not much more help. Some users have installed software on their client machines that have gone searching through subdirectories for thumbnails and such, and that may be related to your problem. I don't know. There's so many ways for the problem to be on the server, the client, or both that it's hard to narrow down without troubleshooting.

I think you might have gotten me and marian78 confused. You are right, programmers are awful about breaking things, mostly because if it's documented to work it should work. I was on the receiving end of that multiple times.

 

[cyberjock]

Haha. I did!

I have a few programmer friends. I love them to death, but they call me to fix some of their hardware related stuff because they can't do it themselves. I also call them when I need programming help(for example, some stupid simple app I can't find elsewhere).

One of my friends is a FreeBSD developer. He's awesome! Without his help I could have never gotten the minecraft guide working. I'm not sure he could get a working high end ZFS system for his life, but I can't program for my life either. So he calls me up with hardware questions with relation to ZFS and I ask him FreeBSD questions. ;)

It's not necessary to know everything about this stuff. But if you know enough to be useful to others and know who to ask for help when you need it, that can be worth gold.

Networking at it's finest!

 

[mstang1988]

Haha. I did!

I have a few programmer friends. I love them to death, but they call me to fix some of their hardware related stuff because they can't do it themselves. I also call them when I need programming help(for example, some stupid simple app I can't find elsewhere).

One of my friends is a FreeBSD developer. He's awesome! Without his help I could have never gotten the minecraft guide working. I'm not sure he could get a working high end ZFS system for his life, but I can't program for my life either. So he calls me up with hardware questions with relation to ZFS and I ask him FreeBSD questions. ;)

It's not necessary to know everything about this stuff. But if you know enough to be useful to others and know who to ask for help when you need it, that can be worth gold.

Networking at it's finest!

Haha, yes, programmers in general are not good with hardware but not always the case. I worked on hardware diagnostics in my career, some network device driver and virtualization experience, and some experience with Samba many years ago.

 

[marian78]

marian78,

Don't take this as a personal attack, but I've found that programmers are the worst for breaking some of the FreeNAS taboos. They always seem convinced that tweaking software is the way to go. I'm not a programmer so I don't try to understand why. Maybe it's related to the average programmers thought process, I don't know.

Most of those settings mean almost nothing on all FreeNAS installs with 9.x. Samba4 just makes some of them obsolete.

SMB3 is only supported in Samba4 afaik, so enabling it is going to be..... bizaare. (This kind of goes with what I said that the settings you are using aren't really applicable for your version of Samba). For example, the following parameters shouldn't be set with any modern version of Samba...

socket options = TCP_NODELAY IPTOS_LOWDELAY SO_KEEPALIVE
read raw = yes
write raw = yes
max xmit = 65536
write cache size = 5124288

As for the others, I'm not sure on them. I'm pretty sure getwd cache gains you nothing.

Anyway, sorry I'm not much more help. Some users have installed software on their client machines that have gone searching through subdirectories for thumbnails and such, and that may be related to your problem. I don't know. There's so many ways for the problem to be on the server, the client, or both that it's hard to narrow down without troubleshooting.

Cyberjock, don't take this as a personal attack, i am not a programmer, but also i am not sheep. I wanted only TEST some combination of settings and reproduce problem on my server. My fault was, that i not searching my problem with samba service button in bugs (maybe i was lazy in that evening).

 

[mstang1988]

Cyberjock, don't take this as a personal attack, i am not a programmer, but also i am not sheep. I wanted only TEST some combination of settings and reproduce problem on my server. My fault was, that i not searching my problem with samba service button in bugs (maybe i was lazy in that evening).

He got the wrong person, he was trying to talk to me. I was the programmer he was referencing

 

[marian78]

ok, :). I want to only learn and then test what i learned. I like it this way. But, sometime i test something before learn. :D

 

[c0re]

cyberjock
1. I understand the settings and the problem persists with/without the settings.
2/3. I'm using 9.2.0 so it's Samba 3.X and those settings are applicable. I tried 9.2.1.4 as well with Samba 4.X and the issue persists on a clean install without the settings.
4. 4 x 3TB Seagate 7200.14 drives with 16GB RAM, Intel CT NIC, i5-2400 (yes, no ECC RAM and I'm working on fixing this with a new build). I am not hardware bound and in a home setting I'm following the rule of >1GB per 1TB Raw

What seems odd to me and what I've never quite understood is that it seems Samba recursively populates directories or potentially the CIFS client is recursively requesting all children's children. I'll have to look at the Samba source to understand why it does this or take wireshark traces and see if it's client/server requesting the recursion. Stopping it from traversing the recursive dir's would solve the problem as I can populate a single Max 50GB subdir (one of the 750ish) quickly. The fact that 1 folder with only 762 folder children (NO FILES) takes 30 seconds to populate with those 762 folders have large children points to this. With small children it's instantaneous to the user.

I would agree with AlainD that Samba must be using attributes from each file or perhaps all the clients are. In either case it does bog down the browsing.

Just for your FYI, I do have an extensive background in Unix (A former programmer on a commercial *IX that I left about 1 month ago after 10 years).

Hi mstang1988,

It's interesting that you've been experiencing this as well because your configuration is almost identical to mine (and I have the EXACT same hard drives). I've also tried configuring FreeNAS from scratch and have poor performance with CIFS directory browsing be it from Windows or Samba in Linux.

In my case, I have 5x3TB Seagate 7200.14 drives with 16GB of RAM and the same Intel NIC, but a Pentium G2120.

If you haven't been able to resolve this issue yet, can you try checking the property sheet of a large directory (because it tries to show you a count of all files in the directory which can be VERY slow). In my case, I've noticed that if I go to "Display System Processes" then I get something like this:

Code:

 PID USERNAME    THR PRI NICE  SIZE    RES STATE  C  TIME  WCPU COMMAND
15254 root          1  20    0  335M 16888K zio->i  1  3:10  7.46% smbd

It's always the same story, my directory browsing gets slow whenever smbd gets stuck in the zio->i state. Since we have the exact same HDD model, it makes me wonder if perhaps these particular drives don't like being in RAID arrays or zpools. I tried flashing the firmware on the drive to a newer version just today actually because of a bug with these disks in RAID arrays that Seagate supposedly resolved, but it made no difference.

In any case, when you get a chance, please check your "Display System Processes" because maybe we are having the same issue...