Slow, terribly slow smb in directory with multiple files (+30.000 files)

Status
Not open for further replies.

virusbcn

Explorer
Joined
Apr 22, 2016
Messages
92
Hello, i have a new HP Microserver with G1610T @ 2.30GHz, 8 Gb RAM, 4 WD RED 3Tb configured in ZFS1, and 2 NICs gigabit configured in LAGG

I use it basically as a file server backup Windows servers (NAS) on a little business, the performance with big files (100Gb) is perfect, saturating the network gigabit, with "normal" file directory i have a "normal" performance (aprox. 200/400 Mbps) but when i have to backup a directory that contains 30.000 files (from 5Kb to 1.000Kb) of document management program of the business, the backup drops to 100 Kbps. My software backup tells me need 30 days to backup this :-( ..... and meantime copying this directory all backups lower to this speed until i pause this and recover your normal speed (200/400Mbps through SMB in a real life with tons of files and directories)

Yes the CPU of freenas goes to 50% when backup this, i know this is a low cpu but when i copy to another nas with a cpu very low it copy more fast.

I search in this forum about tunning SMB but i don't have luck, because this i question here, how can i make to improve performance of my NAS ???

Maybe tunning ZFS ???
 

virusbcn

Explorer
Joined
Apr 22, 2016
Messages
92
This is the top info meanwhile copy this horrendous directory



login as: root
root@192.168.x.x's password:
Last login: Wed May 25 19:16:23 2016 from 192.168.x.x
FreeBSD 10.3-STABLE (FreeNAS.amd64) #0 r295946+0ea9076(9.10-STABLE): Mon Oct 3 06:51:23 UTC 2016

FreeNAS (c) 2009-2016, The FreeNAS Development Team
All rights reserved.
FreeNAS is released under the modified BSD license.

For more information, documentation, help or support, go here:
http://freenas.org
Welcome to FreeNAS
[root@freenas] ~# top
last pid: 72772; load averages: 1.14, 1.11, 0.98 up 1+17:06:55 12:16:10
44 processes: 2 running, 42 sleeping
CPU: 49.4% user, 0.0% nice, 0.8% system, 0.0% interrupt, 49.8% idle
Mem: 81M Active, 543M Inact, 5591M Wired, 1700M Free
ARC: 4799M Total, 459M MFU, 3885M MRU, 800K Anon, 40M Header, 415M Other
Swap: 8192M Total, 8192M Free

PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
68928 root 1 103 0 354M 47968K CPU0 0 35:12 100.00% smbd
2404 root 1 -52 r0 6300K 2272K nanslp 1 7:39 0.10% watchdogd
4655 root 6 20 0 398M 184M select 1 10:46 0.00% python2.7
4957 root 12 20 0 221M 20232K nanslp 0 5:45 0.00% collectd
10452 root 1 20 0 282M 28200K select 0 2:34 0.00% smbd
4630 root 1 52 0 217M 56992K select 0 1:27 0.00% python2.7
13543 root 2 20 0 111M 23316K select 0 0:30 0.00% python2.7
6837 root 6 20 0 117M 24196K select 0 0:11 0.00% python2.7
1886 root 2 20 0 75164K 9368K kqread 1 0:06 0.00% syslog-ng
2720 root 1 20 0 36492K 18212K select 1 0:05 0.00% ntpd
30485 www 1 20 0 40352K 6632K kqread 0 0:03 0.00% nginx
10454 root 1 20 0 403M 29016K select 0 0:01 0.00% winbindd
10442 root 1 20 0 229M 20912K select 1 0:01 0.00% nmbd
7401 root 1 52 0 167M 48420K ttyin 0 0:01 0.00% python2.7
7349 root 1 52 0 29172K 2616K nanslp 0 0:01 0.00% cron
11661 root 4 52 0 169M 48792K usem 1 0:01 0.00% python2.7
 

virusbcn

Explorer
Joined
Apr 22, 2016
Messages
92
Thank you Indy, yes i have disabled DOS attributes, i show you my SMB auxiliary parameters:

socket options = TCP_NODELAY
aio read size = 16384
aio write size = 16384
ea support = no
store dos attributes = no
map archive = no
map hidden = no
map readonly = no
map system = no
veto files = /Thumbs.db/Temporary Items/.DS_Store/.AppleDB/.TemporaryItems/.AppleDouble/.bin/.AppleDesktop/Network Trash Folder/.Spotlight/.Trashes/.fseventd/
delete veto files = yes
hide dot files = yes
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
There is no such thing as "ZFS1". I don't know where this one came from and why it's so popular, but it's wrong. You probably mean RAIDZ1.

2 NICs gigabit configured in LAGG
Don't.

Yes the CPU of freenas goes to 50% when backup this, i know this is a low cpu but when i copy to another nas with a cpu very low it copy more fast.
Well, there's your problem. It's a very low-end and low-TDP CPU. ZFS is resource-intensive, SMB is a bit, too. If you throw a bunch of tiny files at something, whatever it is, it's going to grind to a crawl - you just saw firsthand how bad it can be.

Easy workaround is to tarball all your tiny files into manageable chunks.
 

virusbcn

Explorer
Joined
Apr 22, 2016
Messages
92
There is no such thing as "ZFS1". I don't know where this one came from and why it's so popular, but it's wrong. You probably mean RAIDZ1.


Don't.


Well, there's your problem. It's a very low-end and low-TDP CPU. ZFS is resource-intensive, SMB is a bit, too. If you throw a bunch of tiny files at something, whatever it is, it's going to grind to a crawl - you just saw firsthand how bad it can be.

Easy workaround is to tarball all your tiny files into manageable chunks.


Yes i have a mistake is RaidZ1, the network is:

lagg0 (loadbalance:bge1,bge0)

And its working because one PC saturate one NIC and another PC saturate another NIC

The CPU, yes its a low end but the CPU of my another NAS its more low and i have better performance, and the result of the top you see only smbd process is working all cpu 100x100, perhaps maybe something can be optimized, i not view any high process about ZFS in top, do you view the top i post before ???
 

virusbcn

Explorer
Joined
Apr 22, 2016
Messages
92
I'm sorry, this directory haves 106.000 files and can not change anything because its controlled by a business program

The CPU is the same for other directories and the performance is good about 200 Mbps but when i have to write in a directory with more than 10.000 files the smbd process is hangs to 100x100
 
Last edited:

MrToddsFriends

Documentation Browser
Joined
Jan 12, 2015
Messages
1,338
The default (share specific) samba setting case sensitive = auto forces samba to compare the name of each new file with all filenames in that directory, which is very CPU intensive in folders with many files.

Explicitly setting case sensitive = yes prevents this name mangling burden, but I can't say if this setting would have unwanted side effects for you.

https://forums.freebsd.org/threads/49732/#post-278735

Changing this setting should affect the performance when creating new files, but not when modifying existing files.

A while back I performed some tests with several thousand small (4 kBytes) files in a single directory, but I kept the default setting because this is not a regular use case for me.
 

virusbcn

Explorer
Joined
Apr 22, 2016
Messages
92
The default (share specific) samba setting case sensitive = auto forces samba to compare the name of each new file with all filenames in that directory, which is very CPU intensive in folders with many files.

Explicitly setting case sensitive = yes prevents this name mangling burden, but I can't say if this setting would have unwanted side effects for you.

https://forums.freebsd.org/threads/49732/#post-278735

Changing this setting should affect the performance when creating new files, but not when modifying existing files.

A while back I performed some tests with several thousand small (4 kBytes) files in a single directory, but I kept the default setting because this is not a regular use case for me.


Thank you, the performance has improved a lot, later i make a tests but now the freenas support backup copys of two servers with 200+200 Mbps in each NIC, before when i make the copy of this directory all i have is 100/200Kbps and smbd to 100%
 

virusbcn

Explorer
Joined
Apr 22, 2016
Messages
92
There is no such thing as "ZFS1". I don't know where this one came from and why it's so popular, but it's wrong. You probably mean RAIDZ1.


Don't.


Well, there's your problem. It's a very low-end and low-TDP CPU. ZFS is resource-intensive, SMB is a bit, too. If you throw a bunch of tiny files at something, whatever it is, it's going to grind to a crawl - you just saw firsthand how bad it can be.

Easy workaround is to tarball all your tiny files into manageable chunks.


I make a test in another FreeNAS i have, i5-2390T CPU @ 2.70GHz, 8 Gb RAM, 4 WD RED 3 Tb with an HP420i card with 512Mb flash, i not have RaidZ1, is controlled by card, the directory have 46.000 little files about 10-500 Kb and i copy from my PC Win10, I5-4440, SSD Sandisk to 2 diferent servers, the FreeNAS and a Raspberry PI3 i have with 2.5" 1 Tb attached, both copy through SMB

The Raspberry IS MORE FAST THAN A FREENAS CORE i5 O:O 3MB/s -> 200KB/s !!!!!


Can anybody tell me because FreeNAS is more, more slow than a Raspi3 ????
 

Attachments

  • freenas.PNG
    freenas.PNG
    59.3 KB · Views: 957

MrToddsFriends

Documentation Browser
Joined
Jan 12, 2015
Messages
1,338
The Raspberry IS MORE FAST THAN A FREENAS CORE i5 O:O 3MB/s -> 200KB/s !!!!!

I would be interested in
  • the samba version in use on both platforms
  • a comparison of smb.conf files
  • a comparison of the CPU load caused by samba
A closer look to samba surely won't hurt as you saw approx. 100% WCPU for smbd in both FreeNAS installations (the HP Microserver and the i5).
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
4 WD RED 3 Tb with an HP420i card with 512Mb flash, i not have RaidZ1, is controlled by card
That's just going to make things even slower. At the very best, the whole stack gains latency, which is going to kill performance with many small files.
 

virusbcn

Explorer
Joined
Apr 22, 2016
Messages
92
That's just going to make things even slower. At the very best, the whole stack gains latency, which is going to kill performance with many small files.

Sorry, it is not my intention to offend you, but I think you are not being objective or constructive

Two Freenas with diferent hardware has the same poor result, 100/200Kbps and a poor Raspberry with a poor USB disk have 3/4 Mbps , evidently have a freenas configuration problem
 

virusbcn

Explorer
Joined
Apr 22, 2016
Messages
92
I would be interested in
  • the samba version in use on both platforms
  • a comparison of smb.conf files
  • a comparison of the CPU load caused by samba
A closer look to samba surely won't hurt as you saw approx. 100% WCPU for smbd in both FreeNAS installations (the HP Microserver and the i5).


Yes i'm sure too that the problem is in smb.conf, two freenas have a process smb to 100% cpu, and if you see the screens attached you can view the top is freenas and the down is Raspberry, and Raspi is stable in time, but freenas starts good but when have copied about 5.000 files go down, and from 10.000 goes down to 100/200 kbps

This is the smb.conf of the raspberry:

config file = /etc/samba/smb-local.conf

workgroup = WORKGROUP
security=user
follow symlinks = yes
wide links = no
unix extensions = no
lock directory = /var/cache/samba
load printers = no
printing = bsd
printcap name = /dev/null
disable spoolss = yes
security=user
follow symlinks = yes
wide links = no
unix extensions = no
lock directory = /var/cache/samba
load printers = no
printing = bsd
printcap name = /dev/null
disable spoolss = yes
log level = 1
map to guest = bad user
usershare template share = automount template

read raw = Yes
write raw = Yes
strict locking = no
min receivefile size = 16384
use sendfile = true
aio read size = 2048
map to guest = bad user
usershare template share = automount template

read raw = Yes
write raw = Yes
strict locking = no
min receivefile size = 16384
use sendfile = true
aio read size = 2048
aio write size = 2048
socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=131072 SO_SNDBUF=1310$

[osmc]
browsable = yes
read only = no
valid users = osmc
path = /home/osmc
comment = OSMC Home Directory
socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=131072 SO_SNDBUF=1310$

[osmc]
browsable = yes
read only = no
valid users = osmc
path = /home/osmc
comment = OSMC Home Directory

[automount template]
browseable = yes
-valid = no
valid users = osmc
path = %P
hide files = /$RECYCLE.BIN/System Volume Information/desktop.ini/thumbs.db/
 

SweetAndLow

Sweet'NASty
Joined
Nov 6, 2013
Messages
6,421
I make a test in another FreeNAS i have, i5-2390T CPU @ 2.70GHz, 8 Gb RAM, 4 WD RED 3 Tb with an HP420i card with 512Mb flash, i not have RaidZ1, is controlled by card, the directory have 46.000 little files about 10-500 Kb and i copy from my PC Win10, I5-4440, SSD Sandisk to 2 diferent servers, the FreeNAS and a Raspberry PI3 i have with 2.5" 1 Tb attached, both copy through SMB

The Raspberry IS MORE FAST THAN A FREENAS CORE i5 O:O 3MB/s -> 200KB/s !!!!!


Can anybody tell me because FreeNAS is more, more slow than a Raspi3 ????
You have a terrible setup and should start by reading all the threads in my signature. Your problem is your silly raid card and the fact that you are using raid.

Sent from my Nexus 5X using Tapatalk
 

MrToddsFriends

Documentation Browser
Joined
Jan 12, 2015
Messages
1,338
Yes i'm sure too that the problem is in smb.conf, two freenas have a process smb to 100% cpu, and if you see the screens attached you can view the top is freenas and the down is Raspberry, and Raspi is stable in time, but freenas starts good but when have copied about 5.000 files go down, and from 10.000 goes down to 100/200 kbps

Decreasing write rate over time (with default setting case sensitive = auto) vs. constant write rate (with case sensitive = yes) was the difference in behavior I was observing during my experiments when copying thousands of small new files to a SMB share, all within the same folder. What is your setting on the i5?

Also you forgot to mention the samba version on your raspi ( smbstatus --version). This could be important as the implementation of some features as well as the default values for some options might differ between different samba versions. I'm not s samba expert, though, so hoping for others that could provide educated guesses. What's not evident from your screen shot is the CPU load caused by samba on the rapsi.
 

virusbcn

Explorer
Joined
Apr 22, 2016
Messages
92
You have a terrible setup and should start by reading all the threads in my signature. Your problem is your silly raid card and the fact that you are using raid.

Sent from my Nexus 5X using Tapatalk

Thank you in your signature i not view anything about SMB, please can you be more exactly ?
If you would to make tests in two diferente Freenas to view real performance, please say me the commands that you would to make and i show you the output test to compare with u

Please, read good the posts, i make the tests in two diferent freenas, the first post not say nothing about raid card and the result is the same
 
Last edited:

virusbcn

Explorer
Joined
Apr 22, 2016
Messages
92
Decreasing write rate over time (with default setting case sensitive = auto) vs. constant write rate (with case sensitive = yes) was the difference in behavior I was observing during my experiments when copying thousands of small new files to a SMB share, all within the same folder. What is your setting on the i5?

Also you forgot to mention the samba version on your raspi ( smbstatus --version). This could be important as the implementation of some features as well as the default values for some options might differ between different samba versions. I'm not s samba expert, though, so hoping for others that could provide educated guesses. What's not evident from your screen shot is the CPU load caused by samba on the rapsi.

Thank you MrTodds, this afternoon i think i have time to make tests and show you, in the freenas I5 the config is by default, and in the first freenas i try a lot of configs.
The speedy test of pause copy, change case sensitive and restart the copy have a very good performance than the default config
 

virusbcn

Explorer
Joined
Apr 22, 2016
Messages
92
All those who say that the problem is the hardware, please remember that I tried on two Freenas with different configurations, both in normal situations copy, saturating the gigabit network, even in large files get saturated the two gigabit cards in the same time, but performance 100Kb / s is not acceptable even for a Pentium 2, so the Raspberry goes faster.
I've been in the computer for almost 30 years, working with Unix was normal to work with this type of directories with so many files and SCSI and their controlling drives and gave more performance than those 100Kb / s, the first Freenas that I rode will 15 years with IDE disks and P3 also gave this performance.
The truth I refuse to believe that with hardware today, can not get more than 100Kb / s even a "strange" directory
 

virusbcn

Explorer
Joined
Apr 22, 2016
Messages
92
Decreasing write rate over time (with default setting case sensitive = auto) vs. constant write rate (with case sensitive = yes) was the difference in behavior I was observing during my experiments when copying thousands of small new files to a SMB share, all within the same folder. What is your setting on the i5?

Also you forgot to mention the samba version on your raspi ( smbstatus --version). This could be important as the implementation of some features as well as the default values for some options might differ between different samba versions. I'm not s samba expert, though, so hoping for others that could provide educated guesses. What's not evident from your screen shot is the CPU load caused by samba on the rapsi.

In Raspberry the version of SMB is 4.2.10 Debian and in FreeNAS is 4.3.11 git-unknow , this is the status of Raspberry CPU when i copy this directory with many files, it's only 50/60% one core the little cpu of Raspi3


upload_2016-10-13_10-32-43.png
 
Status
Not open for further replies.
Top