Performance 9.3 vs 9.10 (folder browsing) + Linux

[pernils]

I have done some folder browsing test and also some huge file copy.
The hardware used for the test is as below.


WIN 2008 r2
Intel Xeon E5507 2.27 GHz
8 GB memory
Hardware Raid1 SCSI disk 10k rpm

FreeNAS Mini (Asrock C2750D4I)
32GB memory
Raid 10 (4* 4TB WD Red)

FreeNAS Backup
8 GB memory
Raid1 (2* 9 TB WD red)
Athlon AMD 64 4200+ 2.2 GHz

Proxmox 3.2
7GB memory (only 1024 MB is designated for the container)
Single old spinning disk
Athlon AMD 64 4200+ 2.2 GHz (same old hardware as FreeNAS Backup but with slower disk)

Client : HP Z600 workstation
12 GB memory
SSD 120GB
Win 7

FreeNAS versions :
9.3
9.10.1-U2

Proxmox 3.2
Using here a openvz container for my test. Witch in it's turn adds a bit overhead. The Linux kernel should be based on RHEL7x
Samba version 3.6.6
File system is ext3


Scripts.

For making 50k+ files

Code:

#!/bin/bash
for n in {1..50000}
do
touch $n;
done

Folder browsing script (obs windows bat file)

Code:

@echo off
REM http://stackoverflow.com/questions/9922498/calculate-time-difference-in-windows-batch-file
cls

CALL :GetStartTime

rem Any process here...

CALL :GetStartTime
CALL :GetEndTime

for /l %%i in (1, 1, 3) do (
set /A newstart=end
CALL :GetEndTime
dir manyfiles >NUL
set /A elapsed=end-newstart
echo For run %%i
CALL :ShowTime

set cnt=%%i
)

echo .............
echo .............

echo for %cnt% iterations:
echo total time :
set /A elapsed=end-start
CALL :ShowTime


echo .............
echo average time :
set /A elapsed=(end-start)/cnt
CALL :ShowTime

::
:: End main program
::

GOTO End

REM *** Get end time:
:GetEndTime
for /F "tokens=1-4 delims=:.," %%a in ("%time%") do (
set /A "end=(((%%a*60)+1%%b %% 100)*60+1%%c %% 100)*100+1%%d %% 100"
)

GOTO :EOF

REM *** Get start Time ***
:GetStartTime
rem Get start time:
for /F "tokens=1-4 delims=:.," %%a in ("%time%") do (
set /A "start=(((%%a*60)+1%%b %% 100)*60+1%%c %% 100)*100+1%%d %% 100"
)

GOTO :EOF

REM *** Show elapsed time: ***
:ShowTime
set /A hh=elapsed/(60*60*100), rest=elapsed%%(60*60*100), mm=rest/(60*100), rest%%=60*100, ss=rest/100, cc=rest%%100
if %mm% lss 10 set mm=0%mm%
if %ss% lss 10 set ss=0%ss%
if %cc% lss 10 set cc=0%cc%
::echo %hh%:%mm%:%ss%,%cc%
echo %ss%,%cc%
GOTO :EOF

:End
pause

Iperf
First I measured the network with iperf. Iperf version 2.09 is used on the windows client and WIN2008. Iperf 2.0.5 is used on the *nix boxes.

For server side :

Code:

iperf -s

For client side :

Code:

iperf -c 192.168.100.2 .. and so on

All three servers (FreeNAS mini, FreeNAS backup, WIN 2008) clocked in at 860 MBits/sec. The Openvz container on the Proxmox box was this day a fraction slower 800 MBits/sec. This box sits next to its brother FreeNAS backup box.

Preparation :

On each box 2 folders is generated.
Folder 1 50k files is added.
Folder 2 will hold a huge file 10.5 Gb for copy testing.

Each folder listening is done with the same bat file. For BSD boxes cpu core is monitored with top -P


Win 2008 r2
1,1 sec folder browsing script
58 MB/sec copy huge file from server
94 MB/sec copy to server

FreeNAS mini 9.10
17,8 sec folder browsing script (87% utilization on single core)
18 – 30 sec while jumping in and out with windows file explorer.

FreeNAS backup 9.10
28.20 sec for folder browsing script (75% utilization on single core)
28 sec more or less steady while jumping in and out with windows file explorer.

proxmox 3.2
1,9 sec folder browsing script
85 MB/sec copy huge file from server
55 MB/sec copy huge file to server



After adding auxiliary parameter on shares ...

Code:

store dos attributes = no
ea support = no
map archive = no
map hidden = no
map system = no
map readonly = no


case sensitive = true
default case = lower
preserve case = no
short preserve case = no

FreeNAS mini 9.10
17.3 sec folder browsing script
89 MB/sec copy huge file from server
98 MB/sec copy huge file to server

FreeNAS backup 9.10
27.9 sec folder browsing script
95 MB/sec copy huge from server
98 MB/sec copy huge to server

Not much of a change ...
Booted up the FreeNAS mini into 9.3

FreeNAS mini 9.3
3.6 sec for folder browsing script
110 MB/sec for copy huge file from server
85 MB/sec for copy huge file to server


Summary :

9.3 outperform 9.10 when it comes to folder browsing. But windows will even be around 3 times faster then 9.3. Proxmox (based on debian) will be between 9.3 and win2008.
Yes the test is done over different hardware, but the win2008 box i several years old. And still it doing better on this topic then the samba shares.

If we sort them from fast to slow.

win2008 (ntfs) 1,1 sec
Proxmox 3.2 (ext3) 1,9 sec
FreeNAS 9,3 (zfs) 3,6 sec
FreeNAS 9.10 (zfs) 17,3 sec



What is your measurements from my scripts ?

For easy mimic my result make this folder structure.

Code:

-
|--manyfiles -| (add 50k+ files in this folder)		
|-measure.bat

In measure.bat paste the folder browse script code above.

From your win client map the share to some letter and just hit the bat file. If you want to have the bat file locally just change the line dir manyfiles >NUL to suite your needs.

One ting is odd thought. When running my browsing script on Proxmox 3.2 it miss its first pass. Se example output bellow.

Code:

For run 1
00,01
For run 2
02,01
For run 3
02,01
.............
.............
for 3 iterations:
total time :
04,03
.............
average time :
01,34
Press any key to continue . . .

This is not the case with FreeNAS ZFS. Se bellow

Code:

For run 1
03,44
For run 2
03,50
For run 3
03,50
.............
.............
for 3 iterations:
total time :
10,45
.............
average time :
03,48
Press any key to continue . . .

 

[bigphil]

Freenas versions :
9.3
9.10.1-U2

Did you ever test FreeNAS 9.10.2-U1?

 

[anodos]

Just being pedantic, but running dir on a bunch of files isn't exactly the same thing as browsing a share. You'll also want to clear out your arc before running the tests to compare 9.3 with 9.10. Having some of the share already cached might make 9.3 look faster.

I'm a bit curious if performance would be different if the datasets were created as "case insensitive" and you removed the following auxiliary parameters

Code:

default case = lower
preserve case = no
short preserve case = no

Also retest with oplocks turned off on the share. Then retest with aio read size = 1. Then repeat all this while running the procsystime dtrace script http://www.brendangregg.com/DTrace/procsystime

Then bring me a pony. :D

 

[pernils]

bigphill:
No I have not tested FreeNAS 9.10.2-U1. I lost to much "steam " while fighting this Asrock watchdog nand wereout.



anados:

My findings is that my browser script due in fact give you a decent average result. Hence why I also include the time (used my cellphone) while using windows file explorer.


FreeNAS mini 9.10
17,8 sec folder browsing script (87% utilization on single core)
18 – 30 sec while jumping in and out with windows file explorer. <-- here

FreeNAS backup 9.10
28.20 sec for folder browsing script (75% utilization on single core)
28 sec more or less steady while jumping in and out with windows file explorer. <-- and here

You (anados) have tons of more experience then I do, so with my scripts you will easily mimic my test. Give it ago and share your experience.

My windows box is way more snappier then any of the samba shares hands down. Right now we have around 30k+ files so this lag is starting to be noticeable.

The good thing is that rumors is saying this issue is having been noticed by the developers and attempt to fix it is in progress.

 

[bigphil]

It turns out that this may be a bug...see this post by @cyberjock (he mentions this thread in that update).

 

[pernils]

I have been on this since 9.10 went public. Contacted ix in end of 2016 about this matter. Was asked to collect my measurements and make a post / bug report.
A lots of stuff get in the way so only when it already have been addressed by the developers I finally manage to make some sort of report.

So here we are ...

 

[pernils]

Have added measurement for ext3 filesystem. The hardware is exactly the same as FreeNAS backup box but with original single disk which slower then the WD reds. The box must be 7 years now.

My verdict is that samba + ext3 is exchangeable with windows NTFS performance wise. When you use ZFS you start to get into trouble if you have lots of files in single directory. The actual transfer is comparable to a windows box but this time delay to list your directory is annoying.

 

[anodos]

Have added measurement for ext3 filesystem. The hardware is exactly the same as Freenas backup box but with original single disk which slower then the WD reds. The box must be 7 years now.

My verdict is that samba + ext3 is exchangeable with windows ntfs performance wise. When you use zfs you start to get into trouble if you have lots of files in single directory. The actual transfer is comparable to a windows box but this time delay to list your directory is annoying.

Since you already have a linux system set up, try ZFS on Linux with SA-based XATTRs enabled. http://open-zfs.org/wiki/Features#SA_based_xattrs

This should bring performance up to EXT3/4. Please note that the resulting zpool will be incompatible with FreeNAS because FreeBSD hasn't implemented SA based XATTRs.

 

[anodos]

It's also possible that the performance problems are caused by a change from Samba 4.3 -> Samba 4.4. Is this bug ticket public? I'm a bit interested in seeing how the investigation is going.

 

[pernils]

I'm not aware of any bug ticket on this.
Cyberjock have stated that he will keep us informed when new info arrives.


Skickat från min SM-G900F via Tapatalk

 

[cyberjock]

The bug ticket is internal only. There's been no fix yet. Lots of theories and John Hixson is working hard on this (apparently he's gotten almost no sleep the last 2 days).

What I heard (and this info could be wrong based on future troubleshooting, my memory failing, etc.) is that Samba is making syscalls that are behaving correctly on older FreeBSD releases (behavior is proper and Samba and FreeBSD are doing what we would call 'the right thing') but future versions of Samba and FreeBSD have Samba making proper syscalls and FreeBSD is falling all over itself.

 

[pernils]

As the bug now have been "fixed" (9.10.2-U3) we take another round of measurements. This time I focus only on the FreeNAS Mini with fresh booted ARC.

The announcement 9.10.2-U3
The bug #23459


FreeNAS Versions.
9.3-STABLE -201605170422
9.10.2-U3

The test :
I use the same bat script on the same share wich contain 50k files. I run the script a couple of times just to let the ARC to have it initialization period.
Then I copy a 10 GB file to and from my client HP Z600 with SSD. This is done after business hours so no busy network.
The transfer rate is average read from windows own copy msg.box.


Aux. parameter on the share

Code:

case sensitive = true
default case = lower
preserve case = no
short preserve case = no

store dos attributes = No
ea support = no
map archive = no
map hidden = no
map system = no
map readonly = no

FreeNAS 9.3
10 GB read from the Mini 111 MB/sec
10 GB write to the Mini 86 MB/sec
browse 50k files 3,5 sec

FreeNAS 9.10.2-U3
10 GB read from the Mini 107 MB/sec
10 GB write to the Mini 99 MB/sec
browse 50k files 4 sec

For upcoming FreeNAS 11 I assume it will follow 9.10.2-U3.

Experience of 9.3 is that it have a Windows feeling. With this I mean it will slowly get "slower", untill you do a reboot or deletes some snapshots. My snapshots scheman is generating around 300 of them.
I have 2 schema on each separate dataset witch runs every 10 minutes with retention of 2 days. On top of this I have once a day for 100 days.
I also have periodic replications to my backup box.

Not much from reading what others have in place but still it will bog down the box after a few weeks uptime. Nothing is shown in the report it's just have slower response. This is my experience with 9.3. How 11 will behave will the future tell.

Summary :
From this small test I see that zfs + samba will not be faster as FreeBSD progress in it's releases. The regression from 9 to 11 is 0.5 sec. Is the FreeBSD beginning to be bloated ?





***************************************************************
********************** Bonus *******************************

Aux. parameter .. Removing the dos attributes etc ...

Code:

case sensitive = true
default case = lower
preserve case = no
short preserve case = no

# store dos attributes = No
# ea support = no
# map archive = no
# map hidden = no
# map system = no
# map readonly = no

FreeNAS 9.10.2-U3
10 GB read from the Mini 110 MB/sec
10 GB write to the Mini 99 MB/sec
browse 50k files 8,2 sec

 

[mav@]

Is the FreeBSD beginning to be bloated ?

The problem was caused by attempt to better conform POSIX standard to fix some other problem. Present workaround is not perfect, but almost as good as it can be done with small blood without heavily breaking binary compatibility with FreeBSD. Hopefully FreeBSD 12 get it fixed in better way (by the cost of breaking binary compatibility with 11).

 

[anodos]

The problem was caused by attempt to better conform POSIX standard to fix some other problem. Present workaround is not perfect, but almost as good as it can be done with small blood without heavily breaking binary compatibility with FreeBSD. Hopefully FreeBSD 12 get it fixed in better way (by the cost of breaking binary compatibility with 11).

@mav@, I seem to recall that you did a lot of the work to improve the situation. Thanks, by the way! I'm a bit curious about what's causing the slowdown. Is it a particular system call that's slower on FreeBSD? Is it the way that Samba likes to recurse through all the things?

 

[mav@]

I'm a bit curious about what's causing the slowdown. Is it a particular system call that's slower on FreeBSD? Is it the way that Samba likes to recurse through all the things?

While reading directory, Samba calls telldir() libc function for every read directory entry. Due to the way FreeBSD kernel provides directory information (same as DragonflyBSD and NetBSD), it is not trivial to implement telldir() API in a way specified by POSIX. At some point FreeBSD 10 tried to better conform POSIX requirements there by storing some information in a list each time when telldir() is called, but that predictably caused O(n^2) delays when searching through that list on every call. I found the way how to optimize the search algorithm specifically for the Samba case (https://svnweb.freebsd.org/base?view=revision&revision=317064). It is not perfect, but very simple and seems like sufficient until modified kernel API one day allow to do it right.

 

[pernils]

@mav@ My main occupation is engineering and I'm working daily in 3d cad software. When you are doing an assembly you constant dive into folders to collect parts files sheetmetal, bolts, valves etc.
To not have a responsive directory tree really makes a impact on your work flow. You get disturbed when waiting for the folder browse to be populated every time you change folder.
If you want I can make a video record of my workflow so you can get a feeling of the impact this will have for a end user.

One can think: "big deal it's just 1,5 sec more then a genuine windows server" But I can ensure you it will not go unnoticed after a few weeks of modeling.

As iXsystem is targeting the storage market I think it could be a good idea to start to lobby for what overhaul who had to be done to get a more efficient samba implementation.
From the crappy hardware I have tested it on it seems not to be solvable by throwing more CPU at it.
What do you do with your Truenas customer when they make request how to improve the responsiveness ?
Adding features like Bhyve etc is really nice but I and I guees many other sees Free/TrueNAS, forehand as a storage unit.

For my case I'm starting to think if I should move over our drawings files to a window server again. In the end it's not my data. For our work case the prize for data integrity is becoming a bit to high compare to other solutions.

(For my private data at home I go with freenas)

I don't want to critise I'm just trying to leave feedback from a real work case with Freenas. And hopefully help to point out where iXsystem can improve it's product.

 

[mav@]

Sorry, but SMB is generally not my personal primary area, which is mostly OS and lower (ZFS, SCSI, iSCSI, FC, etc.). As you've confirmed above, my patch greatly improved the situation. May be not up to the point of 9.3, but close, comparing to alternative. Take that for now. If you need more, continue bug Samba people to further investigate that. On my synthetic libc tests I was getting sub-second delays up to 2 million files in a directory, so I don't think that delays of 4 seconds you see any more related to this specific bug, neither I have a way to improve it much now.

 

[anodos]

@mav@ My main occupation is engineering and I'm working daily in 3d cad software. When you are doing an assembly you constant dive into folders to collect parts files sheetmetal, bolts, valves etc.
To not have a responsive directory tree really makes a impact on your work flow. You get disturbed when waiting for the folder browse to be populated every time you change folder.
If you want I can make a video record of my workflow so you can get a feeling of the impact this will have for a end user.

One can think: "big deal it's just 1,5 sec more then a genuine windows server" But I can ensure you it will not go unnoticed after a few weeks of modeling.

As iXsystem is targeting the storage market I think it could be a good idea to start to lobby for what overhaul who had to be done to get a more efficient samba implementation.
From the crappy hardware I have tested it on it seems not to be solvable by throwing more CPU at it.
What do you do with your Truenas customer when they make request how to improve the responsiveness ?
Adding features like Bhyve etc is really nice but I and I guees many other sees Free/TrueNAS, forehand as a storage unit.

For my case I'm starting to think if I should move over our drawings files to a window server again. In the end it's not my data. For our work case the prize for data integrity is becoming a bit to high compare to other solutions.

(For my private data at home I go with freenas)

I don't want to critise I'm just trying to leave feedback from a real work case with Freenas. And hopefully help to point out where you can improve your product.

Do you mind sending me a current debug file? I can take a look at it and see if I can make any recommendations.

 

[pernils]

Take that for now. If you need more, continue bug Samba people to further investigate that.

There is a little problem due I don't know any samba people to bug.
Instead I will try to collect as much metrics for this as I can and post it in this tread if not iXsystem have any against it of course.

On my synthetic libc tests I was getting sub-second delays up to 2 million files in a directory, so I don't think that delays of 4 seconds you see any more related to this specific bug, neither I have a way to improve it much now.

You say my 4 second is indeed hardware related ? A bit confused here.

Thanks by the way for your effort on this bug..

 

[mav@]

You say my 4 second is indeed hardware related ? A bit confused here.

I haven't told that. I just told that there also must be some inevitable delays on Samba and network layers.