Performance Complete TEST (NFS,AFP,CIFS) with FreeNas 8.0.3, 8.3 and Nas4Free 9.1.0.1

[benamira]

I ran a deep test for a new NAS i am building for storing media files...
I want to share with speed gurus or experts for your comments. The results are pretty strange...

Attached the picture with the results:



Clearly in my environment something is not working good with newest versions of Nas4Free and FreeNAS when writing over AFP or NFS.

Any ideas? Suggestions?

- - - Updated - - -

I have all the details of the test so if you need furhter info tell me and i will provide all the info of the test and the config i have.
Thanks for your help!!

 

[cyberjock]

Re: Performance Complete TEST (NFS,AFP,CIFS) with FreeNas 8.0.3, 8.3 and Nas4Free 9.1

Firstly, I just want to say that's a cool comparision.

What is the value of the y-axis? MB/sec?

I'd start looking to see if the default settings changed between the stuff that performs and the stuff that doesn't perform. I don't have any ideas off of the top of my head personally. I'm CIFS all the way on my FN servers.

 

[benamira]

Re: Performance Complete TEST (NFS,AFP,CIFS) with FreeNas 8.0.3, 8.3 and Nas4Free 9.1

That's correct, the value for the y-axis is MB/s.
My production environment is FreeNas 8.0.1 which currently is resilient for me, and works almost perfect for its purpose. Media repository for streaming to local clients in my home.
The test i have made is to decide the version for a twin NAS i am building for a friend.

The main change in the test is that my config works with a ZFS pool v15 (i think from my FN version), and the 2 newest SO Nas4Free 9.1.0.1 and FreeNas 8.3.1 works with ZFS v28. I wanted to make the fourth test with a legacy FreeNAS version that used to work pretty well, but i couldn't make it work (FreeNAS 0.7 Sabanda) with problems to recognize the network card...

 

[benamira]

Re: Performance Complete TEST (NFS,AFP,CIFS) with FreeNas 8.0.3, 8.3 and Nas4Free 9.1

Any ideas of the bad behaviour when writing over AFP or NFS??
I have been investigating for more than a week and tried fine tuning in ZFS parameters but nothing works.
FreeNas 8.0.1 works almost perfect when writing over AFP and NFS so i am a bit confused...

 

[cyberjock]

Re: Performance Complete TEST (NFS,AFP,CIFS) with FreeNas 8.0.3, 8.3 and Nas4Free 9.1

Judging from the fact that CIFS provides good performance the issue isn't ZFS. If it were ZFS they'd all be held down. In your case, there's something about AFP and NFS that makes it perform differently. That's why I said you should look and see if the default values for AFP and NFS settings changed between the really old FreeNAS and the newer software you compared it with. If some caching parameter for AFP and NFS changed that could have a big impact. The whole thing is odd because typically NFS should be faster than(or at least equal to) CIFS. I have no comment for AFP. I hate protocols that a single company basically invents to be special and promotes, so I'll never use AFP if I can help it.

Protocols do behave differently and do get "upgrades" (think SMB1 to SMB2) that can cause problems with compatibility when the client and server don't match. I'm not a NFS guru, but I don't believe there's been any significant upgrades to NFS in 2 years or so, and I know nothing about AFP since I hate it.

Also, you never listed your hardware specs, which may provide a clue.

 

[benamira]

Re: Performance Complete TEST (NFS,AFP,CIFS) with FreeNas 8.0.3, 8.3 and Nas4Free 9.1

Thanks for your attention and advice.
I will continue my investigations and tests and will share here in the forum. I'm getting fan of performance tests ;)

Regarding my Config, you're right, i should have given more details, here you have:

ASUS E35MI-I mobo
8 GB DDR3 RAM
6 x WD 3 TB Caviar Green
Lian-Li PC-Q25

Build ZFS Pool raidz1, enabled 4k advanced format.
Big Share, around 13 TB with all the drives. Then exporting through AFP & NFS to MAC Clients (iMac & Mac Mini) & Raspberry Pi , CIFS & SMB to Windows 7/8 & Raspberry Pi.

 

[cyberjock]

Re: Performance Complete TEST (NFS,AFP,CIFS) with FreeNas 8.0.3, 8.3 and Nas4Free 9.1

My first guess is that you may be short on RAM and AFP/NFS doesn't cache stuff the same way as Samba does(CIFS) and that's hurting performance. 18TB of storage with 8GB of RAM may be a little thin.

The only other idea I can give is that your mac clients may not like the newer versions of the NFS and AFP services.

In any case, you're kind of on your own to find and fix it. There's no easy way to troubleshoot performance issues like you're seeing except to start testing your hardware and software until you get to the answer. :(

 

[hotalot]

Re: Performance Complete TEST (NFS,AFP,CIFS) with FreeNas 8.0.3, 8.3 and Nas4Free 9.1

Did you set nfs async transfers?
If you don't have a zil that option should be checked.

 

[atakacs]

Just out of curiosity what tool(s) did you use to obtain those benchmarks ?

 

[benamira]

Re: Performance Complete TEST (NFS,AFP,CIFS) with FreeNas 8.0.3, 8.3 and Nas4Free 9.1

Did you set nfs async transfers?
If you don't have a zil that option should be checked.

[hotalot]

no advanced config in NFS, do i have to set async transfers in the GUI or through the CLI??

Just out of curiosity what tool(s) did you use to obtain those benchmarks ?

[atakacs]

I used 2 test files: 1 x 1,2 Gb and 1 x 7 Gb aprox because the purpose of this NAS is store Media Files.
I manually copy/move to and from in every scenario to get the results.

 

[cyberjock]

Uhh, setting async tranfers can be dangerous for your data. You can performance at the risk of corrupting files. It's not as simple as "check the box if you don't have a ZIL". It's a risk assessment that the server admin should be making. Hint: I'd never check that box if I valued my data.

 

[benamira]

Uhh, setting async tranfers can be dangerous for your data. You can performance at the risk of corrupting files. It's not as simple as "check the box if you don't have a ZIL". It's a risk assessment that the server admin should be making. Hint: I'd never check that box if I valued my data.

Ok, thanks for your advice.
My previous understanding is that ZIL and async transfer were important for apps intensive in I/Os like VMWare or DBs, and not as important for a big repository for media files (big files, few clients...). I mean in terms of reliability of your data, or i missed something else potentially dangerous for a Data Loss??

 

[hotalot]

From the ZFSTuningGuide (https://wiki.freebsd.org/ZFSTuningGuide)



NFS tuning

The combination of ZFS and NFS stresses the ZIL to the point that performance falls significantly below expected levels. The best solution is to put the ZIL on a fast SSD (or a pair of SSDs in a mirror, for added redundancy). The next best solution is to disable ZIL with the following setting in loader.conf (up to ZFS version 15):

vfs.zfs.zil_disable="1"

In latest ZFS (version 28) the vfs.zfs.zil_disable loader tunable was replaced with the "sync" dataset property. You can now enable/disable ZIL on a per-dataset basis.

zfs set sync=disabled tank/dataset

Disabling ZIL is not recommended where data consistency is required (such as database servers) but will not result in file system corruption. See http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#Disabling_the_ZIL_.28Don.27t.29 .

ZFS is designed to be used with "raw" drives - i.e. not over already created hardware RAID volumes (this is sometimes called "JBOD" or "passthrough" mode when used with RAID controllers), but can benefit greatly from good and fast controllers.

 

[cyberjock]

And just like disabling the ZIL is not recommended, setting sync=disabled is also a recipe for disaster.

Be very careful and do a lot(I'm talking reading everything you can find on it) before choosing to set sync=disabled. Using that gives you some amazing performance, but a single improper shutdown of your FreeNAS server from a kernel panic, UPS that doesn't pick up(or doesn't exist) and you can probably kiss your entire zpool goodbye. The warnings against disabling sync writes are scattered throughout the forum, and more than 1 person has lost everything because they made this setting change.

If you do use that setting, you'd better have very thorough and frequent backups. It turns your zpool into a car. When it works, it works great. When it doesn't work, you are stranded with no options whatsoever.

 

[benamira]

Thanks a lot for this information. It is very useful. Data transfers are quite Ok for my environment right now, so i will take a conservative approach and don't touch anything.
Also i am reaching 80% of the full space, so i have to begin thinking in building a new and larger system. (oh my god!!)

For the new NAS i have many ideas, regarding the hardware, box, drives to use... and better config for a budget of no more than 2000€ including drives.
Purpose is the same: Big Media Files Repository to be access for several clients (Media Center).
I am thinking in a 10 drive config. Does it make sense for you? Any trick from the gurus to take into consideration?

 

[hotalot]

And just like disabling the ZIL is not recommended, setting sync=disabled is also a recipe for disaster.

Be very careful and do a lot(I'm talking reading everything you can find on it) before choosing to set sync=disabled. Using that gives you some amazing performance, but a single improper shutdown of your FreeNAS server from a kernel panic, UPS that doesn't pick up(or doesn't exist) and you can probably kiss your entire zpool goodbye. The warnings against disabling sync writes are scattered throughout the forum, and more than 1 person has lost everything because they made this setting change.

If you do use that setting, you'd better have very thorough and frequent backups. It turns your zpool into a car. When it works, it works great. When it doesn't work, you are stranded with no options whatsoever.

Sure, I think backup is mandatory, I have 2 of them.... for my really important files - less than 120Gb. My point with sync=disabled is that for a media server if it crashes you rip (or download) your movie again, no big deal, and yes, I agree that sync=standard is the "right" way to go.

 

[cyberjock]

In all seriousness, I'd look at upgrading your case first. Get a 24 drive case and move your hardware to the new case. Then adding more storage space provides you with 2 options:

Option 1: Add more vdevs and keep the disks you have.
Option 2: Create a new zpool and copy all of your data to the new zpool.

I have done both and with lots of free disk slots you have lots of options both during your migration to bigger drives or bigger zpools in the near future as well as long term.

I will tell you that adding more vdevs can provide some big performance improvements so option 1 may appeal to you. If the disks are getting old and you are concerned they may start failing you might want to do option 2 in the near future and then add a second vdev later.

 

[benamira]

Sure cyberjock. this is an option too.
But i think i have limitations with my mobo (with 6 SATA ports), RAM...
So it is not only adding drives and buying a new case, it is also RAM, new mobo or RAID controller...
So i will probably build a new one and sell the old one to one of my friends, they will be happy to get a "cheap" and big NAS.... :)

 

[marcevan]

I haven't seen the CPU spec'd out in your config.

Samba is single-threaded as each request take a thread. Your manual file is a single user so 1 thread is being used.

NFS is multi-threaded.

So it depends on the CPU you have that if it's a single-core like a Sempron vs a x4 or something.

 

[cyberjock]

You just responded to a thread from 7 months ago....