CIFS - NAS4Free=100MB/Sec, Freenas=~40MB/Sec - Same System, Same Volumes (ZFS)

[enly1]

Hi all,

Really struggling to stay with freenas as I can't get the performance I know the system can provide.

I am a Freenas noob, so please allow for that in the commentary and also, I'm not a nas4free advocate - I just tried a liveCD to see if it made any difference to cifs performace as I can't get a fast system (HP Microserver N40L + 2GB RAM + 2x500GB Raid1 ZFS)

PS - I've scanned many other posts and tried lots of random stuff, but nothing seems to make any differences.
I'm changing settings and restarting cifs service each time, so hopefully thats enough to pick up the changes for non-cifs settings I was tweeking.

So here we go (on my machine Freenas):

1. CIFS performance on Freenas 8.2 p1 is poor ~ 40Mbit/Sec transferring 2GB File to Desktop on Gigabit Switch with lots of transfer speed instability.
2. iPerf = 940Mbit/Sec with window sizes of 64K+, 8K=260Mbit/Sec / 16k=420Mbit/Sec / 32k=740Mbit/Sec
3. DD reads to /dev/null result in :

[root@freenas] # dd if=/mnt/mdata/personal/Movies.tc of=/dev/null bs=2k
1011712+0 records in
1011712+0 records out
2071986176 bytes transferred in 29.724186 secs (69707079 bytes/sec)

[root@freenas] # dd if=/mnt/mdata/personal/Movies.tc of=/dev/null bs=1m
1976+0 records in
1976+0 records out
2071986176 bytes transferred in 24.151185 secs (85792319 bytes/sec)

[root@freenas] # dd if=/mnt/mdata/personal/Movies.tc of=/dev/null bs=1500
1381324+1 records in
1381324+1 records out
2071986176 bytes transferred in 30.831289 secs (67204007 bytes/sec)

Quick Nas4Free test ( mounting the same zfs disks/volumes and exporting through cifs ):

1. Cifs Performance on Nas4Free - Stellar ~ 100MB/Sec sustained stable for the same file transfer.

So hardware wise, the box can do it, but freenas seems to be much slower.

I've compared smb.conf and can't see any material differences.
I tried a variety of smb socket options and other buffers
I tried a number of sysctl changes for buffers using larger settings from other forum posts

Basically, nothing seems to want to help.

So anyone have any ideas what to try next - I want to stay with freenas 8, but its a little hard when its so slow and runs so much faster on a different freebsd implementation.

Thanks in advance.

 

[cyberjock]

Did you try enabling auto-tune? There's lots of tuning that can be performed. Of course, I don't know much about it, but does it offer the same kind of reliability and recovery from boot drive failure as FreeNAS?

 

[Milhouse]

The 2GB of RAM is the likely culprit - with 4GB or less, FreeNAS will disable ZFS prefetch, there is a warning at the end of the boot sequence informing you of this. I wonder if Nas4Free enables prefetch, even with 2GB of RAM (this isn't usually recommended).

Your disk performance is suffering as a result... here are my results for an HP N36L with 4GB RAM (prefetch force enabled with vfs.zfs.prefetch_disable = 0 in System -> Sysctls), 8.3.0-RC1, 4x1TB RAID-Z1:

Code:

[root@freenas3] /mnt/share# dd if=./test.dat of=/dev/null bs=2k
4096000+0 records in
4096000+0 records out
8388608000 bytes transferred in 47.090596 secs (178137648 bytes/sec)
[root@freenas3] /mnt/share# dd if=./test.dat of=/dev/null bs=1m
8000+0 records in
8000+0 records out
8388608000 bytes transferred in 35.410159 secs (236898344 bytes/sec)

When transferring files over CIFS to a Windows 7 PC (GigE connection), the average transfer rate is between 60MB/s-65MB/s.

 

[enly1]

I was waiting for the statements about RAM etc, but the question is why is this only an issue with freenas 8 ?
What makes freenas8 so much heavier than the old freenas7 ?

Nas4Free is a continuation of freenas7, and when mounting the same ZFS volumes and exported over cifs hits 100+MB/Sec ( with prefetch also disabled ).
Both are freebsd, both are exporting the same ZVOLS, both running on the same hardware (just rebooting between USB sticks ) and both are using CIFS/Samba.

When I was trying with UFS insead of ZFS to avoid the RAM issues in earlier testing, I still never got anywhere near 100+MBytes/Sec rates ... so RAM shouldn't come into that.

Also, this is the only file accessed following reboot, so RAM shouldn't have been consumed yet ...

Thoughts ?

 

[enly1]

PS - I'll double check if pre-fetch is enabled again tonight, but pretty sure I saw it reporting the same disabled as <4GB ram present. Hard to remember as I've seen several boot screens last night....

 

[enly1]

Just a quick update before I head off to work - just forced prefetch on the freenas8 box and sure enough, 100+Mbytes/Sec across CIFS, so looking like nas4free didn't disable pre-fetch.

So if thats all it is, I suppose I'll have to spend another £50 on RAM :-(

Cheers - will confirm tonight.

 

[cyberjock]

That's pretty amazing that the prefetch causes such a performance boost. Gotta remember this thread for people that have <4GB of RAM and complain about read performance.

 

[paleoN]

That's pretty amazing that the prefetch causes such a performance boost. Gotta remember this thread for people that have <4GB of RAM and complain about read performance.

Keep in mind enly1 is using a mirrored configuration. Essentially mirrors will always benefit from prefetch as this will allow the system to start reading from the other disk in the mirror concurently. For raidzN and single disk systems YMMV in particular.

 

[enly1]

Right - quick update which confuses matters more.

I just shoved in the nas4free usb drive, setup the cifs again and retested.

New findings.

On nas4free, the prefetch is actually disabled:

nas4free:~# sysctl -a | grep prefetch
vfs.zfs.l2arc_noprefetch: 1
vfs.zfs.dedup.prefetch: 1
vfs.zfs.prefetch_disable: 1
vfs.zfs.snapshot_list_prefetch: 0
kstat.zfs.misc.arcstats.prefetch_data_hits: 0
kstat.zfs.misc.arcstats.prefetch_data_misses: 0
kstat.zfs.misc.arcstats.prefetch_metadata_hits: 8
kstat.zfs.misc.arcstats.prefetch_metadata_misses: 14

With the default nas4free config, I got about 40MB/Sec again.
Looking at CIFS, Async IO was disabled out of the box - I ticked it and it jumped back to 100+MB/Sec with pre-fetch still disabled.

So prefetch isn't the difference between the two.

But my testing this morning, the only change was pre-fetch between a slow transfer and a fast one. Turning it on and off resulted in the same slow performance with prefetch off on freenas8, but 100+MB/Sec with it on.

In both cases, no changes to cifs service were made.

So further investigation is definately required before we know whats going on.

I've actually gone out and bought 8GB ECC RAM based on this mornings test, so I'm now fully comitted to working out what the issue is.

But its getting late now, so looks like its going to be another night and another update to follow....

 

[enly1]

Okay - worked out some more tests on nas4free, so will try recreating on freenas tomorrow.

1. 100MB/S+ - Async IO + No Prefetch
2. 100MB/S+ - Async IO + Prefetch
3. 100MB/S+ - No Async IO + Prefetch
4. 40MB/S - No Async, No Prefetch.

So from the above, I'm guessing that async IO wasn't functioning correctly on freenas as I was seeing option 4 results when I had prefetch disabled but will try again tomorrow night post reboot.

 

[RAJOD]

I've had similar experience with freenas and a system with 2 gigs of ram.

NFS was faster on nas4free and for some reason NFS did not speed things up much on freenas 8.3.

ZFS nas4free was better on all tests and 2x faster on intel content creation test.

For people with lack of ram and want to run a NAS it seems nas4free is thinner thus a better choice.
during some file copy tests I monitored the cpu and ram use. It never went over 36% ram, 84% cpu.

I might hook it to my 8 core 32 gig dell server and see how it does

For power reasons I can use a cheap ivy cpu I go laying around.
I like putting old hardware to use so ill probably keep the A64 going for awhile to see how it does.
It only draws 80 watts at the wall with 6 TB of HDS.

with ZFS on I can copy files at around 48 MB/sec to the NAS. So its definitely very useable as is.

Just don't try to run a bunch of memory hogging features and it will be fine with 2 gigs.

 

[cyberjock]

So you are pretty much stating the obvious? The FreeNAS manual recommends 6GB of RAM minimum for ZFS and NAS4Free give you a ZFS warning if you have less than 512MB of RAM(as well as warning you to "expect unstable behavior").

Anyone working in IT should be consulting the minimum and recommended system requirements for any operating system they are about to install, especially if they've never used it before.

Not to belittle your post, but if the person setting up a server can't figure this out they are in very bad shape. However, your observation that NAS4Free does work well with as little as 2GB of RAM is observed on occasion. I've only used NAS4Free for about 3 hours to see it in action. But when people are stuck with no options except to use ZFS and have very little RAM I try to encourage them to consider NAS4Free. While I'm definitely happier with FreeNAS its very much a personal choice what OS you use and how you design your server.

 

[jgreco]

Yes, I've observed numerous times that there are design decisions made in FreeNAS that aim it straight at more heavily resourced systems. Some of them are trite things that can be undone via a little strategic tuning, some of them (such as the design of the middleware) are basically permanent features of the landscape (how do you reduce bloaty middleware?)

From a development perspective, I am guessing that the FreeNAS team finds it faster and easier to do their development without worrying excessively much about small amounts of resource consumption. With a baseline expectation that a system will have ... 6? 8GB? (I don't really know what their baseline expectation is) ... they can optimize for high performance fast systems and not need to worry too much about that guy trying to ZFS on 768MB of RAM. The high performance fast systems are going to be what pays the bills at iXsystems, as well, so that's not entirely unreasonable.

In the meantime, in ~1990, I remember paying $100 per 4MB of RAM. In 2005, it was $125 per 1GB of RAM. Now in 2013 it's $130 per 16GB of RAM.

So back in 2005 when a moderately large server was 1GB, and it was suggested that a server might need 1GB of memory just for its filesystem to be able to function, that was kind of a nonstarter. Even today, it strikes me as a little piggy, but I can see some justification for it. In another year or two, at the rate RAM prices are falling, 32GB will be less than $200.

So I remember the pain and agony of hitting the 440BX 1GB limit, as the ASUS P2B-DS and Intel ISP1100 were extremely good server choices that we ran for many years, and then the i386 4GB soft limit. Looks like the new pain is going to be the 32GB wall separating the E3 from the E5's, but fortunately I think most home user FreeNAS installs will experience decent performance on 16GB or 32GB of RAM for the next few years.

 

[RAJOD]

So you are pretty much stating the obvious? The FreeNAS manual recommends 6GB of RAM minimum for ZFS and NAS4Free give you a ZFS warning if you have less than 512MB of RAM(as well as warning you to "expect unstable behavior").

Anyone working in IT should be consulting the minimum and recommended system requirements for any operating system they are about to install, especially if they've never used it before.

Not to belittle your post, but if the person setting up a server can't figure this out they are in very bad shape. However, your observation that NAS4Free does work well with as little as 2GB of RAM is observed on occasion. I've only used NAS4Free for about 3 hours to see it in action. But when people are stuck with no options except to use ZFS and have very little RAM I try to encourage them to consider NAS4Free. While I'm definitely happier with FreeNAS its very much a personal choice what OS you use and how you design your server.

Never assume something is obvious to everyone just because it is to you. Its FREE. So many people with less than optimal systems will be attempting to run it on old hardware. Rather than tell them to buy something they might not be able to afford, just point them in the direction of how to get the most of what they have.

Who said anything about working in IT? The guy that started the post said he was a noob. Just trying to help him.

 

[cyberjock]

Never assume something is obvious to everyone just because it is to you. Its FREE. So many people with less than optimal systems will be attempting to run it on old hardware. Rather than tell them to buy something they might not be able to afford, just point them in the direction of how to get the most of what they have.

Who said anything about working in IT? The guy that started the post said he was a noob. Just trying to help him.

THe OP didn't say he was a noob. What he said was:

I am a Freenas noob...

Being a FreeNAS noob is radically different from being a noob to IT. I've worked/played in IT since I was in 3rd grade programming my family's Apple IIe with the help of books from the local library.

Guess what? I had never touched FreeBSD or FreeNAS until February 2012. I knew that it existed and that Linux was similar to it, but I had very little Linux experience too. But thanks to my background in IT(both hardware and software), reading the manual cover to cover, and using the forum search engine I had zero problems getting my first FreeNAS machine up and running with leftover hardware. Yes, I bought more RAM before I even tried to install FreeNAS.

My first few posts were questions regarding how FreeNAS worked internally. I didn't know if my IT background and reading the manual gave me the right conclusions about things like SMB2 and the relationship between hard drives, VDEVs, and ZPOOLs. I got pretty good responses because it was obvious I had done my homework, attempted to follow the manual's recommendations when provided, etc. (Hint: You know what gets us highly experienced users interested in your post? When you do everything correctly and something still doesn't work right. The more obscure the problem the more we like it. We love a good challenge and ignore the "you need more RAM...again" threads that the OPs should have figured out for themselves.)

So what did I do after that? I created the "Slideshow explaining VDev, Zpools, ZIL and L2arc for noobs!" so I could share some of the information I had figured out that wasn't easily explained anywhere I could find. With each update of FreeNAS I've taken the time to update the presentation with new information, tried to add and expand the information the presentation contains to help other newbies to FreeNAS/FreeBSD,(especially the regular mistakes that are made.. insufficient RAM being the most common) among others.

Where would I be right now without my background in IT? I'd almost certainly have no data. Unless you have been reading the forums daily for many months you probably don't realize how many people have lost their data because they didn't understand the relationships between ZPOOLs, VDEVs, and hard drives, that opting to not use a UPS is a big mistake, and that not having enough RAM causes an unstable system. All 3 of those mistakes have caused many people main because they lost their entire zpool despite trying to do other things right.

So let me ask you a question RAJOD. The minimum system requirements for Windows 7 are a 1Ghz 32-bit or 64-bit CPU, 1GB of RAM(32-bit) or 2GB of RAM(64-bit), 16 GB of hard drive space(32bit) or 20GB of hard drive space(64-bit) and a Direct X 9 compatible video card. Would you ever consider going below that and expect the system to be stable and reliable? I don't know about you, but I've never seen a Windows 7 system at any computer store with less than 3GB, and many of my friends that have bought computers from the store that had 6 or 8GB of RAM. I'd bet if you had a system with only 2GB of RAM many people would recommend you put XP on it(Hint: Windows XP minimum RAM requirement is 64 MB and recommended RAM is 128+). XP is far from the RAM hog Windows 7 is. So in essence most people choose to install the OS based on your hardware. So explain to me why you'd go cheap on RAM for a server when the RAM needs are clearly mentioned in the manual as well as my guide, and the forum is laced with people who have lost all of their data because they "thought they could get by with less". That's why I said if you aren't following they system requirements you are in very sad shape. It doesn't take a rocket scientist to acknowledge that some minimum system requirements are there for a reason and not to line the pockets of IT companies.

At the end of the day, anyone without a background in IT of at least a few years is in horrible shape to jump into FreeNAS without a lot of experimenting and learning(months maybe?). If you don't know what you are doing you can cost yourself all of the data you are trying to protect. Even people with 10+ years of IT experience sometimes get their butts kicked and have had to kiss their data goodbye on many occasions. FreeBSD is not for newbies to jump into and immediately trust. It has a very high barrier of entry. FreeNAS is an attempt to make it easier by creating a GUI that makes administration easier, but it doesn't alleviate anyone of the responsibility of understanding how all the stuff works under the hood. Anyone that thinks that is true in for a rude awakening when things get ugly and they are in a situation where the system isn't working right. Even for myself, who has about 12 hours a day to experiment with stuff as I please spent 3 weeks reading and forum searching before I even installed FreeNAS on a VM. I'm fairly lucky that I'm very smart and a quick learner compared to many people(I was a licensed nuclear reactor operator at the age of 19) and I try to help out others where I can. I definitely don't expect your average "noob" to be as thorough as me(I'm OCD like that sometimes) nor do I expect them to retain 95% of everything they read the first time. I also despise threads that I read where people have almost certainly lost their data forever. It's not fun to read and its not fun when it happens to you.

So before I put you to sleep I'll just say this... if I don't assume you have some IT background, my answers would always be "get away from FreeNAS for about 5 years" because the barrier to entry is high, and the costs of many of the smallest lessons are "up to and including all of the data on your server". Some users have had too little RAM, had 1 kernel panic and the zpool never mounted again. Where have you seen a Windows machine crash from too little RAM corrupted the file system and the end result is a complete loss of all data on the server? Don't like this high barrier to entry? Go back to Windows where 80% of the world is. It's not going to get any easier anytime soon, and the fact that the barrier is so high is too much for many people(even those who have been in IT for a decade) so what kind of end result should be expect from the poor soul that has just realized "ZFS is awesome" and wants to use it but can't even explain the difference between a partition and file system?

 

[RAJOD]

THe OP didn't say he was a noob. What he said was:

Being a FreeNAS noob is radically different from being a noob to IT. I've worked/played in IT since I was in 3rd grade programming my family's Apple IIe with the help of books from the local library.

hehe you an old computer fart like me. Aplle IIe! I go back pre Atari 800. A 2k timex sinclair left by a dead uncle.

Yea I hear ya on all your points. I'm use to hardware raid controllers, knock on wood never lost data yet. Well I have had to pull data off dead drives many times but always got it back.

I was going to get another hardware raid when a systems analyst buddy of mine told me about freenas. I was a bit skeptical on software raids so I decided to check it out with some old hardware.

I've since taken the drives out of that A64 box and used a IVY G1610 /8 gig I was going to use for a HT box. I really like it so far. transfer speeds up in the 70-90 MB/sec now. Almost doubled performance. I liked the Freenas interface a bit better than nas4free but either is fine with me.

Hehe on that 2 gig question on windows. My mother inlaw got a free (busted) laptop with vista on it. Its a dual core intel 1.8 with 2 gigs of ram. I put win7 Ult on 64 it and it runs dam good, better than it did with vista 32.
I'll prolly max it out for her (4 gig) but with 2 it runs well. Windows with low ram like 2 gig just gets slow (as you know more swapping) I've not had too may crashes due to low ram. But 2 gig is lower limit for 7.

Yea I don't know all the horror stories you must know with freenas/nas4free. Since I think I'll be keeping it for awhile I did meet the specs.

I might make a video server but that built in UpnP can only be seen by 1 of my 4 HT devices. No Samsung smart TVs or blue ray can see it. Only my sony blue ray players smart section can see and play videos. Thats another topic and one I should post in another area.

 

[cyberjock]

When I was in the Navy we had a system that was a Pentium III and was a software RAID5 running Windows 2000 Server. It had an amazing 1.5MB/sec write speed. I tried to defrag the array(120GBx4 drives) and it took 5 days! I was shocked when I started it Friday after work and when I came in Monday morning it was only at 40% or so. Then when the OS died recovery was all but impossible. I had to use R-Studio with its software RAID5 rebuild tool to recover the data since my supervisor refused to buy new tapes for our backup system and (surprise!) our 5 year old tapes were trash when we tried to do a restore.

I was skeptical as heck about trying a software RAID again after that horrible experience, but I figured ZFS can't be "that" bad if its being trusted for enterprise class performance and reliability.

It seems like with ZFS it's all "peaches-and-cream" or you just face-planted into a brick wall. There really isn't much "in-between". We've had people lose their entire zpool from a single kernel panic, from 1 stick of bad RAM(a ZFS guide I found does list ECC RAM as a "requirement"), a single loss of power without an UPS and single drive failure when they thought they could expand a RAIDZ2 vdev with a single drive. It seems ZFS works really well when properly used. But if you aren't willing to buy it the hardware it needs and understand its limitations, then kiss everything good-bye. I was a little worried when i crashed my system a few days ago with an improper command the other day while trying to test AES performance. Lucky for me the only thing that went bad was one drive appeared to have a corrupt primary GPT table.

 

[RAJOD]

I had to use R-Studio with its software RAID5 rebuild tool to recover the data since my supervisor refused to buy new tapes for our backup system and (surprise!) our 5 year old tapes were trash when we tried to do a restore.

I was skeptical as heck about trying a software RAID again after that horrible experience, but I figured ZFS can't be "that" bad if its being trusted for enterprise class performance and reliability.

[/URL].

Yea I own a copy of R-studio, decent program for sure.

You have a 18x2TB raid? All one pool?

I'm chicken, I like non parity mirrors Raid 1 or 10. The thought if loosing a huge pool during a rebuild or something gives me a gut ache. Cost a lil more per GB but i've had good luck with them. I have an old adaptek raid card in a p3 running 10 years 24/7 doing FTP backup via script. Had one HD die on it, put in new one booted and it auto rebuilt like nothing every happened. It might be 12 years, long time.

Check this out. http://www.standalone-sysadmin.com/blog/2012/08/i-come-not-to-praise-raid-5/

I purposely corrupted one of my ZFS mirrors to see how it did. I don't think it was automated like my cards, I think I had to tell it to start the rebuild. Took like 8 hours to rebuild 3TB.

Yea I saw ECC was recommended, for now I'm just using 1333 DDR3 but ECC probably a good idea wth ZFS. I'm still not 100 percent sold on freenas mainly because I don't know enough yet. So i'm reading and tinkering with it. Its got back up data on it now so no biggie if it blows. But I'' likely swap in a ECC MB/CPU when/if I put real data on it.

You might like this ECC discussion with ZFS.

http://hardforum.com/showthread.php?t=1689724

 

[Stephens]

Yea I hear ya on all your points. I'm use to hardware raid controllers, knock on wood never lost data yet.

Not that you know of. Bitrot is silent and with most file systems, you wouldn't detect it.

 

[cyberjock]

That's the problem with "silent corruption". The very definition of silent corruption is corruption you aren't aware of. The hardware has no idea anything is wrong. It's one thing for the hard drive to say "I'm messed up.. but here's some garbage data" and it's another to say "Here's your data.. everything is fine".. but it isn't.