At a loss... Need some insight

[jsilvagi]

So could use some help here guys...

I am in the process of building my second FreeNAS box, been running another one on 8.0.2 for around 18 months. Both servers are nearly identical specs.

Mobo - Asus M2N32-SLI Deluxe
RAM - 8GB DDR2-800
CPU - AMD Phenom ( Original server dual core, new server quad core ).
Network - Dual gig-E NICs
Using onboard SATA controllers
Mixture of disks, all SATA 7200 RPM drives. All using ZFS w/ RAIDZ.

All riding on a gigabit network

So my original 8.0.2 system the only tweaking I did was add the following to the loader.conf

Code:

vm.kmem_size="512M"
vm.kmem_size_max="512M"
vfs.zfs.arc_max="60M"
vfs.zfs.vdev.cache.size="10M"
vfs.zfs.prefetch_disable="1"
vfs.zfs.txg.timeout="5"
vfs.zfs.zil_disable="1" 

With this config I could sustain a combined read/write of 800Mbps


So on the new system I decided to start up with 8.3.0. I could not get more than 40Mbps sustained read/write. I turned on auto-tune, I tried the above tweaks to the loader.conf. Same result. So I tried 8.2.0. Exact same result.

After hours of tweaking and messing with it, I tried 8.0.2 for the hell of it. Able to get 800Mbps out of it...

*scratches head*

Any ideas?

 

[CAlbertson]

I asked a pperformance question here and the most helfull answer I got was to try some tests to measure the raw read/write speed on the disks on each system. So from the terminal try "dd if=/dev/zero of=mytestfile" and then do a few control "t" and look at the write speed that is not using the network. Do this on both FreeNAS boxes and see if they are the same. next use iperf and measure the network speeds. Do it in both directions with servers and clients on each computer you have.

In my case I found the network interface was very slow in just one direction

I guess in your case maybe some device driver in BSD got "updated" and no longer works with some bit of hardware you have

 

[cyberjock]

8.0.2 is quite old and you should consider upgrading due to scurity vulnerabilities in 8.0.4, not to mention the new zfs version in 8.3.0. There is more overhead with 8.3 due to all of the new features, zfs version, etc. Some people have noticed the same performance issue compared to old version when you don't have good tweaking skills but have tweaked the system and/or if your computer doesn't have a sufficiently powerful CPU.

I'm not a fan of AMD so I really can't vouch for how "fast" your CPU is, not to mention the fact you gave no Ghz rating for your system.

 

[jsilvagi]

noob :

Procs are in the 2GHz range, so plenty fast for what it is doing. When I was doing my tests in 8.3.0 and 8.2.0 it was barely even touching the CPU.

As for upgrading, I would love to... but not at that level of performance cost. Going from 800Mbps to 50Mbps just really makes the environment useless to me.


CAlbertson : My thought was the same as yours. It has to be a driver issue as the same hardware performs perfect in the previous version.


At this point I can't do testing, shortly after typing this post the power supply in the original system blew. I had to canabalize the PS from the new one to get the original back up and running as it has many VMs on it that keeps my environment running.

 

[cyberjock]

Few things to mention.. you may know these. These are common noob mistakes.

1. After changing a sysctl I believe a reboot is required.
2. If you enable autotune and then disable it later the entries for autotune stick around. You have to manually delete them. I just noticed this recently. I normally enable autotune and do no other tweaking at all and I have yet to build a system that doesn't give at least 70MB/sec across Gb LAN.
3. vfs.zfs.prefetch_disable="1" will definitely hurt write performance. Your other tweaks don't look like stuff I would recommend either. I would try disabling all of your sysctl and tunables tweaks and see if that helps. I tend to trust the philosophy that a sufficiently powerful system(which I think yours qualifies) shouldn't need tweaks to get good performance.
4. Pre-8.3 ZFS used v15. Some of your tweaks may not have been valid with v15 but suddenly are in v28. I really don't like those tweaks you have. All of these to me are just terrible ideas considering your RAM:

vfs.zfs.arc_max="60M"
vfs.zfs.vdev.cache.size="10M"
vfs.zfs.prefetch_disable="1"
vfs.zfs.zil_disable="1"

Keep in mind I'm not a tweaker despite reading up on them for curiosity. It almost looks like you were deliberately trying NOT to use your RAM for anything.

ZFS prefetch is a MAJOR performance booster and is normally autoenabled when you have >4GB of RAM. Yet you have deliberately disabled it.

Also you have vfs.zfs.zil_disable which doesn't appear to be valid for v28. The replacement is discussed at http://wiki.freebsd.org/ZFSTuningGuide. Of course, as that link says it is really not a good idea to use this feature as a loss of power can have serious consequences for your data. Other places tell you that you should only do this for testing to determine where your bottleneck is and should never ever be used in an environment where data is important.

These settings are from someone that tweaked their FreeBSD(not FreeNAS, but close enough) for a system with 768MB of RAM...

vm.kmem_size="330M"
vm.kmem_size_max="330M"
vfs.zfs.arc_max="40M"
vfs.zfs.vdev.cache.size="5M"

If I had $1000 I'd totally wager that your tweaks are to blame for your performance. I'd ditch all of your tweaks, do a reboot and see what performance is. With your hardware you should be sitting at least 300Mb/sec(absolutely worst case).

If you have adequately tested your system and determined that your settings aren't the issue(please don't dismiss this as I'm fairly sure this is your problem) then I'd see about doing a RAM test and hard drive SMART test. v15 doesn't work the same as v28 and maybe it is stressing your system differently than v15 and identifying some issue you don't know you have.

In any case, even if you fix the problem I'd still seriously consider if you want your zil disabled setting in there. While it can give you amazing performance gains it does so at the potential risk of your data if you lose power while unwritten data is in RAM.

 

[CAlbertson]

...Some people have noticed the same performance issue compared to old version when you don't have good tweaking skills but have tweaked the system and/or if your computer doesn't have a sufficiently powerful CPU.

The OP is seeing a 20X reduction in performance. Yes the new ZFS is slower but only by a few percent, not thing like 20X slower.

CPU speed can't be the issue either because his new CPU runs the old FreeNAS 8.0 just as fast as his old CPU.

The first step is to isolate the problem better. One good test is to build a UFS volume and measure the speed using UFS compared to ZFS. I did this after people told me that ZFS on a 4GB system would be slow. Turned out UFS as just as slow because my bottle neck as in the network card. Swapped out that and saw a 10X performance gain. Still not great but I've using a low-end Atom powered computer to save power. When I find a 20W CPU that can handle 16GB RAM with price under $100 I'll upgrade.

Let's wait for the OP to post his test results. I just can't believe those ZFS tweaks are the cause of a 20X slow down but I could be wrong.

 

[cyberjock]

Oh, I can give you a few tweaks that will make ZFS so slow the machine won't finish booting until next year! Just as they can cause amazing performance gains they can also cause hell like you'd never want to know.

One of the admins of the forum has seen his system go from 80MB/sec to about 40MB/sec with each successive release getting slower. The OP has said he was on 8.0.2. So yeah, I could see that going from 8.0.2 to 8.3 might matter. I haven't had that problem though since all of the system I have built are always overbuilt enough to allow for great performance. My home server can scrub at 800MB/sec!

Another one of the more senior guys in PM has said that the tweaks made are completely bogus and a clear sign that the OP was trying stuff out of desperation without a clear understanding of what was going on. My recommendation when there is a problem is to always take a step back and remove tunables and sysctls. I'd say about 75% of the time when people start tuning stuff they don't spend the necessary time to really understand what they're doing. Their problems often disappear when their "customization" is removed. I know I've personally helped 4 people via teamviewer and all except 1 was self inflicted...by tunables and sysctls. The problem is that the defaults are defaults because they are a happy medium for most people. Too many people think they're not part of "most people" and that they can tweak out more performance. Especially considering that your bottleneck will almost always be your LAN connections.

People don't want to spend an extra $50 for a faster CPU, but have no problem spending 10+ hours trying to get tunables/sysctls working "just right". Why not just work 10 hours of overtime at work, spend the extra $50 on the CPU and pocket a couple hundred dollars in your pocket to boot! I'd bet there is less than 5 people in this forum that could actually effectively tweak their systems. The rest of us(yes.. including myself) would be crazy to try and likely won't get proper results.

Who in their right might would really go research every single tunable and sysctls they use to make sure that when a new zfs version comes out their tunables still do what they want? *crickets* I know I wouldn't think of that right off the bat. When you want to upgrade FreeNAS you read the release notes, see if anyone has any glaring "ZOMG not working" problems and do the upgrade.

 

[sy5tem]

People don't want to spend an extra $50 for a faster CPU, but have no problem spending 10+ hours trying to get tunables/sysctls working "just right". Why not just work 10 hours of overtime at work, spend the extra $50 on the CPU and pocket a couple hundred dollars in your pocket to boot! I'd bet there is less than 5 people in this forum that could actually effectively tweak their systems. The rest of us(yes.. including myself) would be crazy to try and likely won't get proper results.

i think anyone think of their time as "FREE" . also, thinking like that ends with spending thousand of $. i do not know a lot about freebsd , linux of other nix software. But like the OP, im trying to get the best out of anything. and the guru and other alike lurking in these forums, can give us tips. Because also , tweaking a system is like 50% of the game. :P

i must say that 800Mbps is lile OMG WOW!!!! gima proof :) ,

and all i know is that autotune = death ... all the system i builded and clicked on autotune would crash from 2 h to 1 week...

 

[cyberjock]

I'll totally agree that tweaking can be alot of fun, but when you've never played with FreeBSD, FreeNAS, or ZFS you have months of reading to do before you can really have a clue and tweak stuff KNOWING what you are doing. I've done lots of reading about ZFS and I'm not sure I'd even start playing around with the tweaks. Who do you ask for help? There's only 2 or 3 people in this forum that probably have even remotely a clue how to tweak(no.. I'm not one of them) and they're far too busy to try to explain all of the intricacies of tweaking to everyone that wants FreeNAS on old/underpowered hardware.

Just check out http://forums.freenas.org/showthrea...e-(log-(zil)-and-L2ARC)/page4&highlight=l2arc and tell me that all that stuff makes sense after a quick read. Let's not even consider the fact that there are probably a boatload more tunables that affect the tunables we discussed there. And if you don't know about them and how the all interact and work together(or against each other) you can't even begin to hope to tune a system that's underpowered to start with. All that stuff and much much more all has to work together to make a well oiled machine. When ZFS was invented they KNEW that RAM prices were going to be dirt cheap in the future, so they chose to rely on that heavily because it was a sure thing to happen.

As for autotune I've had great results on all 3 servers I've enabled it with.

Personally, if I was handed that server and said "make it better" I'd drop more RAM(probably 16GB), upgrade the NIC to Intel if it wasn't, and ditch all of those tweaks. None of them look very good at all. Even got a PM from someone else asking me if I had a clue what the OP was trying to do with those tweaks. They just don't make any sense at all. Those tweaks are artificially starving ZFS's ARC and write cache for no reason. Why you'd deliberately starve a service that was designed to use lots of RAM is completely beyond me.

Anyway, I'm done. I've said what needs to be said before. Take my advice or leave it. The choice is yours.

 

[papageorgi]

i do have to say, i choose AMD for my server since the cost/performance ration is usually best and essentially sun built ZFS with AMD in mind since they were the first to have x64 truly capable processors.

 

[dmt0]

I have a similar problem as the OP.
Also have 8GB ram and also major slowdown after upgrade.
I used to have speed of transfer over NFS of 20MB/s - not much but useable.
Now it's at most 3.5MB/s, no matter how much I play with tuneables.
Did the dd thing to check disk performance as recommended by CAlbertson, got around 125MB/s write performance.
So what does that leave me with? Network? I have an Intel NIC supposedly so well supported by BSD. Could the driver be messed up in this release?

Update: Wireless. Works fine over ethernet. Stupid me.

 

[cyberjock]

i do have to say, i choose AMD for my server since the cost/performance ration is usually best and essentially sun built ZFS with AMD in mind since they were the first to have x64 truly capable processors.

So what you are saying is that you also buy Ford vehicles because Ford was the first to make a mass produced affordable vehicle so the road must work better with Fords? I fail to see that logic.

 

[papageorgi]

So what you are saying is that you also buy Ford vehicles because Ford was the first to make a mass produced affordable vehicle so the road must work better with Fords? I fail to see that logic.

no not at all it was just a nice bonus for me that it was originally designed by sun to work on amd because of it support for x64 before intel. as i stated though i chose amd for the price to performance ratio. intel does make great processors as amd. the main reason is and almost always is, intel processors are very expensive and with that expense you get more performance but for my build i just don't need a lot of performance. for home users or small business it's usually the same scenario. this being just storing data and streaming audio/video, backups.... so for these reasons i forgo the intel chips. most complete purchase solutions that target these use cases will have low power intel atom or even arm processors.

(note: i do have intel chips in other computing devices and also i'm typically not a fan of ford at all).