Help me increase ZPOOL performance please!

[xic044]

Hello guys,

I have been trying so hard to tune my FreeNAS box for while now, but it only get worse than the autotune defaults. :(

This is my setup:

FreeNAS-9.1.0-RELEASE-x64 (dff7d13)
AMD Athlon(tm) 64 X2 Dual Core Processor 6000+
Memory 4073MB (only identify this out of 4GB don't know why as I have no video board installed)

6 x Seagate 500MBs 5400rpm in RAID-Z2 (I want to keep the z2 as this drives are stinky)

Intel Pro 10/100/1000

On a iSCSI zvol I'm getting average 30MB/s..

on autotune:

[root@freenas] /mnt/pool1# time dd if=/dev/zero of=./file.out bs=1M count=10k
10240+0 records in
10240+0 records out
10737418240 bytes transferred in 67.044975 secs (160152468 bytes/sec)
0.047u 19.542s 1:07.91 28.8% 25+2724k 325+0io 0pf+0w

With tunning described here (http://icesquare.com/wordpress/how-to-improve-zfs-performance/):
Ajusted for 4073M RAM

[root@freenas] /mnt/pool1# time dd if=/dev/zero of=./file.out bs=1M count=10k
10240+0 records in
10240+0 records out
10737418240 bytes transferred in 178.566668 secs (60131145 bytes/sec)
0.063u 17.551s 3:00.65 9.7% 25+2739k 648+0io 0pf+0w

Can anyone help me to get the max out of my setup please?

 

[gpsguy]

With 4Gb of RAM and an ancient machine, that's probably all you'll get. While you don't have a video card installed, I presume it might have an onboard one, that's stealing a little RAM.

With less than 4Gb, caching is disabled on your server.

With iSCSI, you'll need a minimum of 8Gb RAM, perhaps even more. Search the forum for other threads on iSCSI performance.

At the end of the day, you'll need to upgrade your hardware.

 

[xic044]

gpsguy,

This motherboard has no onboard video.

Regarding my hardware, you saying that there is nothing I can do, right? Just for me to understand, why then The RAM memory usage never went too far from 2GB utilization ( the and swap partition was never used (12GB - 2GB each disk). CPU levels never got to high either. I understand that with top hardware the things would be easier but it hard to believe that I am stuck because of hardware when I can see that the utilization is at the edge.

Anyway thanks for you opinion.

 

[Michael Wulff Nielsen]

Maybe try with an intel nic? You might have a bottleneck if you have a Realtek card. And 4 gigs of ram wouldn't hurt :)

 

[cyberjock]

Here's where the manual helps...

The best way to get the most out of your FreeNAS® system is to install as much RAM as possible. If your RAM is limited, consider using UFS until you can afford better hardware. FreeNAS® with ZFS typically requires a minimum of 8 GB of RAM in order to provide good performance and stability. The more RAM, the better the performance, and the FreeNAS® Forums provide anecdotal evidence from users on how much performance is gained by adding more RAM. For systems with large disk capacity (greater than 8 TB), a general rule of thumb is 1 GB of RAM for every 1 TB of storage. This post describes how RAM is used by ZFS.

If you don't have at least 8GB of RAM with ZFS or 2GB of RAM with UFS, you should consider getting more powerful hardware before using FreeNAS® to store your data otherwise data loss may result.

I'm not sure what isn't clear about that message. You have 1/2 of the minimum recommended. Not surprisingly, you have performance problems. You aren't going to win without doing some upgrades.

 

[Yatti420]

For anybody considering less then 8gb of ram try Nas4free maybe.. It should be more lightweight..

 

[xic044]

Hi Michael,

Yes I am using an Intel NIC with jumbo frames at 9k. I think the issue is with the zfs zpool itself (160152468 bytes/sec on time dd if=/dev/zero of=./file.out bs=1M count=10k ). Tried some manual tuning but just got worse. People can point out to my 4GB RAM, but I don'[ t see being fully utilized. The system didn't even had to page memory to swap yet.

I feel that I need help tuning the system to fit my hardware..

 

[xic044]

cyberjock,

I am not doubting that 8GB minimal is the ideal. My question is, if my 4GB RAM is the zfs bottle neck why I don't see it getting fully used? Why the swap is not being used at all?

 

[cyberjock]

You shouldn't be using swap, ever. If you are using swap that means your system "should" have crashed when you ran out of RAM and you have a hail mary pass right now because you had the swap space.

Performance will tank so badly if you start using swap you might question if your server is on. One person in IRC last night said that when he starts using swap space it goes from normal speeds to "404-server not found" errors.

This isn't like Windows where using swap is considered to be normal. If you are using swap anywhere else except Windows you've f'ed up. Also, the utilization numbers aren't the true numbers. If you read up on how all this RAM stuff gets allocated in FreeBSD some RAM that is "free" is being used by ZFS' cache and its released when another application needs it. So don't let that "free" RAM confuse you. In fact, Windows does the same thing. Most of the "free" RAM is actually used by the Windows cache. It just dynamically allocates the cache as programs need more RAM. There' no point in letting RAM not be used as that's simply unused resources.

 

[jyavenard]

You shouldn't be using swap, ever. If you are using swap that means your system "should" have crashed when you ran out of RAM

Are you only referring to the use of swap on FreeNAS or is that a broad statement related to the use of swap in general ?

 

[cyberjock]

From what I've read this is pretty much true of all servers as the swap space doesn't provide ECC protection, so you lose a lot of reliability(not to mention the potential performance implications). I've read that in Linux its consider a failure of a linux admin if you are actually having to use swap space and FreeBSD uses it only if your apps are trying to allocate more RAM than you actually have on the machine. Note that I'm not too heavy on Linux so I'm not too familiar with how memory is allocated in Linux. I know its MUCH simpler than Windows but that's about all I know.

This is one of the reasons why when you go to edit the swapspace field on FreeNAS it says something like "STRONGLY DISCOURAGED".

 

[jyavenard]

Reason I asked is because, swap isn't used only when you run out of ram. Certainly not the case on most modern OS.

Say your application hasn't been active for a while, and it had allocated a fair chunk of RAM. That application will be paused and its memory content will be flushed on swap until the system needs it again.

This frees up memory for applications that would require memory now.

Some OS are more aggressive than others when it comes to the use of swap space. OSX and Windows do it more than others. That's why you often see Windows or OSX allocating GBs of swap space even when you have free RAM left.

Linux dumps pages regularly on swap.

As they state better than I would:
http://www.linux.com/news/software/applications/8208-all-about-linux-swap-space

a significant number of the pages used by an application during its startup phase may only be used for initialization and then never used again. The system can swap out those pages and free the memory for other applications or even for the disk cache.

It is still the default on most linux distribution to create a swap that is the same in size as your amount of memory.
FreeBSD is even more hungry, and allocates by default a swap partition that is twice the size of your RAM.
Their tuning guide states:

Size swap space to approximately twice the size of main memory on systems with less than 4GB RAM and the size of main memory for systems with more than 4GB. If in doubt, allocate more swap; allocating insufficient swap is far worse than allocating too much. If the system has multiple disks, reduce swap I/O contention by spreading swap across the disks, ideally in equally sized partitions.

It has other advantages, it prevents a system to system crash because you have a runaway daemon leaking over time....
So even on system with a lot of RAM, use of swap can, under some circumstances speed up the reactivity of a system.
On laptop, swap size is critical as this is usually where the memory is dump to when suspending to disk...
sorry for derailing the thread again :)

 

[cyberjock]

I don't really know how to respond. The linux server admin I know that's been doing it since the early 90s said that swap should never be used on production machines where data corruption in RAM is not acceptable.

I know I've talked to the a developer in IRC here and they said that if FreeNAS is using any swapspace something is terribly wrong. This stems from the fact that your services should use a small fraction of your RAM, and ZFS will use the rest. Normally ZFS should automatically downsize its cache so the swapspace should never be used. This is why we have people with 40TB of storage, 8GB of RAM, and they complain about poor performance. By virtue of how FreeNAS works you can get a "working" system that works with very little RAM, although the performance might make you want to jump of a nearby bridge.

/storytime

I had a 30TB pool and 12GB of RAM(back in January or February 2013). Everything performed acceptably for me(single user at home) so I didn't buy more RAM since the performance was fine. I was saturating Gb LAN. But something happened one night. I went to bed and the next morning when I woke up I couldn't get a single movie to stream. Performance of the pool was absolutely horrible. The server was unusable. And don't even try to copy a 10MB file to the server while streaming a very low bitrate video as you could expect it to freeze the video for 10-30 seconds.

After 2 weeks of desperate efforts to figure out what was wrong I added an 8GB stick to bring the total RAM to 20GB. Poof, dropped the RAM in and performance was instantly back to normal. I was convinced something had broken(leading to 2 weeks of troubleshooting) because you don't go from a pool that could internally do 1GB/sec to barely being able to do 5MB/sec overnight without something breaking, right? Turns out I was wrong. pebkac. My system never used swapspace at any time, which only added to our confusion(and lead to some light being shed on the swap space on my part). Normally, a failing disk in a pool will cripple a pool's performance like I was seeing. But I couldn't prove any disks were bad(which lead me to dropping money on a stick of RAM) despite my financial situation at that time.

I know on FreeNAS, if we had a runaway daemon you'd have SERIOUS performance implications on the ZFS side of the house before you'd use swapspace. ZFS would expect to keep decreasing itself until your pool performance was complete poop.

/endstorytime

We have had a handful of users that have had swapspace being used. Usually we saw swapspace being used on systems that weren't necessarily short on RAM but had major configuration issues or a corrupt boot disk. They'd have weird nondescript errors on bootup, weird errors in the log, and usually they didn't know how to properly configure FreeNAS. One person had 8GB of RAM, but was running active directory(which the manual says needs more RAM) and a few other things that made his configuration unique.

I chatted with one of the FreeBSD gurus a few months back in IRC and he said the same thing about swap. A properly managed server should not be using swap space ever. In specialized cases you overdesign the system and disable swap as its better to crash the system than run without the ECC protection. But those are usually highly specialized setups where you smartly plan ahead for that possible scenario. Still in others you leave the swap enabled but setup the system to email you if you start using swap so you can order more RAM immediately.

If you were on IRC last night someone was discussing their faux pas with ZFS' dedup. He said that he tried it not realizing how bad dedup is for systems with low RAM. He said the webGUI would go from operating normally to 404-not found in a split second as swap would start being used for the ddt. Probably not the norm as its a very well known fact that the ddt is quite active all of the time and there's no way you are going to have a machine that is performing fine with the ddt on swap space. Got a good laugh even if he did lose his pool when he made that mistake a year or so ago. I think he said he was using NAS4Free though, but the ddt still can't go to swap ever or you'll be doing the /sadface.

Around these parts for FreeNAS if anyone says they are using swap space its a dead giveaway that they:

1. Need more RAM
2. Need to fix their configuration problems
3. #1 and #2

 

[xic044]

Ok guys,

just installed 4GB more RAM (8GB now)and ZFS write still very low. (tnned like this: http://icesquare.com/wordpress/how-to-improve-zfs-performance/)Please help.

[root@freenas] /mnt/pool# time dd if=/dev/zero of=./file.out bs=1M count=10k
10240+0 records in
10240+0 records out
10737418240 bytes transferred in 177.433551 secs (60515152 bytes/sec)
0.031u 14.607s 2:57.43 8.2% 25+2713k 0+1io 0pf+0w

[root@freenas] /mnt/pool# time dd if=/dev/zero of=./file.out bs=512k count=10k
10240+0 records in
10240+0 records out
5368709120 bytes transferred in 63.525458 secs (84512718 bytes/sec)
0.047u 6.832s 1:03.57 10.8% 25+2716k 0+0io 0pf+0w

 

[cyberjock]

Have you tried getting rid of all of your tuned settings and just using FreeNAS with the default config? I've found that FreeNAS really shouldn't need tuning normally.

Also, when you do those dd tests did you run top? If your CPU usage is at 100% that might be all your system can do.

I noticed you mentioned using autotune. Autotune is only for very large system >64GB of RAM. Using it will artificially limit the RAM available for ZFS, so I recommend you disable it. Note that you will have to disable autotune and then delete any tunables/systctls created manually.

 

[xic044]

As soon as I got the extra RAM , I destroyed my zpool and intalled FreeNAS again from scratch, this time disabling swap and autotune (as advised). With default settings performance was not good compared to what people has posted ( using that dd command as benchmark). With the tunning got worse..

My setup:

FreeNAS-9.1.1-RELEASE-x64 (a752d35)
AMD Athlon(tm) 64 X2 Dual Core Processor 6000+
RAM 8170MB <----- Could not find what component is getting 22MB out of my memory as this mobo has no buit in video card.

6 x 500GB SATA 2 (3GB/s) hard disk 5400 rpm

PS- I have no important data in my zpool, so any radical sugestion is welcome.

Anyway thanks for all the attention on this matter.

 

[TenzoR]

I noticed you mentioned using autotune. Autotune is only for very large system >64GB of RAM.

That's an interesting bit I didn't know about!

 

[cyberjock]

As soon as I got the extra RAM , I destroyed my zpool and intalled FreeNAS again from scratch, this time disabling swap and autotune (as advised). With default settings performance was not good compared to what people has posted ( using that dd command as benchmark). With the tunning got worse..

My setup:

FreeNAS-9.1.1-RELEASE-x64 (a752d35)
AMD Athlon(tm) 64 X2 Dual Core Processor 6000+
RAM 8170MB <----- Could not find what component is getting 22MB out of my memory as this mobo has no buit in video card.

6 x 500GB SATA 2 (3GB/s) hard disk 5400 rpm

PS- I have no important data in my zpool, so any radical sugestion is welcome.

Anyway thanks for all the attention on this matter.

I never told you to disable swap. The GUI tells you "NOT RECOMMENDED" on purpose. That should be left alone. That's not going to affect performance, but you should go back and readd it. What I said is that it shouldn't be used by the system. If it is being used you need to upgrade your RAM. In your case, that means a whole new system(which I think you should do anyway if you want more performance).

When you do a dd write and read from the pool what does top say for CPU usage? Is it near 100%? That CPU is from Q1 2007. In computing years that's so far beyond obsolete...

And what speeds do you get with dd write and dd read?

 

[xic044]

cyberjock, Dec 5, 2013

You shouldn't be using swap, ever. If you are using swap that means your system "should" have crashed when you ran out of RAM and you have a hail mary pass right now because you had the swap space.

CPU usage goes to 50% and RAM usage steady at 4GB



I am going to enable swap back as I got your statment wrongly...

 

[cyberjock]

Yeah, I meant that the system shouldn't be using it, but it should be there.