Poor ZFS RAIDZ2 performance - please help

[dscott]

Hi,

I recently built my first FreeNAS based NAS using an HP N36L with 8GB RAM and 4 x Samsung HD204UI 2TB drives configured as a RAIDZ2 pool.
I'm running FreeNAS 8.0.2-RELEASE-amd64 installed to a 4GB flash drive.
The drives in the pool were added with the "force 4K sectors" check box *not* ticked.
I have no additional cards installed (e.g. separate NIC, remote access card etc.)

I'm pretty happy with the stability and robustness of the system (after having two real-world tests of RAIDZ2 resilience early on in its life!) but I'm not happy with basic read/write performance.

In order to investigate this I first carried out network speed tests using iperf and I'm happy with the 920Mb/s transfer rates I'm seeing over the gigabit connection, so I'm pretty sure the problems aren't network related.

I then tried some disk read/write tests using DD and the results I'm seeing are very poor. For example, for writing I get:

Code:

# dd if=/dev/zero of=/mnt/volume1/test.dat bs=8M count=2000
2000+0 records in
2000+0 records out
16777216000 bytes transferred in 350.594035 secs (47853684 bytes/sec)

i.e. around 45MB/s

And for reading:

Code:

# dd if=/mnt/volume1/test.dat of=/dev/null bs=8M count=2000
2000+0 records in
2000+0 records out
16777216000 bytes transferred in 77.037916 secs (217778685 bytes/sec)

i.e. around 207MB/s

Am I right in being disappointed with these results? The read performance isn't too bad but the write speed looks very poor to me.

I've read on this forum that the performance difference between 512B and forced 4K sector is minimal so I do not suspect that is a major factor here.

I've also read about ZFS pre-fetch but given that I have 8GB I assume that this is enabled automatically? (I also assume that this would affect read performance rather than write performance?)

Where do I go next? Can anyone please suggest what the problem might be?

Also it would be useful to see what typical and best-case performance figures for my setup should be (from what I've seen the N36L with 8GB RAM and using Samsung HD204UI drives seems to be a popular configuration).

All help greatly appreciated.

Regards, Darren

 

[jgreco]

I don't know about the HD204UI drives. However, I can report that our N36L writes at about 60MB/sec, reads around 110MB/sec with some older 7200rpm 750GB drives (sequential) on a ZFS RAIDZ2. I'd say it is within the ballpark. Have you looked at whether or not you've got write caches enabled?

 

[dscott]

Thanks for the reply jgreco. Just before reading your reply I also happened to come across this thread:

http://forums.freenas.org/showthread.php?4713-Slow-RAIDZ1-with-4xWD-Advanced-Format-drives

which also mentions the write cache and the difference it can make to write performance.

After running the diagnostic commands stated in that thread it appears as though I don't have write caching enabled. So, I'll try enabling that now and report back on the subsequent results.

 

[dscott]

After enabling write caches in the BIOS I'm now seeing improved write performance:

Code:

# dd if=/dev/zero of=/mnt/volume1/test.dat bs=8M count=2000
2000+0 records in
2000+0 records out
16777216000 bytes transferred in 229.200484 secs (73198868 bytes/sec)

i.e. up from 45MB/s to 70MB/s

This is a welcome increase but still not as much of an improvement as I've read others benefiting from after enabling write caching. Are the DD parameters I'm using performing a realistic test of performance?

Is there anything else I need to check or any further tuning which would be beneficial?
What's the best read/write speed I can realistically expect with my system configuration?

 

[jgreco]

If you've checked both the BIOS cache setting and the drive cache setting, you're getting close to as good as you're going to easily get on your existing hardware and configuration choices.

A RAIDZ2 on 4 drives is probably a suboptimal configuration; RAIDZ probably works better, at the obvious cost in redundancy. If you're at a point where you haven't done anything to your NAS, might be worth a try just to compare. Going to a 5 drive configuration (with a drive in the optical bay) may help you get a better RAIDZ2 layout, don't know for sure. Also, ZFS benefits significantly from its ZIL, if you wish to throw a small SSD at it. The downside there is that the loss of the SSD can render the pool unusable. Sigh.