Weird raidz1 and raidz2 performance with 4 drives - any explanation?

Status
Not open for further replies.

tpusztai

Cadet
Joined
Sep 26, 2011
Messages
7
I'm setting up a file server with 4 HDs. I decided to try different zfs and raidz configurations and study their performance before I choose my final setup. My hardware is powerful enough so I can be sure that I'm not limited by them:

FreeNAS-8.0.2-RELEASE-amd64 (8288)
Intel Desktop Board DH55TC (with 6 SATA-2 ports)
Intel Core i5-760 (2,8GHz/8MB) (s1156), 4 cores
4 Seagate SATA disks, as:
ada0: Seagate Barracuda Green (Adv. Format) Device Model: ST2000DL003-9VT166
ada1: Seagate Barracuda Green (Adv. Format) Device Model: ST2000DL003-9VT166
ada2: Seagate Barracuda LP Device Model: ST32000542AS
ada3: Seagate Barracuda LP Device Model: ST32000542AS


1. First, I tested the performance of the individual disks. I created 4 simple zfs volumes on the 2 pairs of identical disks, with the "Force 4096 bytes sector size" box checked in the case of ada1 and ada3, and unchecked in the case of ada0 and ada2. Then, in bash, I run a slightly modified version of the usual dd test (which is used throughout this forum) on each volume:

# time ( dd if=/dev/zero of=tank/tmp.dat bs=2048k count=50k ; sync ; )
# time ( dd if=tank/tmp.dat of=/dev/null bs=2048k count=50k ; sync ; ); rm tank/tmp.dat

(I believe, including a final sync command in the measured time makes a more precise estimate of the actual transfer rate, but, in this case with 100GB output files, the difference is not significant.)

Results:

ada0: write 126706521 bytes/sec, read 129541367 bytes/sec (127MB/s, 130MB/s)
ada1: write 134621807 bytes/sec, read 137069217 bytes/sec (135MB/s, 137MB/s)
ada2: write 93163187 bytes/sec, read 114053737 bytes/sec (93MB/s, 114MB/s)
ada3: write 96067721 bytes/sec, read 120459644 bytes/sec (96MB/s, 120MB/s)

It seems that all drives work well, and the system is capable of producing decent results. These numbers are probably the maximum rates of the drives. The first two drives are faster than the last two. It is, however, surprising, that the forced 4k alignment does not significantly affect the results. It probably means, that Seagate's SmartAlign technology (to cure this alignment problem) works as they advertise it.


2. Second, I made a single raidz1 volume including all 4 drives with the web GUI.

The transfer rates were shocking to me:

write: 58848427 bytes/sec, read: 325934858 bytes/sec (59MB/s, 326MB/s)

Though the read speed has increased nearly by a factor of 3 (as expected), the write speed has fallen nearly a factor of 2 in the 3+1 raidz1 configuration!


3. I started reading this forum and googling. I've learnt that the 4 disk configuration is not ideal for raidz1, it should consist of 2^N+1 disk. Since I had no 5th disk at hand, I tried the 3-disk configuration. So I made a 3 disk raidz1 volume and tested it:

write: 212814474 bytes/sec, read: 220332537 bytes/sec (213MB/s, 220MB/s)

Wow! What a difference. Now, both the writing and reading speeds are doubled compared to the single disk performance (again, as expected).


4. It seems that due to the tremendous difference in the write speeds, I have to stick with the ideal configurations in terms of disk numbers. The ideal configuration for raidz2 is 2^N+2, so I tested my 4 disks in a 4 disk raidz2 volume:

write: 63115133 bytes/sec, read: 219386411 bytes/sec (63MB/s, 219MB/s)

These results are almost as bad as for the 3+1 disk raidz1 setup.


Can someone explain, what's happening here? Is it normal to have such a great difference in write speeds for the 3 and 4 disk raidz setups? Why doesn't the 4 disk raidz2 setup work well, if all disks do and even the 3 disk raidz1 setup does?


Thanks for reading this long post. Please, share your thoughts.
 

Joshua Parker Ruehlig

Hall of Famer
Joined
Dec 5, 2011
Messages
5,949
I can't explain it but thank you for benchmarking and posting your results. I also have been keeping with the 2^N+1 rule with raidz1 (with 5 disks) and have seen linear progression with increased disks which is great. Hopefully others can also benchmark and everyone can know the ideal disk amounts really does matter.
 

William Grzybowski

Wizard
iXsystems
Joined
May 27, 2011
Messages
1,754
Yeah, that is a known issue with zfs and freebsd.

Raidz2 has a performance penalty compared to raidz1, i dont think anybody has looked at this issue yet.
 

HAL 9000

Dabbler
Joined
Jan 27, 2012
Messages
42
Can someone explain, what's happening here? Is it normal to have such a great difference in write speeds for the 3 and 4 disk raidz setups? Why doesn't the 4 disk raidz2 setup work well, if all disks do and even the 3 disk raidz1 setup does?

I've 4 Seagate ST2000DL003 4k disks in my box. Raw speed of these disks is about 130 MB/s.
Single drive formatted with ZFS achieve read/write speed of over 110 MB/S.
With two mirrored disks read speed almost doubles, write speed is just a bit lower.
3 disks configured as RAIDZ-1 can do 130 MB/s write and 260 MB/s read.
Adding 4 disk do array causes write speed to drop to about 60-70 MB/s.

Popular belief is that 4 disk RAID-Z is suboptimal because default 128K record can't be spread equally across 3 data disks.
Above is a bit strange if you notice that after turning on data compression record size is variable and does not cause slow down.
Record size of 96K can't be set because record size has to be a power of 2.
Just for test I've set record size to 64K and...
write speed inceased to 150 MB/s and read speed do 300 MB/s :smile:
I can't explain this behaviour however. Maybe someone will try?
 
Status
Not open for further replies.
Top