I'm setting up a file server with 4 HDs. I decided to try different zfs and raidz configurations and study their performance before I choose my final setup. My hardware is powerful enough so I can be sure that I'm not limited by them:
FreeNAS-8.0.2-RELEASE-amd64 (8288)
Intel Desktop Board DH55TC (with 6 SATA-2 ports)
Intel Core i5-760 (2,8GHz/8MB) (s1156), 4 cores
4 Seagate SATA disks, as:
ada0: Seagate Barracuda Green (Adv. Format) Device Model: ST2000DL003-9VT166
ada1: Seagate Barracuda Green (Adv. Format) Device Model: ST2000DL003-9VT166
ada2: Seagate Barracuda LP Device Model: ST32000542AS
ada3: Seagate Barracuda LP Device Model: ST32000542AS
1. First, I tested the performance of the individual disks. I created 4 simple zfs volumes on the 2 pairs of identical disks, with the "Force 4096 bytes sector size" box checked in the case of ada1 and ada3, and unchecked in the case of ada0 and ada2. Then, in bash, I run a slightly modified version of the usual dd test (which is used throughout this forum) on each volume:
# time ( dd if=/dev/zero of=tank/tmp.dat bs=2048k count=50k ; sync ; )
# time ( dd if=tank/tmp.dat of=/dev/null bs=2048k count=50k ; sync ; ); rm tank/tmp.dat
(I believe, including a final sync command in the measured time makes a more precise estimate of the actual transfer rate, but, in this case with 100GB output files, the difference is not significant.)
Results:
ada0: write 126706521 bytes/sec, read 129541367 bytes/sec (127MB/s, 130MB/s)
ada1: write 134621807 bytes/sec, read 137069217 bytes/sec (135MB/s, 137MB/s)
ada2: write 93163187 bytes/sec, read 114053737 bytes/sec (93MB/s, 114MB/s)
ada3: write 96067721 bytes/sec, read 120459644 bytes/sec (96MB/s, 120MB/s)
It seems that all drives work well, and the system is capable of producing decent results. These numbers are probably the maximum rates of the drives. The first two drives are faster than the last two. It is, however, surprising, that the forced 4k alignment does not significantly affect the results. It probably means, that Seagate's SmartAlign technology (to cure this alignment problem) works as they advertise it.
2. Second, I made a single raidz1 volume including all 4 drives with the web GUI.
The transfer rates were shocking to me:
write: 58848427 bytes/sec, read: 325934858 bytes/sec (59MB/s, 326MB/s)
Though the read speed has increased nearly by a factor of 3 (as expected), the write speed has fallen nearly a factor of 2 in the 3+1 raidz1 configuration!
3. I started reading this forum and googling. I've learnt that the 4 disk configuration is not ideal for raidz1, it should consist of 2^N+1 disk. Since I had no 5th disk at hand, I tried the 3-disk configuration. So I made a 3 disk raidz1 volume and tested it:
write: 212814474 bytes/sec, read: 220332537 bytes/sec (213MB/s, 220MB/s)
Wow! What a difference. Now, both the writing and reading speeds are doubled compared to the single disk performance (again, as expected).
4. It seems that due to the tremendous difference in the write speeds, I have to stick with the ideal configurations in terms of disk numbers. The ideal configuration for raidz2 is 2^N+2, so I tested my 4 disks in a 4 disk raidz2 volume:
write: 63115133 bytes/sec, read: 219386411 bytes/sec (63MB/s, 219MB/s)
These results are almost as bad as for the 3+1 disk raidz1 setup.
Can someone explain, what's happening here? Is it normal to have such a great difference in write speeds for the 3 and 4 disk raidz setups? Why doesn't the 4 disk raidz2 setup work well, if all disks do and even the 3 disk raidz1 setup does?
Thanks for reading this long post. Please, share your thoughts.
FreeNAS-8.0.2-RELEASE-amd64 (8288)
Intel Desktop Board DH55TC (with 6 SATA-2 ports)
Intel Core i5-760 (2,8GHz/8MB) (s1156), 4 cores
4 Seagate SATA disks, as:
ada0: Seagate Barracuda Green (Adv. Format) Device Model: ST2000DL003-9VT166
ada1: Seagate Barracuda Green (Adv. Format) Device Model: ST2000DL003-9VT166
ada2: Seagate Barracuda LP Device Model: ST32000542AS
ada3: Seagate Barracuda LP Device Model: ST32000542AS
1. First, I tested the performance of the individual disks. I created 4 simple zfs volumes on the 2 pairs of identical disks, with the "Force 4096 bytes sector size" box checked in the case of ada1 and ada3, and unchecked in the case of ada0 and ada2. Then, in bash, I run a slightly modified version of the usual dd test (which is used throughout this forum) on each volume:
# time ( dd if=/dev/zero of=tank/tmp.dat bs=2048k count=50k ; sync ; )
# time ( dd if=tank/tmp.dat of=/dev/null bs=2048k count=50k ; sync ; ); rm tank/tmp.dat
(I believe, including a final sync command in the measured time makes a more precise estimate of the actual transfer rate, but, in this case with 100GB output files, the difference is not significant.)
Results:
ada0: write 126706521 bytes/sec, read 129541367 bytes/sec (127MB/s, 130MB/s)
ada1: write 134621807 bytes/sec, read 137069217 bytes/sec (135MB/s, 137MB/s)
ada2: write 93163187 bytes/sec, read 114053737 bytes/sec (93MB/s, 114MB/s)
ada3: write 96067721 bytes/sec, read 120459644 bytes/sec (96MB/s, 120MB/s)
It seems that all drives work well, and the system is capable of producing decent results. These numbers are probably the maximum rates of the drives. The first two drives are faster than the last two. It is, however, surprising, that the forced 4k alignment does not significantly affect the results. It probably means, that Seagate's SmartAlign technology (to cure this alignment problem) works as they advertise it.
2. Second, I made a single raidz1 volume including all 4 drives with the web GUI.
The transfer rates were shocking to me:
write: 58848427 bytes/sec, read: 325934858 bytes/sec (59MB/s, 326MB/s)
Though the read speed has increased nearly by a factor of 3 (as expected), the write speed has fallen nearly a factor of 2 in the 3+1 raidz1 configuration!
3. I started reading this forum and googling. I've learnt that the 4 disk configuration is not ideal for raidz1, it should consist of 2^N+1 disk. Since I had no 5th disk at hand, I tried the 3-disk configuration. So I made a 3 disk raidz1 volume and tested it:
write: 212814474 bytes/sec, read: 220332537 bytes/sec (213MB/s, 220MB/s)
Wow! What a difference. Now, both the writing and reading speeds are doubled compared to the single disk performance (again, as expected).
4. It seems that due to the tremendous difference in the write speeds, I have to stick with the ideal configurations in terms of disk numbers. The ideal configuration for raidz2 is 2^N+2, so I tested my 4 disks in a 4 disk raidz2 volume:
write: 63115133 bytes/sec, read: 219386411 bytes/sec (63MB/s, 219MB/s)
These results are almost as bad as for the 3+1 disk raidz1 setup.
Can someone explain, what's happening here? Is it normal to have such a great difference in write speeds for the 3 and 4 disk raidz setups? Why doesn't the 4 disk raidz2 setup work well, if all disks do and even the 3 disk raidz1 setup does?
Thanks for reading this long post. Please, share your thoughts.
