VDEV raidz2 with five disks

[Hudu]

I read a lot about optimal configurations, but I did not find a concrete listing of disadvantages, when
not using an optimal number of disks. The problem is that the most motherboards have only 6 SATA-ports
and, if you want to use a SSD for OS and jails, there are only five ports remaining.

For raidz2 it is known that 4, 6 or 10 disks are optimal.

I found in other threads that there are two possible penalties: performance and space.

It would be nice to quantify both penalties and to know if this penalties are the only one.

 

[HoneyBadger]

Hi Hudu,

I'll copy and paste from another response I wrote to someone asking about non-optimal drive configuration:

---

Computers really like the number two, powers of two, and things that are divisible nicely by twos and powers of two. So it likes 6 disks because it has 4 drives worth of data (2^2) and 2 of parity.

Let's say you write a 128KB stripe of data to that array. We'll ignore the parity drives for now; with four drives holding data, it divides very nicely into four chunks of 32KB, which go to each disk. And if you're using 4KB-sector drives, each chunk takes up eight sectors. Nice round numbers.

But if you have a 7-drive RAIDZ2, you're going to have five data disks. 128KB/5 = 25.6KB per drive. Uh oh, that didn't divide nicely. You're now writing 6 full sectors (24KB) and one partial sector (1.6KB) to each disk.

---

In your case you'll have a 5-drive RAIDZ2, but you'll get the same "doesn't divide nicely" impact, along with having lower overall performance because of fewer spindles to spread out to.

In cases like this you'll typically get a suggestion to add in an extra HBA to handle the drives.

 

[Hudu]

If the stripe size is the only problem: So use a stripe size of a multiple of sector size (4K) and number of spindles. So a stripe size of 124kb is totally used and is similar to the original 128k.

It is possible to set the stripe size?

 

[melp]

Not yet, but apparently it's being worked on (for sizes larger than 128k): https://github.com/zfsonlinux/zfs/issues/2146#issuecomment-36165759

Here's more info (see question #6): View: https://www.youtube.com/watch?v=G2vIdPmsnTI

 

[HoneyBadger]

You can manually set the zfs recordsize, but it has to be a power of two.

 

[Hudu]

I calculated the information for a larger number of different spindles:

So the suboptimal case with five spindles is not very common, since the stripe size of 128k leads with five disks to 43 sectors per data spindle, which leads to a physical stripe size of 129kB on disks.
If I compare it to the capacity of the data spindles, nearly the whole capacity of fifth disk is usable.

I assume that the I/O-performance of the array under raidz2 will be the fraction 128/129 of the "optimal performance" and still saturates a 1GB-network interface.
For a home server it makes no difference, but gives 3 TB of usable space more than a four spindle setup.

 

[Faluzure]

Hi Hudu,

I'll copy and paste from another response I wrote to someone asking about non-optimal drive configuration:

---

Computers really like the number two, powers of two, and things that are divisible nicely by twos and powers of two. So it likes 6 disks because it has 4 drives worth of data (2^2) and 2 of parity.

Let's say you write a 128KB stripe of data to that array. We'll ignore the parity drives for now; with four drives holding data, it divides very nicely into four chunks of 32KB, which go to each disk. And if you're using 4KB-sector drives, each chunk takes up eight sectors. Nice round numbers.

But if you have a 7-drive RAIDZ2, you're going to have five data disks. 128KB/5 = 25.6KB per drive. Uh oh, that didn't divide nicely. You're now writing 6 full sectors (24KB) and one partial sector (1.6KB) to each disk.

---

In your case you'll have a 5-drive RAIDZ2, but you'll get the same "doesn't divide nicely" impact, along with having lower overall performance because of fewer spindles to spread out to.

In cases like this you'll typically get a suggestion to add in an extra HBA to handle the drives.

Thanks for this explanation. This answered my questions perfectly.