Best practise with 24 drives

[ktsiv]

Hello everybody,
On a 24 drive system, wanting raidz2 , what is the best practise for speed?
One pool with 24 drives and raidz2?
Or something different?

 

[sretalla]

The recommended maximum width of RAIDZ2 is 12... even that is probably a bit much. 3x8 is generally a good configuration (I run with that).

Anticipating the question "is that really a limit?"... technically you can go much higher and indeed 24 disks in a RAIDZ2 is possible... you will be running a pool with 24 disks and throughput of somewhere close to that, but the IOPS of one single drive (can be around 100 IOPS... that's abysmal (although with 3x8 you're still only going to get 300... you could go to 12x2 for 1200). Resilver performance is also terrible the wider you go.

Also be aware of the tradeoff you're making...
3x8 loses 6 drives to parity ~300 IOPS
4x6 loses 8 drives to parity ~400 IOPS
12x2 loses 12 drives to parity ~1200 IOPS
1x24 loses 2 drives to parity ~100 OPS

Also means you can lose all parity drives and still keep your pool going... you just need to be lucky and not lose more than 2 in any single VDEV.

 

[danb35]

wanting raidz2 , what is the best practise for speed?

Not using RAIDZ2. Double parity means that every write has to involve at least three disks, and the necessary calculations slow it down even further. And using multiple vdevs, as would be recommended for greater IOPS, means even greater write amplification. If performance is your goal, mirrors are your answer.

 

[ktsiv]

thanks
danb35 you mean 12x2 loses 12 drives to parity ~1200 IOPS , as sretalla said?

 

[Matt_G]

What Danb35 is saying is; if performance is important to you, you should go with 12 vdev's of 2 disks each, which are mirrored.
That would give you 1200 IOPS estimated and have 12 disks worth of usable capacity.

If performance is not important to you, and you don't want to lose 50% of your drives to mirrors, then by all means use RAIDZ2.
As sretella said though, I would not recommend a single vdev of 24 disks.
At the very least, make 2 RAIDZ2 vdev's of 12 disks each. (200 IOPS and losing 4 disks to parity.)
3 RAIDZ2 vdev's of 8 disks each would be better. (300 IOPS and losing 6 disks to parity.)

How big are the drives you're planning to use and what is the purpose/goal of this build?
That info would allow people to make better specific recommendations.

 

[ktsiv]

Thank you all.
Appreciated.