RAIDZ2 with 12 drives ??

[Vincèn]

Hi

After reading numerous documentations here (side note: the forum is really a great ressource for beginners with FreeNAS like me and users very helpful :)), it looks like I should not go with more than 11 drives in a pool. Is it still the case ?

If so I guess I should setup 2 pools of 6 drives each one in RAIDZ2 to keep system safe (till up to 2 drives dead in each pool) right ? Will it change whole capacity avalaible to do two pools instead of one ?

Thanks,

 

[depasseg]

I think it depends on the rest of your system specs and how you plan to use the storage. You might be able to get away with 12.

I think you mis-used the term "pool", I think you mean vdev. You could have a pool made up of 2 6-drive RAID-Z2 vdevs. You would have 8 drives worth of usable space though, instead of 10 (for a 12-drive RAID-Z2).

 

[Vincèn]

I think it depends on the rest of your system specs and how you plan to use the storage. You might be able to get away with 12.
I think you mis-used the term "pool", I think you mean vdev. You could have a pool made up of 2 6-drive RAID-Z2 vdevs. You would have 8 drives worth of usable space though, instead of 10 (for a 12-drive RAID-Z2).

Yep sorry not yet fully familiar with all terms so definitively vdev and not pool ;) I'll use Super Micro MBD-X10SL7-F-O with 16Gb of RAM at start and Xeon E3 1230V3 and quickly upgraded to 32Gb (ECC RAM for sure :) Does it answer your question about system specs ? and if 12 DD vdev is safe or not ?

 

[Robert Trevellyan]

and if 12 DD vdev is safe or not ?

Clearly a single 12-drive RAIDZ2 vdev is not as safe as a pair of 6-drive RAIDZ2 vdevs, because there are fewer drives dedicated to redundancy. Only you can know whether it's safe enough for your intended usage. You should also expect better performance from a pair of 6-drive vdevs than from a single 12-drive vdev, but again, only you can decide whether performance is good enough.

One of the original developers of ZFS wrote a post which he summarizes thus:
"Use RAID-Z. Not too wide. Enable compression."
but the meaning of "too wide" is entirely dependent on the application (and when he says RAID-Z here he means 'an appropriate level of redundancy', not 'single-disk redundancy').

 

[Vincèn]

Clearly a single 12-drive RAIDZ2 vdev is not as safe as a pair of 6-drive RAIDZ2 vdevs, because there are fewer drives dedicated to redundancy. Only you can know whether it's safe enough for your intended usage. You should also expect better performance from a pair of 6-drive vdevs than from a single 12-drive vdev, but again, only you can decide whether performance is good enough.

Thanks for the very interesting link so it looks like unless I'll lose usable space, I should go definitively with two devs of 6 drives to keep safety on my side :)

Thanks for help,

 

[cyberjock]

I did a 16 disk RAIDZ3 up until a year ago. I wanted to see how much it would suck (or not suck) as a home user. Often recommendations are made for businesses and can be ignored by home users. In this case, I had performance issues that just sucked. I was glad to get rid of that zpool when I did. I am glad I don't have that zpool, and I stand fast to the recommendation of not to go beyond 11 drives. So my advice would be go go with 2 vdevs that are 6 disks each in a RAIDZ2.

 

[DifferentStrokes]

Yep sorry not yet fully familiar with all terms so definitively vdev and not pool ;) I'll use Super Micro MBD-X10SL7-F-O with 16Gb of RAM at start and Xeon E3 1230V3 and quickly upgraded to 32Gb (ECC RAM for sure :) Does it answer your question about system specs ? and if 12 DD vdev is safe or not ?

Take a look at the E3-1231v3. Take is usually cheaper and is 100 MHz faster.

 

[Vincèn]

I did a 16 disk RAIDZ3 up until a year ago. I wanted to see how much it would suck (or not suck) as a home user. Often recommendations are made for businesses and can be ignored by home users. In this case, I had performance issues that just sucked. I was glad to get rid of that zpool when I did. I am glad I don't have that zpool, and I stand fast to the recommendation of not to go beyond 11 drives. So my advice would be go go with 2 vdevs that are 6 disks each in a RAIDZ2.

Thanks for sharing these very interesting details about your experience so I'll definitively go with 2 x 6 drives RAIDZ3. It means I can loose 4 drives at max (2 in each vdevs) before everything is gone right ? Also if I upgrade all 6 drives in one vdev I'll then benefit of extra space in pool right ?

Take a look at the E3-1231v3. Take is usually cheaper and is 100 MHz faster.

Thanks for advice, but not here in fact, it's a little more expensive (10€) ;)

 

[Robert Trevellyan]

Also if I upgrade all 6 drives in one vdev I'll then benefit of extra space in pool right ?

Right.

 

[Vincèn]

oki so due to technical constraints I'll use only 11 Drives so still better to do two vdev (one with 5 and one with 6) or is ok both for safety and efficiency to go with all 11 drives in one vdev ?

Thanks

 

[Robert Trevellyan]

oki so due to technical constraints I'll use only 11 Drives so still better to do two vdev (one with 5 and one with 6) or is ok both for safety and efficiency to go with all 11 drives in one vdev ?

Thanks

Two vdevs will be more flexible, have more redundancy and deliver higher IOPS. One vdev will give you more storage. Only you can decide which makes sense for you.

 

[Vincèn]

Two vdevs will be more flexible, have more redundancy and deliver higher IOPS. One vdev will give you more storage. Only you can decide which makes sense for you.

Why more flexible ? I'll combine them both in a pool but it looks like unless the 5 drives vdev will not be best for storage optimisation, I'll get better performance building 2 vdevs ;)

Thanks

 

[Robert Trevellyan]

Why more flexible ?

More flexible because, for example, you could increase pool capacity by replacing each drive in one of the vdevs one at a time with larger drives. In this example, you would only have to replace 5 or 6 drives instead of all 11 in order to see the capacity increase.

That's actually the only reason I can think of when both vdevs are in one pool. Two pools is more flexible than one but also requires a bit more work to manage.

Don't concern yourself about optimizing the number of drives in a vdev, it's pointless unless you have a very specialized application (a database with a large fixed record size that doesn't compress well is the most often quoted example).

 

[Vincèn]

More flexible because, for example, you could increase pool capacity by replacing each drive in one of the vdevs one at a time with larger drives. In this example, you would only have to replace 5 or 6 drives instead of all 11 in order to see the capacity increase.
That's actually the only reason I can think of when both vdevs are in one pool. Two pools is more flexible than one but also requires a bit more work to manage.

Would it be more efficient in term of read/write speed access if I keep both vdevs separated each one in a dedicated pool instead of a big pool with both vdevs in it ? unless it'll need mone management work to distribute my datas on two pools instead of a big one !

 

[Robert Trevellyan]

Would it be more efficient in term of read/write speed access if I keep both vdevs separated each one in a dedicated pool

I don't think so.

unless it'll need mone management work to distribute my datas on two pools instead of a big one !

It would be a bit more effort to manage two pools, but again, a bit more flexible too. Not just distributing your data (as if you had two external drives attached to a PC instead of one), but also setting up scrubs, snapshots etc.

 

[Vincèn]

It would be a bit more effort to manage two pools, but again, a bit more flexible too. Not just distributing your data (as if you had two external drives attached to a PC instead of one), but also setting up scrubs, snapshots etc.

Thanks for advices, so will go this way and will allow upgrades of volume in a more easy way ;)