Best RAID layout for 16-disk chassis?

[rkelleyrtp]

Greetings all,

I am building a pair of high-capacity chassis for our datacenter to become our new backup servers. Each server will have a 16-bay Supermicro chassis, 16x 4TB Seagate NAS drives, 32G RAM, a QC 3.4GHz Intel CPU, and a dual-port Intel 10G NIC.

I have done a ton of research with regards to the RAID strategy for the drives, however, it seems there is no optimum configuration for 16-disk chassis. Evidently, 6+2-RAIDZ2 vdevs are not good, nor is a single 14+2 RAIDZ2 or RAIDZ3 vdev.

Can someone please shed some light on the optimum RAID config for a 16-disk chassis that will be used as backup servers? I thought about creating a zpool with different RAIDZ vdevs, but I am not sure if that is best practice.

Thanks.

-Ron

 

[Whattteva]

I think the general rule is like 1 parity drive for every 5 drives, so in your case, you probably want at least RAIDZ3. That should give you plenty of protection against multiple disk failure/read errors you may encounter.

 

[rkelleyrtp]

So, are you suggesting a 13+3 RAIDZ3 setup? I read something about using multiple, smaller vdevs because of potential IOPS issues. Will I have similar issues with the 13+3 setup?

Thanks for the quick feedback!

 

[Z300M]

I think the general rule is like 1 parity drive for every 5 drives, so in your case, you probably want at least RAIDZ3. That should give you plenty of protection against multiple disk failure/read errors you may encounter.

I've never seen that suggestion before. I just set up a new FreeNAS machine following the n^2 + z rule that I've read here somewhere, E.g., 6 drives (2^2 + 2) for a Raidz2 vdev. The next would be 11 drives (3^2 + 2) for RaidZ2 or 12 drives (3^2 + 3) for RaidZ3, etc.

rkelleyrtp might want to assign a couple of drives as hot spares.

 

[titan_rw]

With 16 drives I'd probably do 2 vdevs of 8 disks in raidZ2. If it's a backup server, I don't think you need to worry about the non optimal vdev size.

Is the server readily accessible for drive replacements? If it's not, ie if it's a remote server that's hard or expensive to get to / service, then having warm spares would be a good idea. Maybe 2 vdevs of 7 drives in Z2 with 2 spares? If the server is onsite, or is easy / fast to gain physical access to you may not need the spares running inside the machine. Simply keeping a couple of drives in a safe location as cold spares, ready to go if you need a replacement should be sufficient.

I wouldn't create one big vdev. Resilver times can go through the roof on really wide vdevs. 11 disks (Z3) is the widest I'd be comfortable. And since you have 16 disks, that means at least 2 vdevs.

 

[rkelleyrtp]

With 16 drives I'd probably do 2 vdevs of 8 disks in raidZ2. If it's a backup server, I don't think you need to worry about the non optimal vdev size.

Is the server readily accessible for drive replacements? If it's not, ie if it's a remote server that's hard or expensive to get to / service, then having warm spares would be a good idea. Maybe 2 vdevs of 7 drives in Z2 with 2 spares? If the server is onsite, or is easy / fast to gain physical access to you may not need the spares running inside the machine. Simply keeping a couple of drives in a safe location as cold spares, ready to go if you need a replacement should be sufficient.

I wouldn't create one big vdev. Resilver times can go through the roof on really wide vdevs. 11 disks (Z3) is the widest I'd be comfortable. And since you have 16 disks, that means at least 2 vdevs.

Thanks for the feedback. My thought was 4x 3+1RAIDZ1 vdevs or 2x 6+2RAIDZ2 vdevs. This way, I loose minimal space (4 drives total) plus get decent IOPS.

Which would be more preferable? The 4-disk RAIDZ or the 8-disk RAID2Z?

(forgot to add: the servers are local, so swapping drives is pretty easy. Thus, no need to hot or warm spares).

 

[Whattteva]

I've never seen that suggestion before. I just set up a new FreeNAS machine following the n^2 + z rule that I've read here somewhere, E.g., 6 drives (2^2 + 2) for a Raidz2 vdev. The next would be 11 drives (3^2 + 2) for RaidZ2 or 12 drives (3^2 + 3) for RaidZ3, etc.

rkelleyrtp might want to assign a couple of drives as hot spares.

That doesn't really make sense. How do you even determine "n" and "z"? It sounds too arbitrary because you have 2 variables that you have to determine. You can't really come up with a comfortable number for 16 drives with that set up.
I'm assuming, the equation is n^2 + z = x where n is the effective capacity, z is your dedicated parity number, and x is your total number of drives?
Following that, for OP's x = 16, you would get n^2 + z = 16. You would have to either go 4^2 + 0 = 16 or 3^2 + 7 = 16... Neither of those setups are sensible. One offers no parity, the other offers way too many.

The reason why you want to go 1 for every 5 drives is because as hard drive space rises, for every failure you get, you generally will get a read error as well while you are resilvering. Of course, as hard drive capacity continues to rise, that 5/1 rule is probably going to change also.

Have a look at this link for more information: http://www.zdnet.com/blog/storage/why-raid-5-stops-working-in-2009/162

Also, if cyberjock is around (sorry for bothering you, cyber ), you can probably get more information about this particular topic from him as he is a lot more knowledgeable in this topic.

 

[titan_rw]

Thanks for the feedback. My thought was 4x 3+1RAIDZ1 vdevs or 2x 6+2RAIDZ2 vdevs. This way, I loose minimal space (4 drives total) plus get decent IOPS.

Which would be more preferable? The 4-disk RAIDZ or the 8-disk RAID2Z?

(forgot to add: the servers are local, so swapping drives is pretty easy. Thus, no need to hot or warm spares).

Avoid raidz1 if at all possible. It only gives you one disk protection within that vdev. So if a disk fails, you're relying on the remaining 3 disks to operate at 100% health with no unreadable sectors, and no data returned corrupted. The only advantage to the 4x z1 config would be random io speed. And for a backup server, you really don't need random io speed. Sequential io speed would probably be similar between the z1 and z2 setup.

If the server is local, then I wouldn't bother with online spares. Simply keep a drive or two in a drawer ready to go in case one starts to fail.

 

[cyberjock]

RAIDZ1 is just crazy in this day and age.

And the appropriate equation for "optimal" vdevs is

n^2 + y = z

n = any whole number except 1
y = number of parity drives you'd want( 1 for RAIDZ1, 2 for RAIDZ2, 3 for RAIDZ3)
z = total number of disks.

For home use it really doesn't matter much if you are non-optimal. For work use I'd stick with optimal.

I'd do 2 RAIDZ2 vdevs of 8 disks each if I were in your shoes.

 

[Z300M]

RAIDZ1 is just crazy in this day and age.

And the appropriate equation for "optimal" vdevs is

n^2 + y = z

n = any whole number except 1
y = number of parity drives you'd want( 1 for RAIDZ1, 2 for RAIDZ2, 3 for RAIDZ3)
z = total number of disks.

For home use it really doesn't matter much if you are non-optimal. For work use I'd stick with optimal.

I'd do 2 RAIDZ2 vdevs of 8 disks each if I were in your shoes.

I thought that I had read that FreeNAS 9.* will not permit one to create non-optimal vdevs.

 

[hotalot]

Thanks for the feedback. My thought was 4x 3+1RAIDZ1 vdevs or 2x 6+2RAIDZ2 vdevs. This way, I loose minimal space (4 drives total) plus get decent IOPS.

Which would be more preferable? The 4-disk RAIDZ or the 8-disk RAID2Z?

(forgot to add: the servers are local, so swapping drives is pretty easy. Thus, no need to hot or warm spares).

8-disk Raidz2 is preferable, but you can add 2 more drives and have a pool with 3 vdevs of 6-disk raidz2. You can use the side panels of the supermicro case to lodge the 2 extra drives.