RAIDZ Config With 16 x 8TB Disks

[Fabio Rodrigues]

Hello,

I need to setup a FreeNAS server with 16 disks of 8TB. I would like to know what is the best way to configure my array, I need to have at least 95TB available for data.

Thanks.

 

[toadman]

Hard to say what is "best" without knowing the workload.

Your 95TB requirement rules out using mirrors. And I wouldn't use RAIDZ. So you are at RAIDZ2 or RAIDZ3. Seems like the following are "options"...

1 x 16 disk RAIDZ3 104TB raw. Most will tell you not to do this as vdevs over 11 disks are discouraged.
2 x 8 disk RAIDZ2 96TB raw.
4 x 4 disk RAIDZ2 64TB raw.
3 x 5 disk RAIDZ2 72TB raw.

Only 2 fit your 95TB requirement. Seems like you''d want to go 2 x 8 disk RAIDZ2 based on that one requirement.

**EDIT: Please note that 96TB raw *does not* equal 95TB available for "data." You'll probably get about 82 TB out of it. If your data can compress (use lz4) at 1.17 or better then you'll effectively have 95TB of capacity.

**EDIT 2: I assume you know that you shouldn't fill the pool over 80%! So that 82TB above is really a net usable 65TB. Compression would have to yield 1.47 to net you 95TB. Seems like you may just have to use more disks (if you can't hit that compression level). Even if you used a single 16 disk RAIDZ2, with the 80% full limit you can't hit your 95TB. 1 x 16 RAIDZ2 = 112TB raw, likely 95TB effective, and only 76TB with the 80% full constraint.

 

[Ericloewe]

For reference: https://forums.freenas.org/index.php?threads/zfs-raid-size-and-reliability-calculator.28191/

 

[toadman]

For reference: https://forums.freenas.org/index.php?threads/zfs-raid-size-and-reliability-calculator.28191/

Yep, here is the link to the actual app. Was not working for me. I just used some basic approximations. :)

http://biduleohm.free.fr/zfsraidsarc/

 

[Fabio Rodrigues]

Hard to say what is "best" without knowing the workload.

Your 95TB requirement rules out using mirrors. And I wouldn't use RAIDZ. So you are at RAIDZ2 or RAIDZ3. Seems like the following are "options"...

1 x 16 disk RAIDZ3 104TB raw. Most will tell you not to do this as vdevs over 11 disks are discouraged.
2 x 8 disk RAIDZ2 96TB raw.
4 x 4 disk RAIDZ2 64TB raw.
3 x 5 disk RAIDZ2 72TB raw.

Only 2 fit your 95TB requirement. Seems like you''d want to go 2 x 8 disk RAIDZ2 based on that one requirement.

**EDIT: Please note that 96TB raw *does not* equal 95TB available for "data." You'll probably get about 82 TB out of it. If your data can compress (use lz4) at 1.17 or better then you'll effectively have 95TB of capacity.

**EDIT 2: I assume you know that you shouldn't fill the pool over 80%! So that 82TB above is really a net usable 65TB. Compression would have to yield 1.47 to net you 95TB. Seems like you may just have to use more disks (if you can't hit that compression level). Even if you used a single 16 disk RAIDZ2, with the 80% full limit you can't hit your 95TB. 1 x 16 RAIDZ2 = 112TB raw, likely 95TB effective, and only 76TB with the 80% full constraint.

Is it that bad to use 1 x 16 disk RAIDZ3 or 1 x 16 disk RAIDZ2? The server will be used for 1 year only and will be replicated to another server with same configuration.

 

[toadman]

Is it that bad to use 1 x 16 disk RAIDZ3 or 1 x 16 disk RAIDZ2? The server will be used for 1 year only and will be replicated to another server with same configuration.

I would read up on using vdevs over 11 disks. https://forums.freenas.org/index.ph...stions-about-vdev-configuration-please.19809/

See post #16.

I mean, sure, technically it is allowed and would likely work. The issues come in when something goes wrong. Hence the common advice against really wide vdevs. But again, it depends on the workload and many other things, like # and frequency of backups, etc.

 

[Fabio Rodrigues]

I would read up on using vdevs over 11 disks. https://forums.freenas.org/index.ph...stions-about-vdev-configuration-please.19809/

See post #16.

I mean, sure, technically it is allowed and would likely work. The issues come in when something goes wrong. Hence the common advice against really wide vdevs. But again, it depends on the workload and many other things, like # and frequency of backups, etc.

Thanks!

What would be better (or not as bad)?
1) 1 x 16 disks RAIDZ2 or
2) 2 x 8 disks RAIDZ

 

[toadman]

The problem with RAIDZ is resilver time. If you do the math on the time it takes to resilver, and view that against the chance you will have a second disk go bad (and thereby destroy your pool) you will see that it doesn't add up. The resilver time on an 8TB is way too long vs. the MTBF of the other 7 disks. RAIDZ is just not an option in practical terms.

So 1).

However, I do not have experience with super wide vdevs. I think they too have resilver time issues. So RAIDZ2 might not be a good idea either with something that wide. (EDIT: with some searching I think you can find a calculator online that will tell you if you are likely to hit problems just from an error rate / resilver time perspective. You plug in the drive capacity, number of drives and number of redundant drives and it lets you know the probability of failure.)

So while you can obviously ignore the advice, I still think you (most likely) need more disks for a reliable system at that level of data capacity. :)

 

[toadman]

https://www.servethehome.com/raid-calculator/raid-reliability-calculator-simple-mttdl-model/

The probability of data loss with RAIDZ3 is many orders of magnitude better than RAIDZ2 (RAID6).