mirrored vdevs vs. RAID-Z

[Scampicfx]

Hey,

while googling, I found an interesting article to read. I thought I'd share it with you: http://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-not-raidz/

What's your opinions about that?

When I noticed it correct, there's one point this article doesn't deal with: Parity Benefits of RAIDZ-2 vs. RAID10:
One example: Let's imagine, you're running multiple mirrored vdevs (A0, A1, B0, B1, C0, C1, D0, D1, E0, E1, and so on...). At one day, you encounter disk failure of A0. A few hours later, ZFS detects Biterror in A1 (still degraded status, disk a0 is still getting rebuilt)... My question: How should ZFS handle / repair this biterror, since there is no parity or no mirror data available?
In my opinion, this is one point where RAIDZ-2 wins: even in case of a disk failure, it is still able to recover bit errors since parity information is still available... At least I hope that? How do mirrored vdevs handle this issue?

 

[jgreco]

Hey,

while googling, I found an interesting article to read. I thought I'd share it with you: http://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-not-raidz/

What's your opinions about that?

I (and others) consider it to be a post-by-idiot, who basically ripped off the much better http://constantin.glez.de/blog/2010/01/home-server-raid-greed-and-why-mirroring-still-best

When I noticed it correct, there's one point this article doesn't deal with: Parity Benefits of RAIDZ-2 vs. RAID10: [.....snip.....]In my opinion, this is one point where RAIDZ-2 wins: even in case of a disk failure, it is still able to recover bit errors since parity information is still available... At least I hope that? How do mirrored vdevs handle this issue?

Okay, first off, RAID10 is a classic RAID level, so you can't compare that. Makes no sense. Please see the terminology and abbreviations primer.

If we were to assume that you meant "a two way mirror vdev versus RAIDZ2", I would have to ask your rationality for comparing apples and oranges. The proper comparison would be a two way mirror vdev versus RAIDZ1. Both can handle the loss of one disk without losing data, but in both cases you lose redundancy, so any further errors may be unrecoverable.

If you want to compare apples to apples, then compare a three way mirror vdev versus a RAIDZ2 vdev. In both cases you can lose a single disk without compromising redundancy, or lose two disks at which point you lose redundancy.

You can also go with a four way mirror vdev versus RAIDZ3, where you can lose two disks without compromising redundancy, or three disks at which point you lose redundancy.

RAIDZ loses out beyond that, because you can go with a ten way mirror vdev that allows you to lose any eight disks without compromising redundancy. RAIDZ has no such construct.

One example: Let's imagine, you're running multiple mirrored vdevs (A0, A1, B0, B1, C0, C1, D0, D1, E0, E1, and so on...). At one day, you encounter disk failure of A0. A few hours later, ZFS detects Biterror in A1 (still degraded status, disk a0 is still getting rebuilt)... My question: How should ZFS handle / repair this biterror, since there is no parity or no mirror data available?

At any point where ZFS encounters an error for which no redundancy is available, the system returns an error. What else would you have it do? Guess?

 

[Spearfoot]

When I noticed it correct, there's one point this article doesn't deal with: Parity Benefits of RAIDZ-2 vs. RAID10:
One example: Let's imagine, you're running multiple mirrored vdevs (A0, A1, B0, B1, C0, C1, D0, D1, E0, E1, and so on...). At one day, you encounter disk failure of A0. A few hours later, ZFS detects Biterror in A1 (still degraded status, disk a0 is still getting rebuilt)... My question: How should ZFS handle / repair this biterror, since there is no parity or no mirror data available?

ZFS performs checksum validation of all data, whether that data is stored on a RAID-Zn or mirrored vdev. With a RAID10 array, this may or may not be the case, depending on the implementation.

 

[Scampicfx]

Hey, thanks for your posts and your links. I went through them. Again, that's very helpful information! I'm sorry for asking all that questions right now, but I'm still at the point of learning, reading, learning, thinking, making a choise, thinking, reverting the choice, learning and start recursively from the beginning again ;) Right now I'm running an old Windows Server 2008 R2, using Hardware RAID 5, non-ECC-RAM, NTFS and that creepy thing had driven me crazy in the past (complete partition loss when trying to expand raid capacity -> running recovery tools, data corruption errors)... I just want to get away from that insane setup! :D I have backups, but I would like to have a running system that I can rely on for keeping the integrity of files, protecting them from bitrot, sending me mails about server health, basically all that stuff FreeNAS and a complete new server offers ;)

At any point where ZFS encounters an error for which no redundancy is available, the system returns an error. What else would you have it do? Guess?

To be honest, I don't know what happens EXACTLY in the moment of a system error... So, I started looking around a bit and came accross another thread: https://forums.freenas.org/index.php?threads/raidz-and-unrecoverable-bit-errors.27014/

Since one of my aims is to be protected against URE / bitrot even in the case of one hard drive failure, I agree that double redundancy should be the default choice, which means at least RAIDZ-2 or 3-way-mirror-vdev (as you mentioned the basic 2-way-mirror doesn't provide this double redundancy, so I agree, it was a bad idea to compare that). At first impression, RAIDZ-2 sounds good to me when IOPS aren't that important! I'm still not quite sure about starting with 4 disks, 6 or 8 disks... but I will continue reading and then hopefully be able to make a decision ;)

I changed the topic from "ZFS: You should use mirror vdevs, not RAIDZ". to "mirrored vdevs vs. RAID-Z". I hope this fits better!

 

[ttabbal]

You need to define your goals. You mention IOPS not being that important. How many simultaneously accessing users do you have? How much storage do you need? How many drive bays/controller channels? Network speed?

 

[jgreco]

The storage vdevs of ZFS are optimized towards two different things, and you really need to understand that they are totally different things. Mirrors are optimized towards high performance storage, trading space for speed. RAIDZ is optimized towards mass storage, trading speed for space.

For a case where you simply want to store massive quantities of data and store/retrieve it at a non-furious pace, RAIDZn is the obvious choice.

Both authors make the mistake of messing around with trite configurations, so I'm going to put out a requirement spec and show ACTUAL cost.

My requirement is a 32TB pool from which I can lose two disks without losing redundancy (meaning a pool should be rebuildable from the remaining disks even with a few disk read errors, as long as they don't pile up in the same area).

For RAIDZ, I choose RAIDZ3. To accomplish this, I require eleven 4TB hard drives, the ST4000DM000's run around $110, so that's $1210.

For mirrors, four way mirrors are required. So to accomplish this, I require thirty two 4TB hard drives, that runs around $3520. It is also almost three times as many drives.

So the thing is, the constantin article talks about RAIDZ (one drive parity) but then starts to do comparisons against three way mirrors and tries to draw conclusions. This is an apples to oranges situation. There is some *value* to the general idea, including that maintaining mirrors can be easier, but only the most trite pools will have an amount of disk that the "20%" talked about is applicable. In practice, it turns out to be more, especially since we do not recommend RAIDZ, but rather RAIDZ2 or RAIDZ3.

 

[Scampicfx]

Thanks for your replies!

The main point I am struggling about most is how many disks I should use in the initial server-hardware-setup in order to provide best options for future upgrades... The chassis offers 16 hot-swap-bays.
Basically, using either 4 disks vdevs or 8 disks vdevs are the only possible configurations in the event I want to use all hot-swap bays of the chassis,.

Starting with one 4-drives-RAIDZ2 allows upgrade option for another three 4-drives RAIDZ2 until the chassis is fully loaded (four vdevs in total). However, in this setup, 50% of raw diskspace is lost due to parity... The 6-drives-RAIDZ2 allows only two 6-drive-RAIDZ2 in total, however the percentage of used diskspace for parity gets lower. In total, 2x 8-drives-RAIDZ2 would allow an even higher amount of useable space (due to lower parity percentage) and it would also fill out every drive cage of the chassis. However, I don't require that much diskspace of 8 drives at the beginning... There aren't more than 3 parallel users accessing / streaming content and the content right now is about 15 TB... It grows slowly but steadily, so good and easy upgrade possibilities in future are important to me!

Basically, since I create backups and I don't require a 99,999% availability of the storage server, I don't want to use 50% on parity. I had a look on CrashPlan and the options they offer sound quite good to me. Therefore, I'm deciding right now between two options:
2x RAIDZ2 8-disk vdev
or
2x RAIDZ2 6-disk vdev + additional pool consisting of 1x stripped mirror vdev (4 disks) for applications that require higher IOPS

In both cases, initially I would start with one vdev. In a few years, I would expand the pool by adding the remaining vdev. Regardless of which option I choose, this means I only have one expansion option (apart from replacing disks)...

Does this sound stupid? Are there any other options I just don't see?

By the way: Supermicro offers this rear 2,5" expansion kit for SuperChassis... Let's say I buy that kit and connect the backplane of this rear kit with the SATA ports on the mainboard: do the drives of the rear bay get hot swappable in this case? Or do I still need to power down the machine in order to replace a failed drive of the rear-bay? The two additional drives increases the amount of possible hard drives of the chassis up to 18, which means that I basically could to 3x RAIDZ2 vdevs with 6 disks... However, I'm not quite sure if that's a good idea? ;) The other 16 disks are connected to an LSI 9300-8i HBA 12 Gbps, 2 SFF Links, via Backplane of chassis.

By the way: In the meantine I was looking for performance charts. Most of you know this already, but for some newcomers maybe this is pretty interesting: https://calomel.org/zfs_raid_speed_capacity.html It was impressive to see how useful it is to use compression!

 

[Stux]

This is one of the reasons I went with a 24 bay chassis :)

Meant I could use 4x 6 disk vdev or 3x 8. In the end I went with 8 wide, but our storage requirements are quite high.

FWIW, I think your idea to have 2x 6 + a dual mirror is good.

You don't *need* to have your vdevs balanced.

So you could go with 4 disks, then add 6, then add 6. And you could even start with 4, then add 4 and recreate the array into an 8 wide (creative partitioning and musical chairs)

 

[Robert Trevellyan]

Are there any other options I just don't see?

Consider that by the time you really need more than 2 x 6-disk RAIDZ2 vdevs, so much may have changed that any decision you make today could be moot.

 

[jgreco]

By the way: Supermicro offers this rear 2,5" expansion kit for SuperChassis... Let's say I buy that kit and connect the backplane of this rear kit with the SATA ports on the mainboard: do the drives of the rear bay get hot swappable in this case?

Basically everything provided by Supermicro in the server realm is capable of hot swap. There are two basic things you need. One is for the chassis to support this from an electrical perspective, including being sufficiently sized and engineered not to cause mayhem when a new drive is suddenly plugged in. The other is for the SATA (SAS etc) chipset to "notice" the drive replacement.

Or do I still need to power down the machine in order to replace a failed drive of the rear-bay? The two additional drives increases the amount of possible hard drives of the chassis up to 18, which means that I basically could to 3x RAIDZ2 vdevs with 6 disks... However, I'm not quite sure if that's a good idea? ;) The other 16 disks are connected to an LSI 9300-8i HBA 12 Gbps, 2 SFF Links, via Backplane of chassis.

If you're using a 2.5 chassis, such as the SC213, that'd make sense except that the rear expansion kit isn't available for it.

The rear expansion kit is intended for disk-full chassis such as the SC216

http://www.supermicro.com/products/chassis/2U/216/SC216BE1C-R920LP.cfm

or the SC846

https://www.supermicro.com/products/chassis/4U/846/SC846BE1C-R1K28B.cfm

In the latter case, the drives are simply intended to be used as boot drives because of the smaller form factor. In the SC216, however, if you pick a chassis with an expander, you could easily wire them in the back to the HBA as well, and end up with a 26 drive chassis all on the HBA.

 

[Scampicfx]

Cool, thanks for your replies and your input!