RAIDz fail save with stripes and mirrors

[cgasser]

Hi,

I have a server with 8 disks. 8TB each. I want to configure it with 2 disk groups with 4 disks each. Each disk group as stripe set.
If I am correct I can lose up to 4 disks within one group. But if I lose one disk of group 1 and one disk of group 2 (the same time) I am doomed.
Right?

Would apreciate if someone can confirm that.

BR
Christoph

 

[chuck32]

If I understand you correctly you want 2 stripes that are mirrored?
In that case no, the first failed drive of any group (vdev) causes the vdev to fail. You lose your mirror instantly, this configuration can only survive one drive loss.

What you probably want is 4x2 mirrors striped.
In that case, you can lose 1 disk per vdev.

First disk is 100 % survival rate.
Second disk you have a 85 % rate (any of the 6 remaining disks that are not part of the vdev that already suffered a loss may fail)
and so forth until each mirror lost a member disk.

This also leaves the not unreal risk that the second disk you lose kills your pool. It's a numbers game.
RAIDZ2 would be more resilient.

Personally I switched to striped mirrors for some of its advantages but may revert to RAIDZ2 in the future.
For mirrors:
+ Faster resilver times
+ More IOPS, not sure if I really need them
+ More flexibility, although with a caveat: I don't want (or can space wise) add another Vdev so I can only upgrade a single vdev for more storage efficiency, however, at some point, all writes will go to that single vdev because it's larger, and I lose the speed advantage of the setup. This also assumes I rebalance my files after expansion.

This is just to add my personal note on the matter. Ultimately you need to decide.

 

[ChrisRJ]

What is your use-cases? That is ultimately the most important factor to determine what makes sense.

Also, I strongly recommend to read up on the ZFS basics. Please have a look at the recommended readings in my signature.

 

[Davvo]

I also reccomend reading the following resources since it appears to not be in Chris' reccomended readings.

ZFS Storage Pool Layout

This resource was originally created by user: @Davvo on the TrueNAS Community Forums Archive. https://www.truenas.com/community/resources/zfs-storage-pool-layout.201/download [1] This amazing document, created by iXsystems in February 2022 as a “White Paper”, cleanly explains how to qualify...

www.truenas.com

 

[cgasser]

Hi! At first: Thank you for the detailed replies.
Second: I read a lot and watched a bunch of Youtube videos. Nevertheless, I want to make sure I am not messing up my first TrueNAS installation and maybe participate a little bit from the experience others have.

What I want to use the NAS for is: storage for some VMs (XCP-NG). But just the non performant relevant data. I would not want to have my MSSQL databases on the NAS for example but will keep them on the virtual machines directly (same hypervisor). Besides that data safety is the most important part for me. I need to be safe even if 2 disks are failing the same time. And yes: I know, RAID is no backup. Even have a shirt with that ...

And with the data safety as prio 1 I was interested to know, which is the best option performance wise with performance having prio 2.

Appologies for not being clear enough in my initial post.

BR
C

 

[ChrisRJ]

I also reccomend reading the following resources since it appears to not be in Chris' reccomended readings.

ZFS Storage Pool Layout

This resource was originally created by user: @Davvo on the TrueNAS Community Forums Archive. https://www.truenas.com/community/resources/zfs-storage-pool-layout.201/download [1] This amazing document, created by iXsystems in February 2022 as a “White Paper”, cleanly explains how to qualify...

www.truenas.com

Thanks, I wasn't aware of this. Added to sig

 

[ChrisRJ]

I read a lot and watched a bunch of Youtube videos.

The vast majority of YouTube videos either leaves out critical details or gives downright wrong recommendations. Lawrence Systems is the notable exception here and literally the only one I would personally trust.

 

[Davvo]

What I want to use the NAS for is: storage for some VMs (XCP-NG).

Then you want mirrors as per the following resource.

Resource - The path to success for block storage

ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle bits modified after creation. This is optimal for RAIDZ. It's what most...

www.truenas.com

Besides that data safety is the most important part for me. I need to be safe even if 2 disks are failing the same time.

You could go 2x three-way mirrors, granting you the resiliency to withstand up to two simultaneus drive failures in each VDEV, and use the remaining two drives as hot or cold spare... or a two-way mirror for a different pool.
The extreme solution would be 2x four-way mirrors, but that might be a bit too much.

The non-mirror solution would be a single 8 disks VDEV in RAIDZ2 or RAIDZ3, but that wouldn't be optimal for the reasons stated in the resource.

 

[ChrisRJ]

Since you need VM backups anyway, what about a 2-way-mirror (ideally with SSDs) for the VMs and a regular backup to RAIDZ2? That is what I run and so far it is the sweet spot for me.