4 drive - mirror vdev or raidz2

[chuck32]

Hello all,

I recently build my first homeserver and want to setup truenas.

Specs:
asrock b560m-itx/ac
i3 10100
32 GB RAM
Delock 5 port SATA PCI Express x4 Card
4x WD Red Plus 4 TB
+ SSDs for proxmox and backup of VMs
Truenas as VM

I'm currently following this guide https://www.truenas.com/community/resources/hard-drive-burn-in-testing.92/ for initial burn in. First drive exited with not output (last time I checked 0/0/0 erros). I'm running a smart long again and wait for the other drives to finish.

Somewhere I read, that not all drives are the same in size and can be helpful to leave some 200 mb free on the and hence partition them, in case you need to replace a drive and the new drive is slightly smaller for whatever reason. Can't find the link to it again though.

So probably tomorrow all drives should have finished badblocks and the smart long afterwards and I can begin to setup my system.

I tried determining what setup is best for but I'm unsure what's right for me. My requirements or non-requirements are:

• NAS should act as local working directory for me and my wife.
• we rarely access the pool at the same time due to different working schedules
• connection is 1 Gbit anyway, so max. 100 mb/s
• availability is not a concern, if a drive failes we can afford to shutdown the server and wait for the replacement to arrive and be burned it
• we would mirror the data to each of our office PCs, specifically I have 2x4 tb in my PC (so I would mirror the whole NAS to my PC) and my wife would mirror her data to her internal 1 tb HDD additionally
• my PC is mirrored to backblaze personal backup
• performance is not a concern
• if I buy a nvme SSD I could use max. 6 HHDs in the future, however:
• 8 Tb of usable space is probably more than enough for the next 5 years
• I purchased 2 drives each from different vendors to get different batches

I'm torn between raidz2 and mirrored vdev.

Here a mirrored vdev is preferred https://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-not-raidz/ this thread (https://www.truenas.com/community/threads/which-raidz2-or-mirror.23401/) prefers raidz2.

Raidz2 has the advantage that any 2 drives can fail, with mirrored vdev I only have a 66.6 % chance to get lucky. However expansion should be easier with a mirrored vdev, which I don't think will be planned during the lifespan of the current HHDs anyway.

Any thing else I need to consider for my setup?
Any recommendations for mirror vs raidz2?

Thanks in advance!

 

[jgreco]

The jrs-s thing is a bit weird.

Some differences between RAIDZ and mirrors, and why we use mirrors for block storage

ZFS is a complicated, powerful system. Unfortunately, it isn't actually magic, and there's a lot of opportunity for disappointment if you don't understand what's going on. RAIDZ (including Z2, Z3) is good for storing large sequential files. ZFS will allocate long, contiguous stretches of disk...

www.truenas.com

Generally it's good to remember that mirrors are better for performance and concurrency, RAIDZ is better for resiliency and error correction.

 

[chuck32]

Thank for your reply!

Generally it's good to remember that mirrors are better for performance and concurrency, RAIDZ is better for resiliency and error correction.

Is raidz inherently better for error correction? I thought the ZFS filesystem itself (protection against bit rot) is error correcting. I do not use EEC RAM but I tested it with memtest86 beforehand. I know that this is not a 100 % solution but probably better than my old setup with internal HHDs (I recently discovered some corrupted files, although they are all from the same google takeout download so maybe that failed years ago).

So raidz2 would be my choice because I value the fault tolerance over the peformance? I would think the chance of 3 drives failing during the longer resilvering is not higher than that the wrong, second drive or third drive fails in a mirror setup.

 

[jgreco]

Is raidz inherently better for error correction? I thought the ZFS filesystem itself (protection against bit rot) is error correcting.

ZFS is error *detecting* through the use of checksums. It can detect checksum errors on a single disk, but in order to correct an error, you need either another copy of the data (flip side of the mirror, for mirrors) or a way to compute the value that was there (RAIDZ). If ZFS can figure out what the data was SUPPOSED to have been, then, yes, it will also write out ("correct" or "heal") the bad data on disk. It cannot do that without some source for the redundancy, so error correction is dependent on additional disks.

Some of us work on the principle that "a single failure shall not compromise redundancy". For example, if a disk fails in RAIDZ1, then you no longer have redundancy. However, if a single disk fails in RAIDZ2, you still have redundancy. Likewise, the failure of one side of a two-way mirror is a loss of redundancy, while the failure of one disk of a three-way mirror is not a loss of redundancy.

 

[chuck32]

Thank you! I'll setup raidz2 then, probably without partitions, just using the whole drive.

Also on posting I realized it would probably be the same thing to switch from backblaze personal backup to the cloud integration.