4 discs: raidz2 or 2 vdev mirrors?

[WillM]

I am about complete running badblocks tests on the four 4TB WD Reds going into my new FreeNAS server and have been contemplating the merits of different approaches to storage design. If I make two vdevs out of mirrored pairs of drives, from what I read it sounds like this will tend to perform better than all 4 in raidz2, is that true?

Then I also have three questions about going with mirrored pairs approach. First, will ZFS treat those two vdevs as stripes and not jbod/concat? Second, if it does treat them as striped, then later when I add another vdev consisting of a new mirrored pair of 4TB drives, will ZFS be able to theoretically increase the data throughput rate for the whole pool by 50%? Third, is this pool made of mirror pairs approach generally safe and recommended? I do recognize there's a little more risk in that only some combinations of 2 drives can fail without losing the whole pool.

Thanks in advance. I definitely still have a lot to learn about zfs.

 

[Robert Trevellyan]

If I make two vdevs out of mirrored pairs of drives, from what I read it sounds like this will tend to perform better than all 4 in raidz2, is that true?

It depends. @jgreco wrote about this in some detail in another thread recently. My limited understanding is that mirrors usually offer better IOPS but RAIDZx can offer better sustained throughput.

will ZFS treat those two vdevs as stripes and not jbod/concat?

I'm not sure I understand your question. When you make a pool out of multiple vdevs, in ZFS terminology, those vdevs are striped together into one pool. But if you make two pools, each one of a single mirror, there is no striping involved.

if it does treat them as striped, then later when I add another vdev consisting of a new mirrored pair of 4TB drives, will ZFS be able to theoretically increase the data throughput rate for the whole pool by 50%?

For workloads for which striped mirrors are a win, more mirrors are better. I doubt you'd ever see such simple math as a 50% improvement for adding a 3rd vdev.

is this pool made of mirror pairs approach generally safe and recommended? I do recognize there's a little more risk in that only some combinations of 2 drives can fail without losing the whole pool.

Only you can determine your comfort level with the risk/reward tradeoff. Regardless of pool layout, you must backup any data you care about.

 

[Arwen]

I had the same decision to make with 4 x 4TB disks. Either 2 x VDEVs of 4TB mirrors, or a
RAID-Z2. Since mine was a backup server and misc. dumping ground, I went with RAID-Z2
so I could loose any 2 disks.

 

[jingle]

What are the risks of using a mirror?

While resilvering a mirror after a disk failure there is NO redundancy:

• What happens if zfs stumbles upon a flipped bit (checksum error) upon resilvering the new mirror half?
• Will just the respective file be erroneous or will the entire pool be marked as faulted?

What are the chances of a mirror resilver go wrong? Any experiences?

 

[Bidule0hm]

What happens if zfs stumbles upon a flipped bit (checksum error) upon resilvering the new mirror half?

It'll see the error and report it, but can't correct it obviously.

Will just the respective file be erroneous or will the entire pool be marked as faulted?

It depends where the error is: if it's in a file then the file is corrupted; if it's in the pool metadata then you can loose the pool.

 

[jdong]

It'll see the error and report it, but can't correct it obviously.



It depends where the error is: if it's in a file then the file is corrupted; if it's in the pool metadata then you can loose the pool.

It's worth noting in the latter case that there is some amount of redundancy in that multiple copies of metadata are written out, so you need slightly worse luck than just a single metadata block error to lose the pool.

 

[jingle]

But it will continue until resilvering is complete and then report the error?

Does anybody know how long resilvering takes on a 4tb mirror (80% filled) vs resilvering a single disk in a 4x4tb raidz2 (80% filled).

I just want to figure out, how safe the bet is to survive a mirror resilver compared to raidz2 disk resilver - due to the fact that mirror resilver has no data redundancy.

 

[jdong]

But it will continue until resilvering is complete and then report the error?

Does anybody know how long resilvering takes on a 4tb mirror (80% filled) vs resilvering a single disk in a 4x4tb raidz2 (80% filled).

I just want to figure out, how safe the bet is to survive a mirror resilver compared to raidz2 disk resilver - due to the fact that mirror resilver has no data redundancy.

I think it's also worth remembering that a mirror resilver only requires one other disk. And if you suffered an incomplete and not total failure, the working parts of the failing disk still can contribute to the resilver.

But a Raid-Z2 resilver requires reading every remaining member and is a more strenuous and time consuming process.

Do you plan on growing your array? Mirrored vdevs give you a slight advantage in flexibility of growth too since you can just replace two disks one at a time.

 

[jingle]

Yes. Flexibility is the main driver for going with mirrors.
+ smaller energy consumption in the beginning
+ ability to easily add mirror vdevs
+ ability to quickly grow a mirror vdev after replacing just 2 drives
- lower space efficiency
Did I miss sth?

But:

• Won't the striping performance gain be neglectable when filling up a mirror and then adding another pair (due to the strong write bias to the almost empty/new mirror)?
• Data corruption upon resilvering a mirror (due to a failed drive) is my biggest fear due to a resilver without data redundancy (yes one needs and has backup). So I'm undecided between raidz2 over 2 mirror stripes (4 disks total)

 

[gpsguy]

You can mitigate the risk by going with 3-way mirrors.

• Data corruption upon resilvering a mirror (due to a failed drive) is my biggest fear due to a resilver without data redundancy (yes one needs and has backup).

 

[jingle]

Yeah, sure.

But any real-world experiences on the likelyhood of a mirror-(complete) resilver data corruption? Will the resilver continue until completeness and list the errors (if data error)?

and again: Any real-world resilver times for 4TB disk resilver (from zero) when 80% filled? (4tb per disk mirror vs 4tb per disk raidz2)?

Thx