Enhancing a pool of mirrors - can I add extra disks in these ways?

[Stilez]

I've tried to research this but can't find an answer, with luck someone here will know.

My new server will have its pool built as a set of mirrored HDDs. I would want to stick with mirrored HDD vdevs, or at most mirrored 2-way stripes, rather than RAIDZ, due to read+resilver speed/resilver stress/splittability, but especially because of the cost of larger HDDs and because mirrored vdevs can be expanded after being added to the pool in ways RAIDZ can't. Large good quality HDDs are expensive and where I want to end up will take some time to finish buying, so I'd like to get my new FreeNAS server running with basic size+redundancy, then gradually enhance the pool to where I really want it.

So here's the thing. I'd ultimately like to be using 100% 3 way mirrors (or 3 way mirrors of 2 striped HDDs), i.e., vdevs like these:

• ( (d1+d2+d3 as a 3-way mirror) + (d4+d5+d6 as a 3-way mirror) striped)
• ( (d1+d2 striped) + (d3+d4 striped) + (d5+d6 striped) as a 3-way mirror) if I want to reduce the performance hit for writes.

Right now I can get going with 2 way mirrors (although with less spare capacity than I'd like until I feel safe breaking up the old server and using the HDDs in it). I want to be sure I can actually get there from here when I eventually buy my extra HDDs. So these are my questions:

1. If the pool is made up of pure mirrors (d1+d2 mirror) + (d3+d4 mirror) etc, can I buy a single new HDD to attach to just one vdev, ending up with (d1+d2+d5 mirror) + (d3+d4 mirror)? Will ZFS and FreeNAS tolerate adding a disk to a pool to make some vdevs 2-way mirrors and others 3-way mirrors, or must all mirror vdevs have the same number of HDDs as mirrors?
2. If a vdev is a mirrored pair of two striped HDDs ( (d1+d2 stripe) + (d3+d4 stripe) as mirror), can I add 2 HDDs to make it a 3-way mirror of pairs, ending up with a 3-way mirror ( (d1+d2 stripe) + (d3+d4 stripe) + (d5+d6 stripe) as mirror)? Does it matter if other vdevs in the pool are of different kinds, such as simple non-striped mirrors?
3. How do I do #2 in the FreeNAS GUI? If I need to use the console ( zpool add/attach) to do it, what would the command be? Would it matter that it's done outside the FreeNAS GUI, and how do I handle issues arising from modifying the pool outside FreeNAS GUI using zpool commands, and not updating the FreeNAS config file/s?
4. If one of the vdevs in a pool is a simple 2-way mirror vdev (d1+d2 mirror), is there a one-step way to directly extend it into a 2-way mirror of stripes ( (d1+d3 striped) + (d2+d4 striped) as mirror)? I'm aware that I can do this by splitting the mirror and extending the single drive to a stripe, then adding a pair of disks to mirror the newly-formed stripe, and I can probably do it by offlining the 2 disks from the pool provided there's enough capacity for ZFS to move the data from those disks to the rest of the pool before they're killed, but is there a simpler way? Would I ever have a good reason for wanting to do this or would I almost always just want to add the 2 new HDDs as an extra mirror vdev?

Hoping someone knows these answers (rather than just guessing them!), so I can get going building the pool!

 

[anodos]

VDEVs are striped.

You cannot mirror VDEVs.

You can have a VDEV that is a 2-way mirror, 3-way mirror, RAIDZ, etc.

 

[Stilez]

So you can't have a vdev that is itself striped, or a mirror of striped disks? Just single disk, mirror or raidz? Not a problem, I must have mis-remembered that one. That mistake simplifies it a lot, actually, it takes care of #2, '3 and #4 at one strike.

What about #1? I would think if one can mix mirrors and raidz (however ill advised), then it's likely one can mix 3-way and 2-way mirrors? Can I?

 

[Arwen]

So you can't have a vdev that is itself striped, or a mirror of striped disks? Just single disk, mirror or raidz? Not a problem, I must have mis-remembered that one. That mistake simplifies it a lot, actually, it takes care of #2, '3 and #4 at one strike.

What about #1? I would think if one can mix mirrors and raidz (however ill advised), then it's likely one can mix 3-way and 2-way mirrors? Can I?

No, you can't have a Mirrored vDev that is it's self striped.

Yes, you can mix 2 way Mirror vDevs with 3 way Mirror vDevs. Data will be striped across the vDevs.
It will work just fine. A bit slower on writes to the 3 way Mirrored vDev, but that's to be expected.

Please note that if you start with 4 disks of similar size, it can be advantagous to use RAID-Z2 instead
of 2 x 2 disk Mirrored vDevs. Same amount of storage available. Basically it comes down to the
recoverability of failed disks. In the case of a 2 way mirror, during disk replacement you are relying on
a single disk for the source of the data to be recovered. A second disk loss with that specific disk, means
it's full Zpool restore time. But, to be fair, a few bad blocks on the source disk would just corrupt a few
files that would require manual restoring.

But, in the case of RAID-Z2, you can loose any 2 disks without data loss. Here is a somewhat humorous
comment by one of the forum posters;

"If you start with 4 drives in RAID-Z2, your pool will be able to survive the
failure of any 2 drives without data loss, and the ZFS police will give you
a pony."

Robert Trevellyan, Aug 23, 2015