Configure vdev or pool spanning two vdevs – sharing special vdev possible?

[xness]

Can I create a vdev or pool spanning two vdevs? – And can they share the special vdev for deduplication or does every vdev in the pool require its separate special vdev in order to function properly?

Also – I just added a mirrored special vdev to an already existing pool and checked the encrypt option, but it didn't download any key file. So I assume it uses the same keys as for the original pool configuration?

Code:

root@lilith[~]# zpool status
  pool: Goliath
 state: ONLINE
  scan: scrub repaired 0B in 00:07:49 with 0 errors on Sun Jun  5 00:07:49 2022
config:

        NAME                                            STATE     READ WRITE CKSUM
        Goliath                                         ONLINE       0     0     0
          raidz2-0                                      ONLINE       0     0     0
            gptid/7a492ffe-d841-11ec-92b7-3cecef0f0024  ONLINE       0     0     0
            gptid/7a57523c-d841-11ec-92b7-3cecef0f0024  ONLINE       0     0     0
            gptid/7a4cccdb-d841-11ec-92b7-3cecef0f0024  ONLINE       0     0     0
            gptid/7a5554d2-d841-11ec-92b7-3cecef0f0024  ONLINE       0     0     0
            gptid/7a501918-d841-11ec-92b7-3cecef0f0024  ONLINE       0     0     0
            gptid/7a852e97-d841-11ec-92b7-3cecef0f0024  ONLINE       0     0     0
            gptid/7a4f10b4-d841-11ec-92b7-3cecef0f0024  ONLINE       0     0     0
            gptid/7a1ba28a-d841-11ec-92b7-3cecef0f0024  ONLINE       0     0     0
            gptid/7a52cf0d-d841-11ec-92b7-3cecef0f0024  ONLINE       0     0     0
            gptid/7a4b4df4-d841-11ec-92b7-3cecef0f0024  ONLINE       0     0     0
        dedup
          mirror-1                                      ONLINE       0     0     0
            gptid/0d64f440-f717-11ec-890c-3cecef0f0024  ONLINE       0     0     0
            gptid/0d50f5d2-f717-11ec-890c-3cecef0f0024  ONLINE       0     0     0

errors: No known data errors

  pool: David
 state: ONLINE
  scan: scrub repaired 0B in 00:07:49 with 0 errors on Sun Jun  5 00:07:49 2022
config:

        NAME                                            STATE     READ WRITE CKSUM
        David                                           ONLINE       0     0     0
          raidz2-0                                      ONLINE       0     0     0
            gptid/7a492fce-d841-11ec-92b7-3cecef0f0024  ONLINE       0     0     0
            gptid/7a5752cc-d841-11ec-92b7-3cecef0f0024  ONLINE       0     0     0
            gptid/7a4ccccb-d841-11ec-92b7-3cecef0f0024  ONLINE       0     0     0
            gptid/7a5554c2-d841-11ec-92b7-3cecef0f0024  ONLINE       0     0     0
            gptid/7a5019c8-d841-11ec-92b7-3cecef0f0024  ONLINE       0     0     0
            gptid/7a852ec7-d841-11ec-92b7-3cecef0f0024  ONLINE       0     0     0

errors: No known data errors

 

[NugentS]

A special vdev is pool level, not vdev level. So it does its thing for the pool.

 

[Samuel Tai]

Can I create a "unified synthetic pool" from two separate pools?

No, but it is possible to fake it with a unionfs mount. This isn't recommended on TrueNAS Core, as unionfs in upstream FreeBSD isn't really stable nor data safe. It may work better on SCALE, as that's based on Debian.

And can I somehow share my special vdev across multiple pools or does each pool require their own special vdevs?

Any VDEV can only be a member of one pool.

Otherwise – can I create a dataset/zvol spanning two pools?

No, sorry.

 

[xness]

Pardon my absolutely terrible wording. What I meant was:

Q: Can I create a "unified synthetic pool" from two separate pools?
A: Adding add new vdev to my existing pool is what I have to do.

Q: And can I somehow share my special vdev across multiple pools or does each pool require their own special vdevs?
A: Thanks – got vdev, pool etc mixed up.

Q: Can I create a dataset/zvol spanning two pools?

Q: How can I edit a vdev to add more disks to it? In the GUI option it seems like I can only add a new vdev to the pool entirely.

Q: How are vdevs in a pool utilized when they are different in size and disk amount? If one vdev is 100TB and the other is 50TB, will it write 2 bytes to the 100TB vdev and then 1 byte to the 50TB vdev? What does that mean if a vdev loses all parity drives – is all data of the pool lost?

 

[Samuel Tai]

Q: Can I create a dataset/zvol spanning two pools?

Not possible.

Q: How can I edit a vdev to add more disks to it? In the GUI option it seems like I can only add a new vdev to the pool entirely.

VDEV topology is fixed at creation. There has been talk of RAIDZ expansion, i.e., adding drives to a RAIDZx VDEV, within the OpenZFS developers, but this hasn't made it to an OpenZFS release yet. So at this time, this isn't possible.

Q: How are vdevs in a pool utilized when they are different in size and disk amount?

The pool limits usable capacity of the VDEVs to equal that of the smallest VDEV in the pool. Note, with a pool consisting of stripes of RAIDZx VDEVs, all VDEVs have to have the same numbers of disks, but not the same capacity. With stripes of mirrors, each stripe can have more drives, so long as each strip has at least one drive. The whole pool will limit each VDEV's capacity to equal the smallest VDEV in the pool.

What does that mean if a vdev loses all parity drives – is all data of the pool lost?

RAIDZx VDEVs don't have parity drives. They distribute parity over all the drives in the VDEV. RAIDZ1 only distributes 1 copy of parity; RAIDZ2 2 copies; RAIDZ3 3 copies. In practice, you can lose n drives in a RAIDZn VDEV before that VDEV and pool is lost.

 

[xness]

Thanks. Very insightful.
So basically a pool with two vdevs consisting of 10 drives RAIDz2 and 6 drives RAIDz2 is not possible?

And what happens if I lose an entire vdev in a pool consisting of two vdevs?

 

[Samuel Tai]

So basically a pool of 10 drives RAIDz2 and 6 drives RAIDz2 is not possible?

No, both VDEVs would have to be 6 drives. You can assign the leftover 4 disks as spares.

And what happens if I lose an entire vdev of a pool consisting of two vdevs?

Your pool would become UNAVAILABLE, with the error message INSUFFICIENT REPLICAS.

 

[xness]

Your pool would become UNAVAILABLE, with the error message INSUFFICIENT REPLICAS.

Unavailable – sounds like something recoverable, but shouldn't it be dataloss?

I thought two vdevs in a pool is like an expansion instead of a mirror. So losing a vdev would mean irreversible losing half of the data.

 

[Samuel Tai]

Unavailable – sounds like something recoverable, but shouldn't it be dataloss?

I'm reporting what you'd see in zpool status. In practice, this scenario usually ends in data loss.

I thought two vdevs in a pool is like an expansion instead of a mirror. So losing a vdev would mean irreversible losing half of the data.

Adding a VDEV creates a stripe over the new VDEV and any existing VDEV(s) in the pool. However, existing blocks in the original VDEV(s) aren't redistributed immediately to the new VDEV. Only updated blocks are preferentially written to the new VDEV until usage approaches the average of all VDEVs.

 

[danb35]

No, both VDEVs would have to be 6 drives.

Nonsense; vdevs of mixed disk count and configuration are a perfectly valid (if discouraged) configuration. The GUI will warn if you try to create them, but it will allow it.

 

[NugentS]

I was going to say that - but you beat me to it.
Mismatched vdevs are possible - but are also discouraged

So a Pool with a vdev of 10 drives RAIDz2 and a second vdev of 6 drives RAIDz2 is absolutely possible?

Encouraged - no
Discouraged - yes
Possible - Yes
Works - Yes

Note that loss of either vdev = loss of pool

Oh and the disks don't even have to be the same size - but within a vdev the vdev will be related to the size of the smallet disk in that vdev. Not sure what happens if the vdev is just striped - but don't, just don't

 

[xness]

Thanks for clearing that up everyone. Any insight to why it's discouraged?

So what would you do if you have a 16 bay enclosure of which 10 drives are already allocated to a pool – build a separate pool, go the discouraged route and add an odd-sized vdev to the already-existing pool or build a JBOD to add the 4 missing disks via HBA?

 

[HoneyBadger]

Thanks for clearing that up everyone. Any insight to why it's discouraged?

Imbalanced vdevs can cause inconsistent performance, as well as management confusion about how many failed drives can be tolerated.

Let's say there's two 12-bay JBODs - the first is a single vdev of 12 drives in Z2, the second is two Z2's of 6 each.

For the first JBOD, two red lights of "drive failed" is a "panic right now" situation, in that you're at risk of data loss. For the second, if those two red lights are in different vdevs, you have no active risk of data loss, but should probably still get on replacing those failed drives.

But at a glance, unless you know the difference between them, someone could assume that the 12-drive vdev is "still okay for the moment" rather than raising the appropriate level of support ticket.

So what would you do if you have a 16 bay enclosure of which 10 drives are already allocated to a pool – build a separate pool, go the discouraged route and add an odd-sized vdev to the already-existing pool or build a JBOD to add the 4 missing disks via HBA?

Personally I'd add/build a JBOD, and possibly build a separate pool to shuffle data around. 16-bay units go nicely with 2x 8-drive Z2 vdevs in each.

Side note, you may be doing that anyways - you've added a dedup vdev to a pool that has a top-level RAIDZ2 vdev. Unless I'm mistaken (or there's been a commit recently I don't know about) you won't be able to remove that dedup vdev in the future.

 

[xness]

Imbalanced vdevs can cause inconsistent performance, as well as management confusion about how many failed drives can be tolerated.

Let's say there's two 12-bay JBODs - the first is a single vdev of 12 drives in Z2, the second is two Z2's of 6 each.

For the first JBOD, two red lights of "drive failed" is a "panic right now" situation, in that you're at risk of data loss. For the second, if those two red lights are in different vdevs, you have no active risk of data loss, but should probably still get on replacing those failed drives.

But at a glance, unless you know the difference between them, someone could assume that the 12-drive vdev is "still okay for the moment" rather than raising the appropriate level of support ticket.

Not sure I understand this example correctly why two unbalanced vdevs are discouraged. After all the amount of disks in each vdev don't influence your parity level. You have have a Z2 on a 6-drive vdev or a 12-drive vdev.

Personally I'd add/build a JBOD, and possibly build a separate pool to shuffle data around. 16-bay units go nicely with 2x 8-drive Z2 vdevs in each.

Well I already committed the 10 drives to a pool – so going 2x 8-drive vdev isn't really a convenient option for prod anymore. Or can I scale the vdevs down afterwards?

 

[NugentS]

err nope you can't

 

[HoneyBadger]

Not sure I understand this example correctly why two unbalanced vdevs are discouraged. After all the amount of disks in each vdev don't influence your parity level. You have have a Z2 on a 6-drive vdev or a 12-drive vdev.

Correct, the redundancy concern is mostly from a management/replacement level. The unbalanced ones being discouraged is from a performance and allocation standpoint.

Using the same 12 vs 6 drive count, let's say that the 12-drive one was built first, using 4T drives, which yields 10x4=40T usable space - and you're able to write (sequential) data to ten of them at a time, giving you say 50MB x 10 spindles = 500MB/s sequential write speeds. You add the 6-drive with 10TB drives figuring on getting the same effective 4x10=40T - but you're limited to 50MB x 4 spindles = 200MB/s sequential write speeds. And because vdevs in a pool are treated in a "spanning/striping" fashion, you'll get one vdev able to do 500MB/s writes and one doing 200MB/s - and ZFS will be trying to allocate more to the slower vdev because it's empty.

Well I already committed the 10 drives to a pool – so going 2x 8-drive vdev isn't really a convenient option for prod anymore. Or can I scale the vdevs down afterwards?

My suggestion would have been to use the JBOD as a "swing space" with a zfs send | recv operation to move your data off the existing pool temporarily, while rebuilding the vdevs in a different way.

RAIDZ vdevs can't be resized or removed from a pool at this point unfortunately.