Metadata vdev Invalid choice RAIDZ2

[Ryan Haver]

I'm attempting to create a new pool with the following disk layout and vdevs, but Truenas Core 12 does not like something about the Metadata vdev. Any help is appreciated.

Data vdev - 8 wide - Raidz2
log vdev - 1 wide
cache vdev - 2 wide - stripe
Metadata vdev - 4 wide - Raidz2

It won't allow me to specify raidz2 when attempting to create the metadata vdev. The only options are mirror or stripe, but when not selecting either of those options it appears to be using raidz2, so I leave it like that and then when attempting to create the metadata vdev I get the following error.

[special] Item#0 is not valid per list types: [type] Invalid choice: RAIDZ2
Error: Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/middlewared/job.py", line 361, in run
await self.future
File "/usr/local/lib/python3.8/site-packages/middlewared/job.py", line 397, in __run_body
rv = await self.method(*([self] + args))
File "/usr/local/lib/python3.8/site-packages/middlewared/schema.py", line 972, in nf
args, kwargs = clean_and_validate_args(args, kwargs)
File "/usr/local/lib/python3.8/site-packages/middlewared/schema.py", line 930, in clean_and_validate_args
value = attr.clean(args[args_index + i])
File "/usr/local/lib/python3.8/site-packages/middlewared/schema.py", line 607, in clean
data[key] = attr.clean(value)
File "/usr/local/lib/python3.8/site-packages/middlewared/schema.py", line 607, in clean
data[key] = attr.clean(value)
File "/usr/local/lib/python3.8/site-packages/middlewared/schema.py", line 470, in clean
raise Error(self.name, 'Item#{0} is not valid per list types: {1}'.format(index, found))
middlewared.schema.Error: [special] Item#0 is not valid per list types: [type] Invalid choice: RAIDZ2

I've tested creating all the other vdevs without the metadata vdev, but it won't let me add the metadata vdev with raidz2 after the fact either.

When selecting a mirror or stripe for the metadata vdev the following message is seen and a force option is presented:

Metadata vdev must be the same type as the data vdevs. First data vdev is a raidz2, new Metadata vdev is a stripe.

 

[LarsR]

I'm guessing it's because your first vdev has 8 drives and your second one 4. If you want to add a second vdev to your existing pool it must also have 8 drives.

 

[c77dk]

Meta just have to use the same level of redundancy as the data vdevs, not be the same type. Search the forums and you'll see

Mirrors are the way to go for meta, even though the data vdevs are setup as raidz2. So as meta for a raidz2 setup, you'll need at least a 3-way mirror.
The warning has been debated a lot here.

 

[jenksdrummer]

Warning shows when data vdev and metadata vdev aren't of the same topology. IE, number of disks, functional-raid type, etc. This doesn't extend to nested functionals; such as stripes of mirrors.

One can force it, but, though it's documentation implies it's technically possible to remove a metadata vdev if it follows topology without dataloss (not sure I believe it until I do it and confirm; that's just me), if the topology isn't aligned it is absolutely not possible.

 

[Ryan Haver]

It appears you can only setup the metadata vdev to match the raid type of a data vdev if it also matches the number of disks in the data vdev.

Hmmm...so essentially best practices is to have a metadata vdev match topology of the data vdev, but it will work if setup with a mirror, but can’t be removed from a pool without data loss if it doesn’t match the data vdev topology 100%.

It’d be nice to have some clearer cut documentation on this. I feel a little like TrueNAS Core was rushed as OpenZFS 2.0 was only officially released yesterday and these features are part of OpenZFS 2.0.

 

[jenksdrummer]

I don't disagree; there's a lot of quirks with 12-RELEASE that has me using 11.3U5 until U1 comes out and even then, testing...

I feel that the last 12RC was a better build.

To clarify earlier...

meta = 2 disks mirrored
data = 2 disks mirrored / 6 vdev

^ that topology is perfectly fine.

Meta doesn't seem to support Z1/Z2, so doing the same on the data vdev doesn't make sense. I think I messed with 2x2 (IE< RAID 10) at the meta, but I think it wouldn't let me on create, but I could add it afterwards.

Meh...

Mostly the meta vdev is for metadata (and dedupe tables if not specified otherwise); which DOES improve performance; and optionally small-block IO, but one has to configure that and right-size it because once that VDEV fills, it goes back to writing on slower data VDEV.

What I found most improved is being able to set dedupe to a dedicated VDEV...that gives a MUCH improved performance vs before - TBH, I haven't noted a degradation between dedupe and no dedupe; which is impressive considering where FreeNAS has been over the years...of note, if you go this route, put some high IOPS devices in the dedupe spot; and keep in mind you lose these; you lose the pool; same as meta VDEV...

 

[Ryan Haver]

I’ve already encountered a number of bugs that have been extremely annoying. It makes me hesitant to deploy TrueNAS Core right now, but I need some of the features.

I think the hardest part is right-sizing these special vdevs. I honestly have no idea what size is needed for metadata and/or dedupe tables. I also haven’t messed with dedupe on FreeNAS/TrueNAS due to all the horror stories I’ve read, but maybe it’s time to try it out.