Anyone knows what this means?

[Whattteva]

I was reading the TrueNAS Hardware Guide, particularly the Virtualized TrueNAS CORE section and this bullet point caught my interest.

• Use a least three vdevs to provide adequate metadata redundancy, even with a striped pool.

It doesn't really go into much detail about what that exactly means. And I couldn't really find something similar on the forums search. I'm used to just seeing RAIDZ2 or striped mirrors. I suppose for striped mirrors, that means 3 mirror vdevs, but what does that mean for a RAIDZ2 pool? 3x RAIDZ2 vdevs? I kinda' think that most people only run single vdev RAIDZ2's. Also, why is this relevant? I thought loss of a vdev means loss of the pool anyway? As I understand it, there's no redundancy at the pool level, only at the vdev level. Am I missing something?

 

[HoneyBadger]

ZFS actually protects its metadata by mirroring copies of it across multiple vdevs, regardless of the pool configuration. This is more so to protect against unexpected corruption in the case where a single device in a mirror vdev (or two devices in a RAIDZ2, to extend the example) fail - the pool is still online, but if the surviving device(s) also have a bad sector or two, the redundant metadata prevents the pool from being corrupt.

It's noted in the "Virtualized TrueNAS" piece more as a way to mitigate the risk of using virtual disk devices with TrueNAS (which isn't recommended in general) and saying "rather than using a single large virtual disk, use multiple small ones and stripe them" - in the latter scenario, all of your data eggs are still in one basket, but your metadata at least has redundancy, and bad metadata will render a pool unmountable in a hurry.

 

[Whattteva]

I see, that makes sense. So in the case that you have multiple catastrophic vdev failures. Your data may be corrupted/unavailable, but at the very least your pool is still mountable and you could MAYBE salvage some files here and there? That's definitely quite an important detail that I haven't really seen any guide mention.

 

[HoneyBadger]

It doesn't even need to be a catastrophic vdev failure (you usually aren't recovering from that) but moreso "drive generates an uncorrectable read error during a window where your redundancy has been compromised"

 

[Whattteva]

Got it. Thanks for clarifying! I think the Guide probably also should be updated with this information.