Spreading a pool across multiple enclosures/VDEVs without creating a SPOF?

[swedish_six_shooter]

Hey folks,

I have a multi-enclosure TrueNAS setup at the moment, with roughly one VDEV/enclosure and one pool/VDEV. This is a pain to maintain b/c it requires periodically re-balancing loads over enclosures, and it seems like one of the key points of ZFS is that you can keep growing pools by adding VDEVs. However, I'm rather afraid of the consequences of having an enclosure suffer some sort of catastrophic failure (water damage, act of god, etc.) that will kill a VDEV and thus my entire pool. Per this old question, it seems like multi-JBOD pools are common in large enterprise situations. How do I mitigate my concerns about catastrophes? Keeping my pools balanced is a pain and I'd love to just let ZFS do it.

 

[Heracles]

A single server will always be a single point of failure. That can only be mitigated by using multiple servers, each exposed to different threats. See my signature about that...

 

[swedish_six_shooter]

A single server will always be a single point of failure. That can only be mitigated by using multiple servers, each exposed to different threats. See my signature about that...

This is true, but in the case what I'm referring to as a "SPOF" is the idea that a single enclosure going down catastrophically in a multi-vdev pool will destroy the entire pool and all of its constituent datasets. At present, if that same enclosure goes down catastrophically I'd only lose half as much data.

What I'd really like to hear is something like: “Don’t worry! When you have a catastrophic enclosure failure your drives will be fine so you should be able to get everything back up w/o data loss as soon as you repair or replace the enclosure!” However, I’m not sure if that’s a reasonable thing to expect.

 

[Heracles]

However, I’m not sure if that’s a reasonable thing to expect.

Indeed, it is not... Once a vDev is lost, you must consider that the pool is lost. vDevs are grouped in a pool, always without any redundancy. There are some professional that may try to recover data in such cases but it will cost you million time more than a backup, without the quality of an actual backup and with a much longer delay than a backup. As such, because your data is still available and you can put in place such a backup, do it.

No important data should go without backups anyway. If your data or of value to you, you MUST have backups.

I'm rather afraid of the consequences of having an enclosure suffer some sort of catastrophic failure (water damage, act of god, etc.)

Water damage, act of god and others are threats that will destroy the entire server, not only a single enclosure. For such a case, you need copy No 2 in the 3 copies rule : an offsite copy, one that will be out of reach of such an incident.

 

[danb35]

What I'd really like to hear is something like: “Don’t worry! When you have a catastrophic enclosure failure your drives will be fine so you should be able to get everything back up w/o data loss as soon as you repair or replace the enclosure!” However, I’m not sure if that’s a reasonable thing to expect.

Well, what kind of catastrophic enclosure failures are you concerned about? Failures of the enclosure itself (backplanes, power supplies, etc.)? Then yes, the drives will likely be fine. Environmental hazards like fire and flood? Nope.

 

[swedish_six_shooter]

Definitely the latter - but I think the broader point might be that in that scenario all of the enclosures will be dead, so I'll likely be relying on backups for my repairs any way.

 

[rvassar]

When I was a MTS 1 noob, I got to work in a large R&D Cluster lab. This cluster product was a node failover solution that could claim to run (redacted) DB as a certified five 9's uptime solution, meaning literally no more than 4 minutes of downtime per year. That wasn't good enough for a certain customer. They added a requirement that a certain terrorist organization could park a truck bomb outside their datacenter and completely destroy it, and one node and half the storage could be as much as 3km away in another datacenter permitting failover and completing any transactions in flight. So we had four huge 3 km spools of 100 mbit fiber sitting on the floor in the lab, and proved it could work, at least theoretically... They paid a princely sum for it.

And they still kept multiple backups...

 

[sretalla]

One simple way... 3 enclosures (let's say 24 disks per enclosure), 24 VDEVs which are 3-way mirrors, one disk from each VDEV in each enclosure.

Otherwise, the way to address it is to assume that if an enclosure somehow falls out of the rack killing itself and somehow all of the disks inside it at the same time (in some kind of "crash and burn" event), you're going to need to restore from backup when your new enclosure and disks arrive.

 

[blanchet]

For my side, I still run one pool per enclosure but I pick large enclosure (44 disks) with multiple VDEV inside.

Maybe I would dare using a pool that spreads across several enclosures if I had a TrueNAS Enterprise server with a support contract from iXsystems and if they told me that I can really use such a setup.

But for a self-maintained solution I prefer keeping things as simple as possible, because it is already difficult enough to manage JBOD