Can ZFS support nested RAID level 555?

[Nikolas Britton]

Hypnotically lets say I have 1024 SAS drives across 64 16-bay SAS expander sleds (4 sleds per chassis) attached to an LSI 9206-16e. What is the optimal ZFS RAID configuration?

The requirements are:

• Able to saturate a dual QDR InfiniBand link (8GB/s).
• Able to survive a single drive failure.
• Able to survive a single sled failure.
• Able to survive a single chassis failure (4x sleds).
• At least 70% usable capacity.

Based on those survivability requirements I'm inclined to think that nested RAID level 555 might work. Each 16-bay sled would be a raidz vdev, then the 4x sled vdev in each chassis can be a raidz vdev, and then lastly aggregate all the chassis vdev into a single raidz vdev. Can ZFS support a triple nested RAIDZ pool?

• aggregate raidz pool
  • chassis 1 raidz vdev
    • sled 1 raidz vdev
      • drive 1
      • drive 2
      • drive 3
      • drive ...
    • sled 2 raidz vdev
      • drive 1
      • drive 2
      • drive 3
      • drive ...
    • sled 3 raidz vdev
      • drive 1
      • drive 2
      • drive 3
      • drive ...
    • sled 4 raidz vdev
      • drive 1
      • drive 2
      • drive 3
      • drive ...
  • chassis 2 raidz vdev
    • sled 1 raidz vdev
      • drive 1
      • drive 2
      • drive 3
      • drive ...
    • sled 2 raidz vdev
      • drive 1
      • drive 2
      • drive 3
      • drive ...
    • sled 3 raidz vdev
      • drive 1
      • drive 2
      • drive 3
      • drive ...
    • sled 4 raidz vdev
      • drive 1
      • drive 2
      • drive 3
      • drive ...
  • chassis 3 raidz vdev
    • sled 1 raidz vdev
      • drive 1
      • drive 2
      • drive 3
      • drive ...
    • sled 2 raidz vdev
      • drive 1
      • drive 2
      • drive 3
      • drive ...
    • sled 3 raidz vdev
      • drive 1
      • drive 2
      • drive 3
      • drive ...
    • sled 4 raidz vdev
      • drive 1
      • drive 2
      • drive 3
      • drive ...

 

[Sakuru]

Can ZFS support a triple nested RAIDZ pool?

No.

I think your best option would be to go with 8 drive RAIDZ2 VDEVs. Each VDEV would contain drives from 8 different chassis.