ZFS Topology Recommendations (60 drives)

Steffwiz

Cadet
Joined
Jul 5, 2019
Messages
3
Hi all,

My current server (24-bay chassis) consists of 4x 6-wide RAIDZ2 vdevs with 10TB drives. I'm sitting at 95% used and one of the drives is faulted. With the sale of 18TB drives, I'm looking to expand, but struggling to land on a new ZFS topology.

I'm not happy with 6-wide RAIDZ2. It wastes too much space and I don't need the performance. I have a pool of mirrored SSDs for VMs/apps/etc. The HDD pool is strictly for Plex, mostly sequential reads and writes.

I have a 45-bay chassis on standby that I can use, and am thinking about eventually upgrading to a 60-bay chassis, so I'm looking for a topology that can fully utilize a 60-bay chassis. I'm not interested in hot-spares.

Initially I was thinking of changing to 3x 15-wide RAIDZ3. I would buy 15x 18TB drives and re-use 15 of the 10TB drives from the old pool and use the rest as cold spares. I can then add another vdev of 15 when I upgrade to a 60-bay chassis.

Alternatively, I could run 3x 10-wide RAIDZ2. I would buy 10x 18TB drives and re-use 20 of the 10TB drives from the old pool. I can then add another two vdevs when I get the 60-bay chassis. This configuration has the same space efficiency (80%), but is easier to upgrade and has better IOPS performance (although that isn't needed). I guess the only drawback is it is slightly less fault tolerant.

Anyone have thoughts on ZFS topology with 60 drives? Thank you for any insight you can provide.

Hardware:
Motherboard: SuperMicro MBD-X11SSH-LN4F-O
CPU: Intel Xeon E3-1230 V6
RAM: 32GB DDR4-2133 ECC
HBAs: 3x LSI 9211-8i flashed to latest IT fw
Case: Norco 4224
Drives: 24xWD100EMAZ
TrueNAS-12.0-U8
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
You don't have the memory, even for your existing pool. Performance is going to get worse and it is not uncommon to hit a point where it feels like you've just hit a brick wall. With 240TB of raw space, you should probably have 64GB-128GB of RAM. With 45 * 18 ~= 800TB of raw space, you should be solidly at 128GB minimum, probably 256GB. Problems can exist both with performance and also the ability to deal with problems during pool import.

15-wide RAIDZ vdevs are not recommended. The typical guidance is no more than 12 wide.
 
Joined
Jun 15, 2022
Messages
674
Last edited:
Top