Build to grow to 500TB

[g4m3r7ag]

Hello all,

I need to build a core machine to grow a storage pool to ~500TB usable. I have a 45 Bay SuperMicro but the backplanes are SAS2 and I don't have a machine to connect it to currently so I am planning on getting a 36 Bay SuperMicro with an X10 or X11 board and SAS3 backplanes. The data is just media for Plex, so not terrible performance requirements. Plex will be run on a different machine, so no need for VMs/Scale. My questions are:

1) What drive size and vdev layout would you go with to eventually reach the ~500TB usable space once all 36 bays are filled? I plan on adding one vdev initially and another every couple of months until it's full while I'm migrating data to this machine.

2) Would I benefit from SSDs anywhere on this machine other than boot drives or would as much RAM as possible benefit more? I imagine I would have to use PCIe NVMe boards if I need SSDs since I'm planning to use all 36 bays for 3.5" drives.

Thank you for any help that can be provided.

 

[danb35]

If you're buying new gear, I doubt you'd find SAS2 or older backplanes--but if used, why not SAS2? I mean, sure, if the cost delta is minimal, I'd probably go for SAS3 too, but I don't see it making much of a difference with spinners.

What drive size and vdev layout would you go with to eventually reach the ~500TB usable space once all 36 bays are filled?

I'm kind of hesitant to recommend a 12-disk-wide vdev, as that's kind of borderline, but your stated use case doesn't seem to call for much in the way of IOPS. Twelve disks in RAIDZ2 would give you the capacity of ten of them, so using 20 TB disks, a full chassis would give you ~600 TB (or 540 TiB) of storage, at 200 TB per vdev.

Would I benefit from SSDs anywhere on this machine other than boot drives

Unlikely. There'd be no use at all for SLOG, but there might be for L2ARC. I'd say build it out with at least 128 GB of RAM and keep an eye on the ARC hit ratio.

 

[g4m3r7ag]

If you're buying new gear, I doubt you'd find SAS2 or older backplanes--but if used, why not SAS2? I mean, sure, if the cost delta is minimal, I'd probably go for SAS3 too, but I don't see it making much of a difference with spinners.

The SuperMicro Chassis I was finding with X10 and X11 boards on eBay were SAS3 backplanes. I thought of buying a 12 Bay and connecting that to the JBOD but I wasn't sure about mixing the SAS3 on the server backplane with a SAS2 External HBA/JBOD connection?

I'm kind of hesitant to recommend a 12-disk-wide vdev, as that's kind of borderline, but your stated use case doesn't seem to call for much in the way of IOPS. Twelve disks in RAIDZ2 would give you the capacity of ten of them, so using 20 TB disks, a full chassis would give you ~600 TB (or 540 TiB) of storage, at 200 TB per vdev.

Would a 12 disk RAIDZ2 be relatively safe with that large of drives? I was debating between

4x9x18TB RAIDZ2 which would be around 504TB
4x9x20TB RAIDZ2 which would be around 560TB
3x12x20TB RAIDZ3 which would be around 540TB

But your 600TB 3x12x20TB RAIDZ2 is another interesting option I hadn't thought of.

 

[Johnny Fartpants]

If it makes you feel better I’ve ran 10TB 15 disk Z3 x 6 vdevs for the last 6 years on about 20 systems and all has been good. Only ever lost one drive at a time and resilver for a full pool (80%) takes approx 48-72 hours. Newer systems run 18TB drives in the same config and resilver for a 50% full pool is about 4-5 days.