Best RAID layout for 36-disk storage?

[tbaror]

Greetings all,

I am building a high-capacity chassis for our datacenter to become our new Xen spin-up vm's and part of it as cifs nas . The storage hw is as follows:
Chassis: Supermicro 4U Storage Server Up 60 H.D. & 6 NVME SSG6048RE1CR60L Integrated HBA LSI 3008
CPU: 2x E5-2620V4
Memory: 512G RAM
Disks: 36x Seagate 3.5”, 10TB, SAS12Gb/s,7.2K RPM, 256M, 512E,Performance (TATSU)Helium ,4x Intel P3600 800GB, NVMe PCIe 3.0,HET MLC 2.5" 20nm 3DWPD
Network: SIOM 4 x port 10G SFP+,Intel XL710 AOC MTG I4S-O, 2x port GbE Intel i350
I have done many of research with regards to the RAID strategy for the drives, however, it seems there is no optimum configuration for 36-disk chassis. Evidently,18x 2-RAIDZ2 vdevs are not good, nor is a single 36+1 RAIDZ2 or RAIDZ3 vdev.
Can someone please shed some light on the optimum RAID config for a 36-disk chassis trade of optimum performance/space/fault tolerance that will be used as virtual spin-up and cifs servers? I thought about creating a zpool with different RAIDZ vdevs like 3 x12 z2 vdev or 6x 6 z2 , but I am not sure if that is best practice.
Thanks.
Tal,

 

[saikee]

Haven't worked with so many disks before but you could try different combinations of RAIDZ in FreeNAS GUI to see how much capacity and the number of disks required you could get out of each combination. The zpool will not be built unless you commit so it is one of the ways to find out. Zpool can be expanded but once the RAIDZ is chosen it may not be changed without some serious work.

If you have several vdev that will have to suit your application same as putting data in one partition against several partitions. You do have jails in FreeBSD systems to control the data usage.

 

[danb35]

If you're using this as a datastore for virtual machines, really the only answer is mirrors.

 

[c32767a]

Can someone please shed some light on the optimum RAID config for a 36-disk chassis trade of optimum performance/space/fault tolerance that will be used as virtual spin-up and cifs servers? I thought about creating a zpool with different RAIDZ vdevs like 3 x12 z2 vdev or 6x 6 z2 , but I am not sure if that is best practice.
,

There's lots of different posts on the subject and it depends on what your workload looks like and how fat your wallet is. Not everyone can afford to deploy sufficient storage on mirrored vdevs.

I don't have all the answers, but our workload is very linear. We have been storing large sequential data files and backups on 16 disk Z3 vdevs for 2 or 3 years with good success.
Our default design has 64 G of RAM and 32 disks in 2 16 disk vdevs and can saturate a 10G link.
We use smaller mirrors of SSDs for VMs over NFS in the same chassis and that works pretty well too.

My strongest recommendation is to build a prototype and test it heavily with your workload.. iperf and dd are great for raw metrics, but they don't tell the whole story. Even after iperf and dd looked great, it still took us several iterations to find what worked well for our specific workloads.

 

[tvsjr]

Striped mirrors. Don't exceed 50% of pool capacity or fragmentation will kill you. Use one SSD for L2ARC, use another for SLOG (underprovision it).

 

[Artion]

Maybe this can help. The best way to go is testing your own hardware to find the best solution.

 

[Stux]

I would consider 3 layouts personally

18x 2way Mirrors.
6x 6way RaidZ2
4x 9way RaidZ2

Depending on if IOPS or stoarage density is more important.

 

[gpsguy]

In addition to @Stux suggestion, you might consider two pools, one with some form of mirrors for your VM's and a second RAIDz2 pool for your CIFS needs.

 

[Nve156]

Hi i
have similar hardware from supermicro with 8tb seagate drive
i decided to go with 6x 6way RaidZ2. but im stating out with 2x 6way RaidZ2
i tested 5x 2way Mirrors, the more pairs i added the better performance i was getting.

my nas now has 2x 6way RaidZ2.
once these get close to filling up and i add another 6way RaidZ2 will all this new data go into these drives?

What configuration did you choose in the end?