BUILD Need Suggestion on Building new FREENAS box

[Vikas Saini]

Hi,
I run a small hosting company and currently we host around 250 VM on KVM hypervisor (managed by SolusVM)

I'm thinking to build a new FREENAS box with the following hardware :

Supermicro 6048R-E1CR36L 4U server
Chasis : CSE-847BE1C-R1K28LPB
MainBoard : Super X10DRH-iT with 2 x 10gbit Intel X540 inbuilt
CPU : 2 x Intel Xeon E5-2630V4
RAM : 16 x Certified 32GB DDR4 2400Mhz ecc REG LR-DIMM (total 512 GB RAM)
BOOT DRIVE : 2 x Samsung 120GB SM863 SSD (via Supermicro MCP-220-82609-0N bracket)
2 x Intel 400GB P3700 PCI-E NVME SSD [for ZIL/SLOG]
20 x HGST 6TB 7K6000 SAS Ultrastar [for HDD Storage Pool] in raid10 like setup
10 x Samsung 480GB SM863 SSD + Bracket [for SSD storage Pool] in raid10 like setup
1 x Samsung 480GB SM863 SSD for L2ARC
1 x Supermicro AOC-STG-I2T dual 10GB RJ45 [for connection to 10gbit switch]

1 x Supermicro SSE-X3348T 48 x 10gbit NIC L3 switch

I will create one HDD array (60TB usable) and one SSD array (2400 GB Usable) for storage pools.

I have few questions which I want to discuss.
1) Do I need a ZIL/SLOG device for SSD array too ? since SSD in raid10 setup will be fast enough, I think it will not benefit much. what you say ?
2) same question regarding L2ARC for SSD array ?
3) NVME SSD are worth their price to use as ZIL/SLOG ? Can they really help in improving performance over SATA SSD ?
4) Above setup seems correct for iSCSI target for running VM ?

 

[tvsjr]

It's not usually wise to put a brand-new driver into a Ferrari with all the nannies turned off. Similarly, you're trying to build a Ferrari (at least) here. You're on the outer edge of what most people in these forums are doing.

Just a few quick comments:
NFS/iSCSI pools shouldn't exceed 50% utilization due to fragmentation issues and massive drops in performance. You also haven't factored in any overhead. Using BiduleOhm's calculator (https://jsfiddle.net/Biduleohm/hfqdpbLm/10/embedded/result/), your 20x6TB pool ends up with 53.7TiB at 100% utilization, which you should never hit. If this is a VM datastore, cut that in half. So you're at 26.85TB. Similarly, your SSD array is only about 1.7TiB, cut in half if you're using NFS/iSCSI.
There is no RAID-10 in ZFS world. "Striped mirrors" is the terms you're looking for.
Yes, NVMe SSD (with power loss protection) is a good idea for SLOG (not ZIL).

You'll want to verify those 10gig cards are well-supported. Some are, some aren't.

You might want to consider building a smaller solution, learning a bit, etc. (call it a lab/demo environment) before you go down this path.