All flash, jbod expandable FreeNAS build.

[AngelFreak]

Hey all, im building a new FreeNAS system.
The internal disks gonna be all NVMe, and SSD for boot drive.
I wanna have expandability in the form of a jbod.

The server will run as a storage server for VMware, SAMBA shares, and some backup's.

I'm in doubt then it comes to the SAS card, so i hope someone can help.
As far as i can tell, they both work, but is one better supported than the other ?

Help / comments are more than welcome!!

Parts list:

1x SuperServer 1029U-TN10RT
https://www.supermicro.com/products/system/1U/1029/SYS-1029U-TN10RT.cfm

2x Intel Xeon Gold 5122 (4 cores at 3.60 GHz, Turbo 3.70 GHz)
https://ark.intel.com/products/120475/Intel-Xeon-Gold-5122-Processor-16_5M-Cache-3_60-GHz

2x Intel DC P4500 240GB, 2.5in SATA 6Gb/s, 3D1, TLC (RAID 1 boot drive)
https://www.intel.com/content/www/u...-s4500-series/dc-s4500-240gb-2-5inch-3d1.html

2x Intel DC P4500 4TB NVMe 2.5 in PCIe NVMe 3.1 x4 (Storage)
https://www.intel.com/content/www/u...dc-p4500-series/dc-p4500-4tb-2-5inch-3d1.html

2x Samsung 64 GB LR ECC DDR4-2666 LRDIMM SDRAM M386A8K40BM2-CTD
https://store.supermicro.com/memory/64gb-ddr4-2666-mem-dr464l-sl03-lr26.html?utm=testedlink

1x LSI SAS 9300-8e ⁄ SGL 8-Port extern LSI00343 von Avago (JBOD) / 1x LSI SAS 9305-16e ⁄ SGL 16-Port extern von Avago (JBOD)

Thanks in advance.

 

[jro]

Looks like that's going to be one hell of a system. How many VMs do you plan on running on it?

Both the 9300 and 9305 LSI cards will work, but they might be overkill compared to the older, 6Gbps cards. Regardless, JOBD expandability is certainly supported.

Here's some discussion from our resident experts on the 9300/9305: https://forums.freenas.org/index.php?threads/does-freenas-support-the-lsi-sas-9305-16i.44069/

 

[AngelFreak]

@jro In the beginning only about 9 VM's will have their storage needs meet by this server.
That said over time (hopefully not to long) it will run 50>. The server will also replace a Windows file share server.

Thanks a lot, that's one of the posts i read, looks like the LSI SAS 9300-8e will be the most reasonable choice.

 

[jro]

Will the P4500's be for SLOG and L2? You'll definitely want at least a SLOG for hosting VMs, especially when you get to 50+. 4TB for a SLOG will be waaaay overkill though; with the default tuning it only has to store 5 seconds of writes. Even if you have quad 100Gbps totally maxed out, you would only need a 256GB SLOG to hold 5 seconds worth of writes. Just something to consider, I know those 4TB Intel SSDs are pricey.

As you add more VMs to the system, keep an eye on ARC size and hit rate with the built-in reporting functions (including the cool new netdata service), iostat, and arcstat.py. If you're blowing out ARC and/or getting a lot of cache misses, you'll want to think about 256 GB of memory. 256GB of memory is probably going to be a more worthwhile investment than an L2 for this kind of build, but keep an eye on ARC to help inform that decision.

 

[AngelFreak]

Yes and no.
The main use for the P4500 NVMe disks are storage for the VM's. But i will probably use a partition on the 4TB NVMe drive as the SLOG, and do not plan to use L2 Cache.

Sure, it makes sense to add more RAM, rather than adding a L2 cache.
When / if i add the jbod i will probably increase the ARC size, or add a L2 cache.

Thanks for the pointers, it's great to get your input.

 

[jro]

Putting the SLOG on a partition on your storage pool likely won't provide you any benefit. The ZIL, by default, sits on the storage pool already. The point of a SLOG is to move it off the pool, ideally onto a much faster, much smaller device. The "S" in "SLOG" stands for "separate" after all :)

I have seen use cases where the SLOG drive is broken up into partitions and the SLOG is striped over top of all these partitions.

Since your primary storage pool will be on NVMe, I think the only thing that'd be faster (in terms of access latency) would be an Optane drive or an nvDIMM. You might see what performance looks like without a SLOG.

 

[AngelFreak]

Yes sure, i get that it's not really a SLOG.
Like you said i don't really wanna put a SLOG in the system, since the NVMe is the fastest device i am gonna use in this build anyways.

We have another build, that uses a partition on a NVMe disk as a "SLOG". The only benefit is that in theory it limits the fragmentation on the disk.
But again, not really a issue on NVMe.

 

[AngelFreak]

Updated parts list:

For anyone curious, this is the hardware that will be used.
The CPU has been downgraded, and the NVMe had a model bump up.

1x SuperServer 1029U-TN10RT

2x Intel Xeon Silver 4122 (4 cores at 2.60 GHz, Turbo 3.00 GHz, DDR4-2400)

2x Intel DC P4500 240GB, 2.5in SATA 6Gb/s, 3D1, TLC (RAID 1 boot drive)

2x Intel DC P4510 4TB NVMe 2.5 in PCIe NVMe 3.1 x4 (Storage)

2x Samsung 64 GB LR ECC DDR4-2666 LRDIMM SDRAM M386A8K40BM2-CTD

1x LSI SAS 9300-8e⁄SGL 8-Port extern LSI00343 von Avago (JBOD)