New TrueNAS for ESXi and SMB Storage

[Herr_Merlin]

I've grown out of my personal dual ESXi Server solution with each hosting a TN with disk/SSD/nvme attached.

I want to build a cheap new "main" storage.
I've got some Intel Servers with 24x 2.5 SAS/SATA laying around. A ton of 1.8TB SAS 10k HDDs. 8x2TB SATA SSDs from one virtualized TN, which will be decommissioned with this step.

The servers I can put full of memory but I don't need as far as I think. Maybe 128 or 256GB should be fine. CPUs some E5-26xx v2 with something between 6 and 10 cores.
Disk layout here comes the question:
- 2x Radion RMS PCIe devices for ZIL -> that issue solved
- 16x 1.8TB SAS 10k HDDs -> in mirrors about 12.8TB usable storage
- 8x 2TB SATA SSD in mirror-> for metadata and all files below 5MB -> 8TB
-> add some NVMe for L2ARC? or don't?

I am a bit unsure as I would fill the pool to about 8TB instantly, which is over 50% and thus not so ideal. Many files (3 to 4TB) are below the 5MB so that could work. Or go even for bigger files like 16MB to the SSD and all above to the HDDs?
Or go for more HDDs and lesser SSDs?

 

[HoneyBadger]

That's two very different workloads (ESXi block vs SMB file)

I'd be tempted to just put the block workload on the SSDs, the file workload on the HDDs, and call it a day.

The servers I can put full of memory but I don't need as far as I think.

The answer to "how much RAM should I install in this machine?" should be "Yes" - stuff it with as much as you can!

 

[nap]

That's two very different workloads (ESXi block vs SMB file)

I'd be tempted to just put the block workload on the SSDs, the file workload on the HDDs, and call it a day.

If it's two different workloads and you can separate them like that, I would do it that way - iSCSI on SSD mirrors with SLOG and SMB file storage on HDD with z1.