Insanity check; mainly on CPU

[jenksdrummer]

Use Case: CIFS and iSCSI storage megatank; VMWare and/or Hyper-V hosts (2-8), VM Workloads of up to 80 VMs per host. Some client-connectivity. I have 2x 10gb connectivity between storage and hosts as well as clients; and wish to potentially soak that network with throughput; may upgrade later to 2x 40gb storage connectivity.

Initially this box will be a VM host until I decide it's time to move to HA for resiliency, then I'll add a pair of hosts to run the workloads...and this build will get converted to being storage - which is why I'm asking if the spec will meet the demand (and, ideally, then some - really aiming to build it once and forget about it!). Of note, the HBA will either be sourced separately or flashed to IT-Mode; depending on warranty status at the time; so that is why the IR model is being spec'd below.

Also, not looking towards refurb units. Yes, can be had cheaper, but, TBH, this isn't that expensive compared other options out there and one would still have to supply at least some of the parts and/or wonder if the box will drop off because it's previous owner didn't believe in HVAC. ;)

Here's the build bits:
* Supermicro Barebone: https://www.supermicro.com/products/system/2U/6028/SSG-6028R-E1CR12H.cfm

* RAM: 128GB (4 x 32GB) Samsung modules sourced from Supermicro directly (not that much more expensive than Amazon for similarly-branded modules).

* HDD: 12 x 10TB WD Gold SATA disks

* SSD: 2 x Samsung 950 Pro 512GB SATA (maybe...reusing...provided they meet the specs and I see a performance advantage!)

* CPU: Here's where I have the question! :)
This box is a 2 CPU board (can't find a single-CPU options for that case that allows for > 64GB RAM).
Initially I was thinking I'd put a pair of E5-2630 V4 in it; but then I got to thinking that likely 20c/40t might be overkill?
At the lowest end, a single E5-2620 V4, as I want to stay above 2Ghz base clock; which is 8c/16t, and leave 2nd CPU empty unless performance, RAM, or PCI Slots dictate that I need a 2nd CPU.

Items of interest:
* Compression: L4Z works well and doesn't load the CPU very much, but in testing I saw a significant increase in compression ratios with GZIP; only performance impact with my test box (QNAP TVS-1282T w/ 32GB RAM running it via USB stick boot) was at the highest setting...test would run consistent 5.5gbps and drop off about 20% per change (to gzip 'fastest', to medium), but at the highest level would never get over 1gbps; which was a significant taper...but the compression rates were pretty tasty!

* Deduplication: tests with the same box would work well for a bit, then performance would tank and get very up/down - run at 5.5gbps for maybe 30s, then drop to nothing for 30s-1m before I'd kill the test. Probably the lack of RAM contributed. Tried using SSDs for L2ARC, only delayed the inevitable. Would like to use deduplication with the vm workloads in play since that may help with storage consumption / snapshot replication that may eventually come into play.

Personal notes:
I've heard both ways, that a fast CPU trumps thread/core count with FreeNAS, but also that it threads processes quite well; which is where I get the CPU scaling question.

Anyhow... sorry for the long read, this is the shorter version after a few re-writes! :D

(Edit: changed my mind to go with the 2U / 12bay unit)

 

[Ericloewe]

This box is a 2 CPU board (can't find a single-CPU options for that case that allows for > 64GB RAM).

All the X10SR* boards should handle at least 128 GB with no problems, 256 GB with some caveats.

As for CPU choice, most people get away with the cheaper Xeons, but most people don't want to host 80 VMs with GZIP and dedup. GZIP is a massive CPU hog and dedup is sucky in every way, so a more capable CPU might be interesting for you.
The upcoming zstd compression algorithm will be a more balanced option than GZIP, with compression levels from "crazy fast, not much compression" to "still faster than GZIP, but compresses about as well as GZIP would", but don't expect miracles in terms of speed if you want something that compresses more than LZ4.

HDD: 12 x 10TB WD Gold SATA disks

Are six vdevs going to keep up with 80 VMs? Honest question, it depends on what the VMs are doing.

SSD: 2 x Samsung 950 Pro 512GB SATA (maybe...reusing...provided they meet the specs and I see a performance advantage!)

In what role? They're no good for SLOG and fairly crappy for L2ARC.

Deduplication: tests with the same box would work well for a bit, then performance would tank and get very up/down - run at 5.5gbps for maybe 30s, then drop to nothing for 30s-1m before I'd kill the test. Probably the lack of RAM contributed. Tried using SSDs for L2ARC, only delayed the inevitable. Would like to use deduplication with the vm workloads in play since that may help with storage consumption / snapshot replication that may eventually come into play.

Dedup has always been crappy. It has a ton of cliffs for performance to fall off of and many of its use cases can achieve similar benefits with clones. Even 256 GB of RAM can be tight if you use dedup on most of your ~54 TB pool.

I've heard both ways, that a fast CPU trumps thread/core count with FreeNAS, but also that it threads processes quite well; which is where I get the CPU scaling question.

SMB uses one thread per connection, hence why single-threaded performance matters to most users. If you're serving 20 clients via SMB, more cores will definitely help. Most other things scale decently with core count, too.

 

[jenksdrummer]

Thank you for the reply, a lot of good info in there.

Noted: 850 Pro SSD's not that great to use. It's what I had when I was testing; seems I set myself up for poor results. Thank you. Justification for something more appropriate to test with. ;)

Noted: CPU - I think I'll opt for the E5-2630 V4 as originally planned. Slightly faster CPU plus 25% more cores/threads, and slightly larger cache, while not terribly more expensive. To that, I can always opt for a 2nd CPU later if performance metrics show it's warranted; but starting off on a better CPU.

Noted: Dedupe and Compression comments - Dedupe is a want more than a need. Compression, at higher levels, is more a want than a need. L4Z being fine as it is, and very low-impact from what I've seen. Nice to know the upcoming compression method, that may be the best option, when available. :)

Also, I mentioned up to 80VM per host (6) earlier. I was basing that solely on CPU which was a mistake... RAM is generally the first constraint; more realistic it'd be around 32-64VMs/host * 5 hosts (N+1), for a total max between 160-320VMs, if I manage to get all 6 hosts purchased. After I get this box, it'll be a minute before I get the first 2 hosts and transition to a HA-pair+storage topology vs an AIO.
As for workload, truthfully, very minimal as this is for a personal lab environment; where I spin up a business' infrastructure to test and vet upgrades/processes/projects/etc before going to the live environment. It's saved my butt a number of times from whitepaper gremlins.
It will also be used for storing backups and personal data, but, that demand is minimal other than I want a lot of speed for those backups to complete quickly.

I may at some point turn this into a commercial platform where I offer either virtualization services or storage, but, for now, it's not, plus I'd want to get this where there's less potential single-points-of-failure in the picture.

 

[jenksdrummer]

So, a bit of an update; changed up a bit; turns out my work has a reseller relationship for Supermicro, so, just scored a really good deal on a built system. ;)

5029P-E1CTR12L, Intel Xeon Silver 4110, 192GB ECC RAM; being the core of it...