Home ESXi Cluster with FreeNAS Storage

[boston243]

Hi all,

I've been working on building an ESXi cluster for home use. I'd like to build it with a home-ish budget, but with (near) enterprise-level redundancy, backups, etc. I've been mixing and matching consumer-grade hardware for 5+ years and it's time for an upgrade.

The goal here is to have a 3-node ESXi cluster for compute and use FreeNAS for media storage and VMware datastores.

For VMs, I'll be running (mostly Windows Server 2016 but perhaps Linux where appropriate):

• Two Plex Media Servers
• Two domain controllers
• One web server for Sickbeard, CouchPotato, Headphones, XDM, HTPC Manager
• One download server (running Deluge, most likely)
• One BDR server for Veeam (VM backups)
• One application server for miscellaneous things like network monitoring, Madsonic, DDNS
• vCenter

For physical servers, my plan is the following:

NAS01: Dell R710, 2x Xeon L5520, 48GB RAM, 4 NICs

• FreeNAS bare metal on Crucial MX300 SSD (in DVD caddy)
• Dell PERC H200 in IT mode - 6x8TB WD Red in RAIDZ2 (for media storage from SMB share)
• Two PCIe SSD adapters - 2x275GB Crucial MX300 SSDs in mirror, for use as ESXI datastore shared via NFS

NAS02: Synology DS413j

• Two 2TB HDDs in RAID1 for location of Veeam repository and miscellaneous backups (hardware configs, etc). Shared via SMB. Synced to Backblaze B2.
• Two 275GB Crucial MX300 SSDs in RAID1, for use as ESXI datastore shared via NFS

ESX01: Dell R510, 2x Xeon L5630, 32+GB RAM, 2 NICs

• ESXi on Samsung 840 Evo 120GB SSD in PCIe adapter
• Dell PERC H200 in IT mode - VT-d passthrough to FreeNAS VM. This is to use up the drives from my previous setup. 4x4TB WD Red in RAIDZ1 (media storage), 4x3TB WD Red in RAIDZ1 (media storage), 2x4TB mirrored enterprise drives for ESXi datastores shared via NFS

ESX02: Dell R210 II, 1x Xeon E3-1230 v2, 32GB RAM, 2 NICs

• ESXi on SSD (TBD)
• Will only host one of the Plex servers - used for low power usage and capable compute resources
• Storage TBD - can hold up to four SSDs

ESX03: Lenovo M900 Tiny, Intel i7-6700T, 24GB RAM, 1 NIC

• ESXi on SSD (TBD)
• Will only host one of the Plex servers - used for low power usage and capable compute resources
• No datastores

This will run on an appropriately VLANed network with redundant layer 3 switching and an Ubiquiti EdgeRouter Lite. It will all be backed by UPSes and all items will be rack mounted.

Much of these (for example, the actual servers themselves) are already purchased. The R710 is already up and running. But I don't have any of the virtualization set up yet, as I'm still hammering out the proper architecture. Which leads me to my questions:

• From what I've been reading, NFS seems like the best protocol to use for VMware datastores in this use case, rather that iSCSI. Is this correct, and does anyone have any suggestions?
• I'm trying to distribute datastore storage as much as possible to avoid single points of failure. However, I'm not obsessed with full hyper-convergence. For example, VMs on the R510 will run off of the datastores on the R710 and Synology, because FreeNAS is virtualized on the R510 and I don't want to open that can of worms. Given that I'm keeping datastores on both NASes, my impression is that something like Starwind or VMware vSAN is out of the question. Is that correct?
• I'm planning on using (good quality) consumer-grade SSDs here. Will these be able to handle the load of my VMs?
• The R210 II will be running an ESXi datastore, but it won't be running FreeNAS, thus no software RAID. It seems like I need a good quality RAID card, as this will have to be hardware RAID. Any recommendations?

In addition to these questions - please let me know if there's anything I'm doing wrong or could be doing better.

Thanks!

 

[Stux]

Biggest most important detail when considering using FreeNAS NFS or iSCSI for ESXi datastores is that sync writes to anything except pure SSD storage, will require a good SLOG drive for decent performance.

All ESXi writes to NFS stores are sync writes, this ensures VM consistency. And best practise would be to force sync=always on VM datastores hosted by iSCSI too, thus forcing all those writes to be sync writes as well.

Thus, the write performance of your VMs will be limited by the sync write capability of your data pool. Which can be very very low. Unless you have a SLOG, in which case it will be bottlenecked by your SLOG performance.

Top picks for SLOG drives:
https://www.servethehome.com/buyers...as-servers/top-picks-freenas-zil-slog-drives/

Best SLOG drive for <40gbps networks is the Intel DC P3700. Depending on budget, you may want to settle for less.

 

[boston243]

Hi Stux,

Thanks for the advice - the datastores will be on pure SSD storage, so hopefully I won't need a SLOG. The only exception will be for secondary, non-OS drives, and I'm willing to see what the performance is like on those before setting up a SLOG.