Build Advice - FreeNAS ISCSI for ESXi Cluster

[potatocake]

I think I have a case of having read far too many articles, guides, and forums that I've found myself in a circle and I'm still not positive on my best course of action. I’m hopeful that some dialogue from the community may solidify my plans before we go live and changing becomes difficult. Apologies in advance at the length of the post, but I'm a firm believer in too much information is better than not enough.

We are a small business with a growing hosted solutions space that our current setup (ESXi, individual hosts, raid 6 datastores) are not cutting it anymore. The biggest factor being that there's no high availability. We have backups and spare parts, but a failure recovery is a manual process. In addition to HA and balancing, we're looking for scalability. Ideally, we'd like to reuse as much of the hardware we already have as we can. We do have budget to spend on this, but not SAN money.

Our primary purpose is as a Veeam Cloud Connect provider, in that much of our needs are a storage location for backups. These are largely never used repositories that on average keep about 2 weeks of backups, taken nightly. We also have a small amount of hosted VM's and are looking to expand that offering, so HA is a big priority for that.

Due to our region and data residency requirements we must keep all the data in house, and we can't leverage any cloud services for any purposes.

As mentioned, we are willing to spend to do it right (as right as you can without a proper SAN) but obviously considering a cost vs. what you get scale. But here's what we've got right now to work with.

1x 4U Supermicro Server

• 1x Intel Xeon E3-1245 v6 CPU
• Supermicro X11SAE-O Motherboard
  • Integrated Dual Intel 1GB NICs
• 64GB of RAM
• Areca ARC-1883ix-24-2G 24 port SAS/SATA RAID controller
• 24x 4TB Seagate Exos 7200rpm SATA/6G drives

1x 3U Norco Server

• 1x Intel Xeon E3-1245 v6 CPU
• Supermicro X11SAE-O Motherboard
  • Integrated Dual Intel 1GB NICs
• 64GB of RAM
• Areca ARC-1883ix-16-2G 16 port SAS/SATA RAID controller
• 16x 4TB Seagate Exos 7200rpm SAS/12G 4kn drives

1x 3U Supermicro Server

• 2x Intel Xeon E5-2620 v4 CPU
• Supermicro X10DRi-T Motherboard
  • Integrated Dual Intel 10GB NICs
• 96GB of RAM
• Areca ARC-1883ix-16-2G 16 port SAS/SATA RAID controller
• 16x 4TB Seagate Cheetah 15krpm SAS/6G drives

1x 2U Supermicro Server

• 2x Intel Xeon E5-2640 v4 CPU
• Supermicro X10DRi-T Motherboard
  • Integrated Dual Intel 10GB NICs
• 96GB of RAM
• Areca ARC-1883ix-16-2G 16 port SAS/SATA RAID controller
• No allocated drives (plenty of 4TB/600GB available)

Random (potentially) relevant parts available for the build

• 4x Mellanox PCIe MCX4121A-XCAT Dual 10GB NICs
• 2x 500GB Samsung 970 Evo nvme drives
• 8x 240GB Patriot Burst SSD drives
• 4x 500GB Mushkin Triactor SSD drives
• All sorts of 4TB SAS/SATA drives
• All sorts of 300/600GB 15k SAS drives

As mentioned, we’d like to re-use what we can, as it makes sense but discarding the stuff and getting new stuff as needed is totally fine.


My Loose plan, thus far.

The 2U (ESXi 6.7) and 3U (ESXi 6.7) Supermicro would connect to the datastores on the 4U (FreeNAS 11.1) via ISCSI and not use their local datastores. The 4U would replicate to the 3U Norco for redundancy. The objective would be to carve out some of the available storage (5-10TB) and make it as fast as possible to use as the datastore running the VM’s themselves, where performance is more important. And the bulk of the storage to be used as storage repo’s mounted by the VM’s where the performance is less important.

• 2U Supermicro
  • Discard the 15k drives, and do a mirrored ESXi install on two 240gb SSD drives.
  • Put a Mellanox card in it, and yank the areca.
• 3U Supermicro
  • Do a mirrored ESXi install on two 240GB SSD drives.
  • Put a Mellanox card in it, and yank the areca.
• 4U Supermicro
  • Install FreeNAS on mirrored 240GB SSD drives.
  • Pass-through the 24x 4TB drives on the Areca.
  • Use nvme (two?) drive for SLOG.
  • Use another SSD drive for ZIL.
  • I’m not 100% on how to split up the data, what sizes and what ZFS to get a balance of my objective above. We’ve plenty of spare drives, so swapping out a failed drive is no issue, but having more resiliency would be better, such as ZFS2 for two drives just in case. But overall the split is something I just keep changing over and over as I read more articles on how to best approach it.
  • Replicate datastores to the 3U Norco server for redundancy.
• 3U Norco
  • Install FreeNAS on mirrored 240GB SSD drives.
  • Pass-through the 16x 4TB drives on the Areca.
  • Use nvme (two?) drive for SLOG.
  • Use another SSD drive for ZIL.
  • Receive replication from 4U Supermicro.

Other than the obvious questions of advise on how to do this better if you were in my situation, my biggest question is if I am wasting potential out of the 4U Supermicro and 3U Norco by having them only do FreeNAS, rather than use their CPU resources as members of the cluster with the other two. Such as installing ESXi on the 4U, and basically following the exact same plans for it except the FreeNAS portion of it will be on a VM rather than physical.

I realize this is a tall order but any advice and critique would be appreciated. I’d be happy to pay for someone’s time as well if I can be directed to a consultant that is recommended.

 

[kdragon75]

My big question is what do you mean/expect when you say HA? FreeNAS does not have any type of live replication or automatic failover. iXsystems TrueNAS products do support failover clustering but you would have to buy their servers.

 

[kdragon75]

I have been looking into setting up a lab with pure FreeBSD and using HAST/ctl HA but even then its not automated, you would need to write the scripts to detect and trigger failovers.

 

[potatocake]

The HA side is at the ESXi level, I don't need that in the FreeNAS (beyond replicating FreeNAS box to FreeNAS box, for backup purposes.

I'm looking for build ideas on the servers, and the ideal config of the zvols, etc.

 

[kdragon75]

ESXi, individual hosts, raid 6 datastores

The HA side is at the ESXi level

Not without vCenter. I didn't see anything about getting a vSphere license in your post.
I know this isn't the FreeNAS side of things but I just want to cover this from all sides.

I'm looking for build ideas on the servers, and the ideal config of the zvols, etc.

You will want at least 2 pools. One should be striped mirrors for IO performance. This is where the VMs Data disks should be located. The other should be a RAIDz2 no wider than 8 drives per vdev with spares on hand. This would be the capacity pool for the backups. Its hard to even think much further without knowing you RPO and RTO. No to mention the number of vdevs, L2ARC size, and SLOG config without knowing your IO workload profile. I can say that we also need to look at the applications your running so we can set ideal zvol record sizes and calculate a reasonable L2ARC size.