High performance storage for VM workloads

[blipblip]

Hi all,
I have been using FreeNAS for various homelab/POC scenarios. I am looking into whether we should go with FreeNAS for production use.

Requirements:
1U form factor only
<250W peak power usage
>=10TB usable space
iSCSI
Support 60 or so low load VM (ESXi, each VM avg < 5 IOPS)
Support 10 or so 'peaky' VMs (RDS; >20 IOPS per VM for 30 or so minutes per day)
Support 10 or so 'heavy' VMs (avg >20 IOPS)

Network:
Cisco Nexus 3K series switches

Currently for FreeNAS path, I have two ideas:
Idea#1:
SuperMicro SuperServer 6018R-TD8
2 x Xeon E5-2620 v4
512Gb RAM
LSI 9300-8i
8 x 4TB 12G SAS 7.2kRPM in 4 mirror pairs
Intel P3700 400G (ZIL/SLOG)
2 x 64GB SATA DOM Boot disk
Chelsio T580-SO-CR

Idea #2
SuperServer 1028U-TN10RT+
2 x Xeon E5-2620 v4
256Gb RAM
LSI 9300-8i
6 x Samsung PM863 3.84TB SSD in 3 mirror pairs
Intel P3700 400G (ZIL/SLOG)
2 x 64GB SATA DOM Boot disk
Chelsio T580-SO-CR

All users currently use 4th Gen Core-i5 laptops with SATA SSD, and I think they expect the same responsiveness when they login to RDS to use office applications.

Do you think idea #1 will provide good enough IOPS performance to support the VMs and user expectation or idea #2 is the best way to go? Or should I go a totally different route?

Also, is there any active/active or active/passive HA solution for FreeNAS?

My other path is to go with HP StoreVirtual VSA cluster. But HP SSDs are $$$$ and I will need to buy twice the amount of disks to get the same usable space (network RAID1).

Thanks for the help!

 

[tvsjr]

So... you want a 1U, <250 watt box that can provide iSCSI services to 80 VMs and 10TB of usable space? Let me introduce you to the "good, fast, cheap, pick 2" theory. In FreeNAS-world, perhaps it's really "cheap, power-efficient, fast, pick two".

Your IOPS calculations suggest a peak of 700 IOPS (which I think is low, but let's go with it). 7.2K drives are typically 50-70 IOPS. So, you'll need a minimum of 10 2-way mirrors, or 20 drives.
You want 10TB usable. To keep performance from tanking, you don't want your pool to exceed 50% utilization. You'd need 2TB drives minimum.

All-flash arrays are pretty new around these parts, but there are a few people running them. Beware write endurance issues... the 3.84TB PM863 is only rated for 5,600TB written, or 1 drive write per day for 4 years. Intel specs some of their enterprise drives up to 10 drive writes per day for 5 years.

HA would require a TrueNAS.

I'd suggest calling iXsystems to see what they can dream up for you, since you have the budget to consider an HP cluster. I highly doubt you'll be able to make this work in a 1U configuration.

 

[zambanini]

you need a better cpu, the system will use much more power, you chose the wrong kind of ssd, with disks you will need at least 10 vdevs.

using freenas as production server is mad. no HA, no hotswap for sas drives (multipath is a b****).
then you will need someone who will tune your network settings.

I agree with tvsjr, that you should really contact ixsystems. I doubt you will find a solution under 30k.

msa2040 with a couple of ssd might fit your needs. not as fast as a all flash TrueNAS system, but cheaper.

you will need to rethink your requirememts. cheap, fast, reliable is a triangle where you can only math two points. less power consume and only 1u kills it.


btw with zfs and iscsi you should only use 50÷ of your netto capacity. otherwise the system will become really slow. so you will need double amount of devices. so no 1u.


a l2arc is also needed.

when I reread your amounts of virtual guests: you absolutly did the wrong math regarding IOs.

just for example: boot storm. windows updates...
even with linux virtual guests the system needs to be 5x -10x mores IOs then your suggested system could provide.

 

[blipblip]

Cheap is not a requirement. It will be nice if it is cheaper than our usual SAN system budget, but not the main objective.

I will have to check the IOPS figure again. We are doing a pilot program, but the size is only about 10% of the projected actual size (in terms of VMs and users). The figure is from the current system for the past week.

Another thought is to use all NVME systems and use 10 x Intel P3700 drives. But those drives use as much power as 15K SAS drives...

The keeping iscsi volumes at <50% capacity utilization is probably a deal breaker (is this particular to iSCSI or is it just the way ZFS operates?)

Thanks for the info guys. I'll give IXSystems a call.

 

[Ericloewe]

Another thought is to use all NVME systems and use 10 x Intel P3700 drives. But those drives use as much power as 15K SAS drives...

Sure, but you need fewer of them to achieve the desired performance.

The keeping iscsi volumes at <50% capacity utilization is probably a deal breaker (is this particular to iSCSI or is it just the way ZFS operates?)

It's inevitable due to how CoW filesystems and abstract block storage interact.

 

[SFoskett]

If cheap is not a requirement, you'd be better served buying a supported enterprise array. Maybe look at Tintri or Pure or Nimble. If you'd like to go cheaper, HP and Dell have some good options. And I'm sure iXsystems could put together a supported HA array for you. But don't try to build this yourself if you want support and enterprise-grade reliability.