FreeNAS + ESXi + iSCSI + 10Gbps Ethernet -- Storage Options?

[kuruption]

This is my second foray into running ESXi VM's using iSCSI from FreeNAS. My original build (going back a few years now) was running ESXi 5.0 with 4x1Gbps teamed NICs (FreeNAS using LAGG/LACP) and some Seagate 1.5TB disks.

This time I want to step up my game a bit...

This is a home lab setup... so I don't need anything super crazy, it's mostly to play/try/experiment and run some VMs. I'm aware I could just throw some drives in the ESXi box and be done, but that's not the point of this effort. ;)

First, my current FreeNAS environment is running on a home-built platform, Tyan S7012 board, 2xX5670 CPUs, 48GB RAM, and two pools: 6x4TB WD Red in a RaidZ2 and Mirrored 64GB OCZ SSDs I had laying around for some jail storage. This was basically being used for backups of my home PCs, media storage, and running Plex and other random jails using the SSDs. I currently do not have any ESXi hosts running off of this.

Second, for my ESXi hosts, I ebay's some Dell R710 rackmounts ($200/ea -- 2xE5507 CPUs, 48GB RAM) followed up with some Mellanox 10Gbps Ethernet cards.

My intent was to use those OCZ SSDs as the primary datastore for ESXi, however... one went dead today, so it sounds like a good time to investigate my options...

Conditions:

• Budget of approximately $300-400 (so don't tell me to buy an array of 2TB NVMe drives)
• Looking for approx. 1TB of usable VM storage
• Both my FreeNAS box and R710s have only PCI Gen.2 slots, but my bottleneck [sans the drives] is still the 10Gbps NICs (1200MB/s)
• I can support another ~6 3.5" drives in my FreeNAS case; beyond that I'll need to consider different cases and therefore blow the budget
• FreeNAS and R710s are directly connected, no switch in between.

So, looking for opinions on my options below, in no particular order, that I think I can get away with for the budget -- and maybe some other options?

Option #1:

• Build a RAID 10 with some currently owned Seagate 2TB drives for VM storage.
• Use NVMe SSDs as L2ARC and SLOG

Option #2:

• 4x 300GB 15K SAS drives in a Raidz1 (or RAID 10?)

Option #3:

• Option #2
• Add some [cheaper] NAND SSDs for L2ARC and SLOG

Option #4:

• Option #2
• Bump FreeNAS memory to 96GB total

Others?

 

[Dice]

You'll want to get some IOPS going (that is - forget about raidz). That is achieved through mirrors. If you've 6st slots left, rotating rust is probably fine. If space is not an issue, you could use consumer grade SSDs.
If you want to run sync=always you'll crave a SLOG device. The SLOG device need power loss protection and high durability for writes and preferably as low latency as possible to do its job. Any fast SSD won't do it's job.
RAM is very important. Max out RAM prior to getting a L2ARC. A couple of mirror vdevs in a pool, coupled with plenty of RAM will mitigate the need for a L2ARC drive. Any SSD would do for L2ARC when that day comes.
Do a search on something like ZIL SLOG L2ARC SSD and you'll hopefully find threads by jgreco that lays out the fundamentals underlying my statements above.

I'd prioritize:
-RAM
-SLOG
-Enough drives to create sufficient space while using the topology of mirrored drives.
-Consider getting a L2ARC.

Cheers.