Will it Virtualize? Esxi with FreeNAS VM

[Snowy]

Lurking around for a couple months and looking to get started. My goal is to have a homelab that I can create/destroy VM's and experiment in a sandbox while learning basics of managing storage and backing it up for D/R. I'd like to get a plex media server going and see if I can passthrough a usb controller to play some Rocksmith as a fringe benefit.

BUILD MATERIALS:
CPU: Intel Xeon E5-1650 V4 3.6 Ghz 6 core w. VT-d/x and Hyperthreading
Mobo: SuperMicro MBD-X10SRH-CLN4F-O with LSI 3008 SAS onboard
RAM: 16 GBx4 Hynix RDIMM DDR4-2400 288 Pin w. ECC
HDD: (6+1)x2TB WD 3.5" Red 5400 RPM (+1 is a spare; 6 in mirror or Raidz2)
SSD: 2x120GB 2.5" Intel DC S3500 (likely for L2ARC/ZIL)
NVME: 2x500GB m.2 Samsung 960 Evo (with PCI-E adapters)
Case: 833T-653B 3U Rackmount Supermicro Chassis (8 hot-swappable SAS) w. 650 watt P/S
SATADOM: 2x16TB SSD-DM016-PHI Supermicro (for booting esxi/freenas)
UPS: PCM Powercom King PRO - 1320 Watt Rated (I'll be using it to supply a couple other items)
External HDD: 6TB WD Mybook
Graphics: Some basic HDMI video card. Nothing fancy.

I've seen others on this forum get the LSI 3008 to work when flashed to IT mode, so I won't need to get another controller. I figured I will connect 6 WD Red drives to it, and have 2 Intel SSDs to experiment with L2ARC/ZIL.

POOL MANAGEMENT:
Are the following two scenarios feasible when doing ESXI hypervisor with NMVE and FreeNAS? Is one preferable regarding performance/ability for snapshot/backing up safely than the other?

a) Have two pools in FreeNAS (LSI 3008 Passthrough + NVME Passthrough)

The LSI 3008 pool will have 6 HDDs and 2 SSDs (ZIL/L2ARC) used for storing large files (via iSCSI/NFS), and a second pool of mirrored NVMEs for hosting OS's for faster access.​

b) Have one pool (LSI 3008 Passthrough) and let ESXI have direct access to NVMEs

LSI 3008 control 6 HDDs and 2 SSDs like option a), but do not passthrough the NVME to FreeNAS at all, and let esxi manage that directly and use it for OS's and then attach the FreeNAS storage to those VM's via iSCSI or NFS?​

QUESTIONS:
My biggest question is the NVME and how to use it. I haven't fully decided if I am going to have FreeNAS manage ALL datastores for ESXI or do a hybrid. I thought about using NMVE for the L2ARC/ZIL and not buying the Intels at all. But NMVE for that seemed overkill.

Can you see any incompatibilities from my list of parts? Will NVME/SSD work together in the above suggested scenarios or is it unnecessarily redundant?

 

[Spearfoot]

Lurking around for a couple months and looking to get started. My goal is to have a homelab that I can create/destroy VM's and experiment in a sandbox while learning basics of managing storage and backing it up for D/R. I'd like to get a plex media server going and see if I can passthrough a usb controller to play some Rocksmith as a fringe benefit.

BUILD MATERIALS:
CPU: Intel Xeon E5-1650 V4 3.6 Ghz 6 core w. VT-d/x and Hyperthreading
Mobo: SuperMicro MBD-X10SRH-CLN4F-O with LSI 3008 SAS onboard
RAM: 16 GBx4 Hynix RDIMM DDR4-2400 288 Pin w. ECC
HDD: (6+1)x2TB WD 3.5" Red 5400 RPM (+1 is a spare; 6 in mirror or Raidz2)
SSD: 2x120GB 2.5" Intel DC S3500 (likely for L2ARC/ZIL)
NVME: 2x500GB m.2 Samsung 960 Evo (with PCI-E adapters)
Case: 833T-653B 3U Rackmount Supermicro Chassis (8 hot-swappable SAS) w. 650 watt P/S
SATADOM: 2x16TB SSD-DM016-PHI Supermicro (for booting esxi/freenas)
UPS: PCM Powercom King PRO - 1320 Watt Rated (I'll be using it to supply a couple other items)
External HDD: 6TB WD Mybook
Graphics: Some basic HDMI video card. Nothing fancy.

I've seen others on this forum get the LSI 3008 to work when flashed to IT mode, so I won't need to get another controller. I figured I will connect 6 WD Red drives to it, and have 2 Intel SSDs to experiment with L2ARC/ZIL.

POOL MANAGEMENT:
Are the following two scenarios feasible when doing ESXI hypervisor with NMVE and FreeNAS? Is one preferable regarding performance/ability for snapshot/backing up safely than the other?

a) Have two pools in FreeNAS (LSI 3008 Passthrough + NVME Passthrough)

The LSI 3008 pool will have 6 HDDs and 2 SSDs (ZIL/L2ARC) used for storing large files (via iSCSI/NFS), and a second pool of mirrored NVMEs for hosting OS's for faster access.​

b) Have one pool (LSI 3008 Passthrough) and let ESXI have direct access to NVMEs

LSI 3008 control 6 HDDs and 2 SSDs like option a), but do not passthrough the NVME to FreeNAS at all, and let esxi manage that directly and use it for OS's and then attach the FreeNAS storage to those VM's via iSCSI or NFS?​

QUESTIONS:
My biggest question is the NVME and how to use it. I haven't fully decided if I am going to have FreeNAS manage ALL datastores for ESXI or do a hybrid. I thought about using NMVE for the L2ARC/ZIL and not buying the Intels at all. But NMVE for that seemed overkill.

Can you see any incompatibilities from my list of parts? Will NVME/SSD work together in the above suggested scenarios or is it unnecessarily redundant?

Nice selection of toys!

For a home lab, I doubt you will need an L2ARC device. These aren't generally needed unless your server is being pounded by multiple users in a production environment.

You will definitely want a SLOG device for your VM datastore. But note that, in a home lab, you can just turn off synchronous writes for the VM dataset in lieu of installing a SLOG. This isn't recommended for any kind of production system, but this is a home lab, right? If you decide to install a SLOG device, an Intel P3700 is good but a DC S3700 SSD will be less expensive. The 3700-series devices are write-optimized, as opposed to the read-optimized 3500 line, and have all of the other characteristics you want in a SLOG: power protection, low latency, fast write speed, high durability. However, if you already own the S3500s, they will work.

If you set up a separate VM pool on the mirrored Samsung NVMe devices, this is the pool where you will want to install the SLOG device(s).

One possible wrinkle I can think of is this: passing the NVMe devices through to the FreeNAS VM may be dicey... It should work, in theory. But I've never tried it. YMMV, and it may very well work fine. Perhaps someone else who's tried will comment and let us know.

One last observation: I have two ESXi home lab servers much like you're planning on building. Both run NFS and iSCSI-based ESXi datastores based on a RAIDZ2 array with an Intel DC S3700 SSD SLOG device. Performance is just fine, for my purposes: firing up various VMs to compile and test programs, experimenting with software and operating systems, etc. For the best VM datastore performance though, the standard advice is to use mirrors instead of any kind of RAIDZn setup.

Good luck!

 

[Snowy]

Great feedback, thanks for the reply.

I checked out the weight of proposed case. Something like 65 lbs! My plan was to do the coffee table home rack (since it would be the only thing in it), but I have no idea if that'll hold. I know someone who's into a bit of wood working, so I'll ask him.

I think I may order all the above, possibly swapping in 3700's if I can find them cheap, and explore the different combinations. Probably be a few more months before Im up and running, but that will help spread out the cost. Good advice on the L2ARC.

I'll order the case and work on the mounting and then order the parts.

 

[Spearfoot]

FWIW, the good, better, and best Intel selections for SLOG devices run sorta like this:

• Good: Intel SSD DC S3700 Series SATA, inexpensive, 10 drive writes per day
  
• Better: Intel SSD 750 Series PCIe, mid-level price, 70GB writes per day
  
• Best: Intel SSD DC P3700 Series PCIe, costly, 17 drive writes per day

All three have the power protection, low latency, fast writes, and high durability you need in a SLOG device.

You can find the S3700 SATA SSDs on eBay all the time, in new condition, often packaged as OEM parts sold by Lenovo or Dell. For example, I bought one of these: 4XB0F28636 Lenovo 100 GB SSD - hot-swap SATA 6Gb/s 2.5" ThinkServer SA120 - Main, which is just a 100GB Intel DC S700, as specified by the OEM part number in the description (SSDSC2BA100G301), installed in a Lenovo hot-swap bracket. I removed it from the bracket, tested it -- it was brand spankin' new, zero runtime hours! -- and added it to my hoard as a spare.

 

[Snowy]

FWIW, the good, better, and best Intel selections for SLOG devices run sorta like this:

• Good: Intel SSD DC S3700 Series SATA, inexpensive, 10 drive writes per day
  
• Better: Intel SSD 750 Series PCIe, mid-level price, 70GB writes per day
  
• Best: Intel SSD DC P3700 Series PCIe, costly, 17 drive writes per day

All three have the power protection, low latency, fast writes, and high durability you need in a SLOG device.

You can find the S3700 SATA SSDs on eBay all the time, in new condition, often packaged as OEM parts sold by Lenovo or Dell. For example, I bought one of these: 4XB0F28636 Lenovo 100 GB SSD - hot-swap SATA 6Gb/s 2.5" ThinkServer SA120 - Main, which is just a 100GB Intel DC S700, as specified by the OEM part number in the description (SSDSC2BA100G301), installed in a Lenovo hot-swap bracket. I removed it from the bracket, tested it -- it was brand spankin' new, zero runtime hours! -- and added it to my hoard as a spare.

I think I'll go with the best of that list for under $130 a piece. I am way up there on my budget but I'll be spreading the buts out over a few weeks.

Once I get the thing mounted I'll be ordering all the parts. Ups just came since it was $30 off I went for it already.

Thanks for the price hunting help! Do you trust eBay sellers with RAM if you get the oem number for the part? Anything you'd avoid getting off eBay due to fraud or bad service?


Sent from my iPhone using Tapatalk

 

[Spearfoot]

I think I'll go with the best of that list for under $130 a piece.

So you're going with the Intel S3700 SSD? Those are what I use.

Thanks for the price hunting help! Do you trust eBay sellers with RAM if you get the oem number for the part? Anything you'd avoid getting off eBay due to fraud or bad service?

eBay can be a crapp shoot... I've had mostly good luck, but I only purchase from US sellers with > 200 reviews - the more the better - and no or very few negative reviews. You can pick up on who the fly-by-night outfits are if you spend a little time researching. And I kinda keep an eye on items of interest using watch lists over time, to try and ascertain trends in price and availability.

I do buy RAM, subject to the conditions above. Most of the RAM I've bought has been older stuff for family member's PCs. Bought 4 x 2GB DIMMs for this purpose last week, and one of the sticks was bad. The seller promptly refunded my money.

Ya rolls yer dice and ya takes yer chances! :)

 

[ajschot]

I have a question related to ESXi and FreeNAS.
I am thinking about using ESXi for VM's like Windows, Ubuntu, OS X and FreeNAS. I use 64Gb Reg ECC RAM with a Xeon 2630v4 but when making a VM it is also putting RAM virtual, will i still have the benefit of ECC then in ZFS pools?
Is there a way of testing the working of ECC in FN10/FreeBSD?

 

[Spearfoot]

I have a question related to ESXi and FreeNAS.
I am thinking about using ESXi for VM's like Windows, Ubuntu, OS X and FreeNAS. I use 64Gb Reg ECC RAM with a Xeon 2630v4 but when making a VM it is also putting RAM virtual, will i still have the benefit of ECC then in ZFS pools?
Is there a way of testing the working of ECC in FN10/FreeBSD?

ECC is inherent in the basic design of the RAM module; it doesn't depend on the operating system. It does depend on the motherboard and CPU; both must be designed to support ECC memory.

 

[ajschot]

Thanks for the confirmation... Ok so it does not matter if i run ESXi in that fact ... i'm going to make lists about pre's and cons

 

[Spearfoot]

Thanks for the confirmation... Ok so it does not matter if i run ESXi in that fact ... i'm going to make lists about pre's and cons

That's correct - it doesn't matter. You always enjoy the benefit of ECC RAM on systems equipped with it.