ESXi and FreeNAS Boot Devices- 3 USBs?

[aftereffectsmagic]

Hi there,

Still working on sourcing parts for my ESXi and FreeNAS (as a VM) build. I am loosely following Stux's AIO build guide (a great resource, to be sure)!

I came across some info saying that ESXi can be booted from a 32 GB Sandisk Cruzer Fit USB 3.0 drive (older reported issues with this seem to be resolved). So would it be absurd for me to go ahead and get three 32 GB Cruzer Fit USB drives, two for a mirrored FreeNAS install, and one to be used as the ESXi boot device? My X8DTU has eight USB 2.0 connectors.

I realize this may hinge on whether or not my datastore will be mostly static- if anyone has a baseline comparison chart or some insight I can use so I can better understand whether my ESXi datastore will be mostly static or not that would be great. My feeling is that it will be mostly static, from the info I've read.

Another option is to go for a 32 GB Intel Optane M.2 drive for the ESXi boot, keep two USBs for the FreeNAS boot, and source an adapter so it can be used with my PCIe riser card on my X8DTU motherboard. This would be the second least expensive option.

Thanks for your quick suggestions / tips! Clearly new to all of this, so am open to all of your recommendations :)

 

[Chris Moore]

I realize this may hinge on whether or not my datastore will be mostly static-

The ESXi boot device is not a datastore. It is only used to boot the hypervisor.

Another option is to go for a 32 GB Intel Optane M.2 drive for the ESXi boot

No point in doing that, but you might add that to use as a datastore while you boot from the USB device.

 

[aftereffectsmagic]

Thanks for popping in, @Chris Moore!

To recap / clarify what you're saying:
-Option A: Two USBs for FreeNAS, and one 16 GB or 32 GB Intel Optane M.2 for ESXi boot / datastore.
-Option B: Two USBs for FreeNAS, and a cheap SATA SSD for the ESXi boot / datastore.
-Option C: Three USBs for FreeNAS and ESXi boots, dedicated M.2 for ESXi datastore only.

That's the idea, yes?

The reason I ask is because due to the limitations of the X8 platform I have an older riser card (RSC-R2UU-3E8G) and I would need to get some sort of adapter for the M.2 because the riser card doesn't have support for M.2 drives. Any tried-and-true PCIe M.2 adapter recommendations for my particular situation? If this starts to add up a cheap SSD connected to one of the motherboard SATA ports is starting to look like a cheaper option vs retrofitting my old equipment for M.2 capability.

But as cyberjock so eloquently put it:

Using those adapters is, in my opinion, lame. You gain very little by moving from SATA controller to the PCIe bus. Sure, you gain a little latency (at least, in theory), but the reason to go with PCIe is for the larger bandwidth. You can't get that from an SSD that is still SATA 6Gb/sec. So the main advantage is basically gone.

Or is it more like what jgreco replied to cyberjock with:

Not lame at all. Quite frankly, when you're doing this stuff professionally, you're always fighting this insipid battle of insufficient space in the chassis, or, more often, imbalanced space. You might only need a slot or two of your PCIe capacity on a filer. Now look at a platform like the X10SRL with an E5-1650 in a SC216BE16 chassis. You put in boot on redundant SATA-DOM, a HBA to the expander and to the two rear facing bays. You're left with five PCIe slots empty and eight mainboard SATA ports available. So ... which is lame: do you ignore the potential to do more with this system, or do you proceed to fill four of those slots with triple-M.2 adapters and be able to cram a dozen more SSD storage devices into the filer at minimal additional cost?

Am I better off doing a small SSD for the ESXi boot / datastore connected directly to one of the SATA ports present on the motherboard to save costs since none of my SATA ports are in use, or should I invest in M.2 adapters?

Thanks!

 

[Chris Moore]

I am no expert on ESXi, but it was my understanding that the boot drive for ESXi could not be used as a datastore, similar to the way that the boot drive for FreeNAS is not allowed to be storage for the NAS. In both instances, use a USB drive or a small capacity SSD or SATA DOM for booting the OS and plan to have other storage that is not on the boot device. The datastore for ESXi can be any drive in the system except for the two boot drives already mentioned, so add a SSD or M.2 or whatever makes you happy to use for that. If you share the FreeNAS storage back to the ESXi by means of iSCSI, you can use your storage pool to house your virtualization, which is what @Stux did in his build, if I recall correctly. I also think that I recall he had some local storage for ESXi but I have not read through that thread recently.

 

[Stux]

If your ESXi boot drive is a proper drive, not a USB, then you can store data stores.

I boot my ESXi off an m2 drive which also holds the FreeNAS vmdk

 

[aftereffectsmagic]

I recently picked up an Intel Optane m2 drive (32 GB). Will this be sufficient to boot ESXi off and hold the FreeNAS vmdk? @Stux, your solution seems to be a higher capacity (250 GB), so not sure what I have is enough.

I also have a 32 GB USB drive that I can use to boot ESXi off if the above solution doesn't work. This is my backup solution.

Thanks for confirming!

 

[Spearfoot]

Am I better off doing a small SSD for the ESXi boot / datastore connected directly to one of the SATA ports present on the motherboard to save costs since none of my SATA ports are in use, or should I invest in M.2 adapters?

You can indeed use the ESXi boot device as a datastore, as long as it's not a USB drive. My 'All-In-Ones' boot ESXi 6.0 from a pair of mirrored SSDs connected to Dell PERC H200 HBA cards. I configure small datastores on these volumes, on which I install the FreeNAS VM, along with a minimal Linux VM I use as an OpenVPN server.

You can also just use a single SSD in the same way, if you don't care about redundancy.

See 'my systems' below for dirty details.

Good luck!

 

[Stux]

I recently picked up an Intel Optane m2 drive (32 GB). Will this be sufficient to boot ESXi off and hold the FreeNAS vmdk? @Stux, your solution seems to be a higher capacity (250 GB), so not sure what I have is enough.

I also have a 32 GB USB drive that I can use to boot ESXi off if the above solution doesn't work. This is my backup solution.

Thanks for confirming!

Yes it should be fine. Optane drives were not available when I built my system.

 

[aftereffectsmagic]

Hi everyone!

Since I created this thread, there have been a few changes / delays to getting my system up and running. Finally installing ESXi now and I attempted to boot off ESXi after having no problems with the install process, but the X8DTU-F BIOS (v.2.67) does not have UEFI capability or any toggle that allows UEFI/EFI and Legacy as far as I can see.

I did try the

Code:

>runweasel / formatwithmbr

command to see if that worked in the early stages of the ESXi install, but no go. Still unable to boot from the options. My options as presented by the BIOS are: SCSI Card / USB / Removable Drive / CD/DVD / Network.

I am trying to think of a solution that would allow me to boot ESXi properly (again, following @Stux's AIO build guide on page 2). The only solution I have thought of so far is what I suggested earlier, Option C: Three USBs for FreeNAS and ESXi boots, dedicated M.2 for ESXi datastore only.

Is this indeed the best solution for my situation? If so, how can I place the datastore on the M.2 Optane I have been using for ESXi, while placing the boot on an USB? I'm sure it's simple enough...

If anyone has a better workaround... I'm definitely game! Thanks ;)