best practise esxi hardware?

[John Doe]

hi,
i am considering to set up an esxi system with freenas but i am not sure about the pysical disc setup.
shall the esxi system have an own physical ssd or can it be combined with a larger ssd for the virtualized vms?

in case, esxi shall have a seperate ssd, shall the storage for the vms be connected directly via motherboard to the esxi, or is it better to have it via HBA connected to the freenas and than passed back to esxi via iscasi?

would be great to have the boot drive and place for vms redundand (mirrored raid)

or with other words, i have 2x 32gb ssd, 1x 512 gb ssd and 2x 1tb ssd, what would be the best setup?

 

[rvassar]

If you're looking to run FreeNAS as a VM, I believe the only way to get it to work correctly is to "pass through" the disks to the VM, so FreeNAS can access the hardware directly. In which case you'd probably want to pass storage back via iSCSI, but you need to worry about the chicken vs. egg problem. The FreeNAS VM needs to reside on a datastore that is available to ESX when FreeNAS is not running. Using the 32gb disks as a mirrored pool sounds right, but I'm not sure what the minimum requirements are for ESX + FreeNAS. That might be a bit tight.

 

[John Doe]

is it benefical to have the ssds mounted to freenas and pass it back to esxi, rather tan just connect the ssds to esxi?
pro cons?

 

[Spearfoot]

is it benefical to have the ssds mounted to freenas and pass it back to esxi, rather tan just connect the ssds to esxi?
pro cons?

You can't boot ESXi nor install the FreeNAS VM itself on storage provided by FreeNAS. So you need a local ESXi data store on which to install ESXi and the FreeNAS VM.

This local datastore cannot be a USB/flash/thumb drive.. Well, actually you can install ESXi to these, but it won't let you use them for a local datastore; meaning you would have to install an additional SSD/HDD. So why bother? It just makes sense to use an SSD for both ESXi and and a local datastore for the FreeNAS VM. The SSD doesn't have to be very large, but I don't think your 32GB SSDs will be big enough for this purpose. I use mirrored 80 or 120GB SSDs (see 'my systems' below for details').

You will need to pass-through an HBA to the FreeNAS VM. Then you can build your pools and configure NFS or iSCSI storage and make it available to ESXi for additional VMs.

There is a wealth of information about virtualizing FreeNAS-on-ESXi here on the forum. You'll have to use the search tool and dig it up. One of the best guides I used to get started several years ago is here:

https://b3n.org/freenas-9-3-on-vmware-esxi-6-0-guide/

I still use more-or-less the same setup, with some changes and enhancements (e.g., RAID-1 boot devices). I prefer and still use ESXi 6.0 because it supports the handy 'thick' vSphere client which VMware dropped in later releases, to much moaning and griping from the VMware community. And I run FreeNAS v11.1-U6 instead of the older version in the blog article. I recommend you avoid 11.2 for now, as there are many reports here on the forum of users having problems with it.

IMHO, once you get your All-In-One up and running, you'll find that you've made yourself a relatively trouble-free and far superior virtualization environment compared to that provided by FreeNAS, which is a great file-server, but not such a great hypervisor, as witnessed by the many virtualization problems users post about here on the forum.

Good luck!

 

[John Doe]

...

many thanks for your post, that really helped!

your IMHO is exactly the reason, why i would like to have esxi.

coming to your sytem, the boot for esxi and the FN vm is in a raid, which i would like to have as well. Is there a software raid available?

I can see a ZIL SLOG, how much benefit do you have with that? for your pool with the spinning rust and for the ssd pool for the esxi VMs?
so would you recommend it or ist it just nice to have?

I am asking because I would like to avoid putting more hardware into the server. power consumption is in germany a big topic and currently i ran my system with ~60w idle. 2x SLOG and one more adapter would probably ~90-110w in idle. so there should be more than just nice to have

 

[Spearfoot]

many thanks for your post, that really helped!

your IMHO is exactly the reason, why i would like to have esxi.

coming to your sytem, the boot for esxi and the FN vm is in a raid, which i would like to have as well. Is there a software raid available?

I can see a ZIL SLOG, how much benefit do you have with that? for your pool with the spinning rust and for the ssd pool for the esxi VMs?
so would you recommend it or ist it just nice to have?

I am asking because I would like to avoid putting more hardware into the server. power consumption is in germany a big topic and currently i ran my system with ~60w idle. 2x SLOG and one more adapter would probably ~90-110w in idle. so there should be more than just nice to have

I'm glad to help!

Yes, I boot from a RAID-1 array set up on a separate Dell H200 RAID adapter; a simple mirror of 2 SSDs. Honestly, you can get by without this kind of setup and simply use a single, reliable (i.e., Intel DC-quality) SSD. I just like the warm and fuzzy feeling I get from having a redundant boot environment. Note that you cannot use the built-in, often so-called 'fake' RAID provided by most motherboards; ESXi doesn't recognize these as they usually require add'l driver support from the installed OS.

I install ESXi, the FreeNAS VM, and a small OpenVPN Linux VM on the local 'boot' datastore; all of my other VMs (Plex, Windows, CentOS, Linux, etc.) I run from an NFS datastore provided by the FreeNAS VM.

You can also get by without a ZIL SLOG device; unless you push your VMs really hard, you may not notice much improvement in performance from having one installed. So it really just depends on your needs.

 

[John Doe]

I install ESXi, the FreeNAS VM, and a small OpenVPN Linux VM on the local 'boot' datastore; all of my other VMs (Plex, Windows, CentOS, Linux, etc.) I run from an NFS datastore provided by the FreeNAS VM.

ok, so that was basically my question above, what is the difference between install VMs on the ssd at the esxi boot vs pass it through FN back to esxi?
of course you have the FN functionality of snapshots and FNs software raid functionality but is there something else?

 

[Spearfoot]

ok, so that was basically my question above, what is the difference between install VMs on the ssd at the esxi boot vs pass it through FN back to esxi?
of course you have the FN functionality of snapshots and FNs software raid functionality but is there something else?

In a nutshell, you get all of the advantages of using a premium hypervisor (ESXi) coupled with all of the advantages of a premium file server (FreeNAS).

ESXi gives you fine control over your virtual machines, without the risk and aggravation many people seem to experience getting these to work on FreeNAS. It's a great hypervisor, but not a great file server.

FreeNAS gives you fine control over your data, with snapshots, replication, backup, hard drive monitoring and maintenance, NFS or iSCSI storage for VMs, SMB file-sharing, etc. It's a great file server, but not a great hypervisor.

YMMV, but for my purposes the FreeNAS-on-ESXi All-In-One is a developer/tinkerer's dream come true. ;)

 

[Scharbag]

Finally update my host to 768GB of RAM. Had to share somewhere. TrueNAS gets 256GB now. Total overkill. But overkill is underrated!!

And if you are interested in using ESXi and TrueNAS on the same host, I would suggest you get some surplus enterprise gear, from a site like UnixSurplus.com or Ebay. Cheapest way to get good quality stuff.