Virtualize it or let it Virtualize? That is the question!

[Shockwaver]

Hello everyone!
As a total newbie (both in this forum and with FreeNAS) I will right away shoot my question.
Please consider this generic scenario (low-ended)

HW
CPU: Intel Xeon E3-1285 v6
MB: Any w/ Intel C236
RAM: 64GB DDR4 ECC
OS(s) Disk: 512GB M.2 NVMe - NOT in NAS pool
Storage Disks: up to 4x 3.5'' and up to 3x 2.5'' (disks may be same if same form factor) - in NAS pool
NICs: variable number for a maximum of 6x GBE IFs

Target OSs (Virtualized or not, all on the NVMe)
1x FreeNAS
1x GW/FW (like OPNsense)
1x Win10 Video Survellance Server (8-16GB RAM)
1x Win10 GP (8-16GB RAM)
1x WinServer GP (8GB RAM)
1x SVN

Let me firstly apologize for the extensive usage of microsoft platforms (I do have apps running on Win only and I'm too lame to go under Wine, again)
I have successfully, for some time, ran all the above with Ubuntu Server as hypervisor (and SVN) host and virtualizing all the rest with KVM.
Then finally came the time when FreeNAS didn't like something (like the NON-ECC ram I was too stupid not to mount in my previous configuration) and I lost my pools.

I was anyway happy with my (poor) home server solution and I'm willing to get another go, with ECC RAM this time around :D.

... And this time around, I spotted in the docs the section about VMs and started to wonder:
Given the above hypothesis (of HW and OSs configurations), what could I do better?
Unleash the FreeNAS directly on the metal and virtualize the rest with it, or do like I did using another (unix, not getting VMWare) host and virtualize the rest with KVM

Or, if you wish:

1. Who is doing a better job in virtualizing: bhyve or kvm?
2. Who is doing a better job in passing-through (OPNsense will need 2 dedicated IFs and operate them at will to work properly - bridged virtIO seemed to be ok): bhyve or kvm?
3. If kvm is outperforming bhyve, how big is the tradeoff I will have to face for not having FreeNAS running on the metal but virtualizing it with kvm?

Thanks for the attention and for any help you will be willing to provide me with,
and I apologize ahead of time if I posted in the wrong section and/or with a wrong format

Cheers

 

[m0nkey_]

Who is doing a better job in virtualizing: bhyve or kvm?

VMware ESXI

Who is doing a better job in passing-through

See above ;)

FreeNAS is fully supported when virtualized under ESXI. Just make sure to passthrough the HBA to the FreeNAS VM. Do not use RDM (Raw device mapping) as this can result in data loss.

 

[Shockwaver]

Alright!
So in this configuration I will have to forget about kvm and learn how to use VMware ESXI (which I know nothing about ATM).
I will soon go through all the guides and docs, but as a primer just to get a glance of what we are talking about...
Is it a HV that is running without an host, on the bare-metal, or did I get the first lines of the pages wrong?

 

[m0nkey_]

Is it a HV that is running without an host, on the bare-metal, or did I get the first lines of the pages wrong?

Yes, ESXI is a bare metal hypervisor.

 

[Shockwaver]

Nice.
I’ve read a few things and I’d be interested in the free version because the 2 limitation it has are none of my concern:

1. vCenter cannot manage free ESXi hosts because their APIs are RO. Fine by me since I will have only 1 host and I’ll manage it with the vSphere Client
2. Script automation N/A. But I’m not planning to use any

Would you then agree I could go with the free vSphere Hypervisor w/ pretty much peace of mind?

 

[Inxsible]

This is something that I have been exploring as well as part of my upgrade. I am planning on setting up a new server using some components that I already have and buying the rest. My current NAS box will then be converted into a backup NAS. My current backup is a bunch of scripts that back things up to 3 different machines (2 laptops & 1 desktop).

Initially I was simply going to install FreeNAS on bare metal in the new 3U/4U chassis, but now I am thinking I could put a hypervisor (likely ESXi) on bare metal and then virtualize FreeNAS and also create 3 separate Archlinux desktop VMs for each member of my family which could be accessed by individual laptops/chromebooks.

 

[m0nkey_]

Would you then agree I could go with the free vSphere Hypervisor w/ pretty much peace of mind?

If you want a super stable virtualization platform, yes. You could also consider Citrix XenServer (FreeNAS works, but not fully supported) or Hyper-V. All of which are free bare metal platforms.

 

[Shockwaver]

Deal! You bought me in...

 

[Zredwire]

Nice.
I’ve read a few things and I’d be interested in the free version because the 2 limitation it has are none of my concern:

1. vCenter cannot manage free ESXi hosts because their APIs are RO. Fine by me since I will have only 1 host and I’ll manage it with the vSphere Client
2. Script automation N/A. But I’m not planning to use any

Would you then agree I could go with the free vSphere Hypervisor w/ pretty much peace of mind?

I use the ESXi free version and have my FreeNAS virtualized in it. A couple more things that the free ESXi cannot do are:

1) You cannot backup VMs directly through ESXi. The AP is locked. This also means that FreeNAS cannot coordinate with ESXi when it does snapshots. You can certainly do FreeNAS snapshots of ESXi volumes but they will be crash consistent. Backups of the VMs have to be done with agents in each VM.

2) You cannot assign more than 8 vCPU per VM.

3) Limited to 2 Sockets on the ESXi host (but unlimited cores). - Actually not unlimited but has the same limits as the paid versions.

 

[Shockwaver]

I can surely come to terms with those points. Further more my FreeNAS won’t even be aware of the other VMs’ virtual local disks, because they will all be sitting together on the NVMe which is not passed through...
Thanks nonetheless!
However ... what are your impressions about your configuration? Stable? Sweet?

 

[Shockwaver]

VMware ESXI

See above ;)

FreeNAS is fully supported when virtualized under ESXI. Just make sure to passthrough the HBA to the FreeNAS VM. Do not use RDM (Raw device mapping) as this can result in data loss.

When I'm trying to passthrough th HBA passing-through the SATA controller, the pci device stays in status "active / needs reboot" even after rebooting and i csnnot add it to the vm just like it is not enabled the passthrough... any idea why?

 

[m0nkey_]

Make sure VT-d is enabled in the BIOS.

 

[Shockwaver]

I will check right away... But I am anyway able to passthrough NICs and USB... Shouldn't it be already enabled then?

 

[Shockwaver]

Yes it was enabled... Tomorrow I will try to use different ports for the disks, maybe they are conflicting...

 

[Chris Moore]

Storage Disks: up to 4x 3.5'' and up to 3x 2.5'' (disks may be same if same form factor) - in NAS pool

Purely out of my own curiosity, why different size disks?

You might want to take a look at this build by @Stux
Build Report: Node 304 + X10SDV-TLN4F [ESXi/FreeNAS AIO]
https://forums.freenas.org/index.ph...node-304-x10sdv-tln4f-esxi-freenas-aio.57116/

and some others about virtualizing FreeNAS and the various details associated with that...
"Absolutely must virtualize FreeNAS!" ... a guide to not completely losing your data.
https://forums.freenas.org/index.ph...ide-to-not-completely-losing-your-data.12714/

Testing the benefits of SLOG
https://forums.freenas.org/index.php?threads/testing-the-benefits-of-slog-using-a-ram-disk.56561

 

[Shockwaver]

Purely out of my own curiosity, why different size disks?

You might want to take a look at this build by @Stux
Build Report: Node 304 + X10SDV-TLN4F [ESXi/FreeNAS AIO]
https://forums.freenas.org/index.ph...node-304-x10sdv-tln4f-esxi-freenas-aio.57116/

and some others about virtualizing FreeNAS and the various details associated with that...
"Absolutely must virtualize FreeNAS!" ... a guide to not completely losing your data.
https://forums.freenas.org/index.ph...ide-to-not-completely-losing-your-data.12714/

Testing the benefits of SLOG
https://forums.freenas.org/index.php?threads/testing-the-benefits-of-slog-using-a-ram-disk.56561

Thanks for the info, I already gave them a look but I'll make sure to read them thoroughly, before closing my configuration!
It's just how my actual case is bayed. Tipically I'm going to pass to the FreeNAS only identical drives (or a JBODed version of them, if I feel crazy enough)
In case of need I could also change the case.

Make sure VT-d is enabled in the BIOS.

As I said before, Intel VT-d was already enabed. Anyhow disks weren't conflicting and I didn't understand why my 2 "ASM1062 Serial ATA Controller" are staying in "Active / Needs reboot" passthrough status even after rebooting (and I'd really like to understand it) but I do was able to set the passthrough of the other SATA controller "Sunrise Point-H AHCI Controller", after adding the proper line to

Code:

/etc/vmware/pass.map

since it was stated "Not capable" and add it to a VM. Next goes the testing.

Hence my question (VMware forums ar quite the... ghostly places...):
As far as my seeing and understanding goes there is a 1-to-1 releshionship beween vmhba# and a mass storage controller (but I could also be wrong) AND I can only passthrough an hba to a VM PCI-Device-wise, and that means that the guest VM will see all the drives managed by that controller not letting me choose if I want to pass only a subset of them. Am I correct? I hope not...

 

[Chris Moore]

As far as my seeing and understanding goes there is a 1-to-1 releshionship beween vmhba# and a mass storage controller (but I could also be wrong) AND I can only passthrough an hba to a VM PCI-Device-wise, and that means that the guest VM will see all the drives managed by that controller not letting me choose if I want to pass only a subset of them. Am I correct? I hope not...

You pass an entire HBA to the VM and the drives attached to that HBA are only available to that VM.

 

[Shockwaver]

That surely makes sense AND is annoying... for me.

 

[Shockwaver]

One last question (by now)
I see in (almost) all configurations around, external PCIe SATA controllers in addition to the on-board ones.
Is there a reason to go with them OTHER THAN having more SATA ports and having more flexibility in passing-through (as I just got to understand)?
I mean... If I don't feel the need to have more drives than the ones I can physically attach to the on-board ports and neither the need of more passthrough, can I just go with the embedded controllers?

 

[Chris Moore]

I mean... If I don't feel the need to have more drives than the ones I can physically attach to the on-board ports and neither the need of more pass-through, can I just go with the embedded controllers?

Some on-board controllers do not work well with pass-through to a VM. It depends on the specific hardware. If you are able to successfully pass the on-board controller to the VM, and you don't need any additional drives that are used by ESXi or another VM; then you don't need another controller.
In my system, I use the on-board controller to provide drives to ESXi and a SAS HBA is passed into the VM to provide drives to a virtual instance of FreeNAS.