SOLVED Can I run VMware on Truenas (nested)

[derlinus]

Hi Community,

I was wondering if I can run ESXi (nested) as a virtual machine on freenas.

Why would I want to do it this way and not the other way around you might wonder.

I would like to run freenas with as efficient as possible with as much performance as possible when necessary (scrub tasks, RaidZ compression, freebasd drivers, etc...)
Having truenas control the hardware (wtr. to CPU and power states) directly it probably offers the best control over power states whenever it needs it or reduce when ther's nothing to do. The ESXi actually would then leverage the performance that's "left over", not to interfere too much with the primary job of the NAS - being a beefy file storage.

Instead I would like to use Truenas to pass through a two or three of the NICs in the system to VMware directly so that I can use them as dedicated NICs for VMware.
This way I could also use Truenas to create snapshots of the VMware system...
But mainly I am after full hardware access for truenas to have no compromise efficiency that can be achieved running native on hardware without a hypervisor in between

I know I may not have been the first one to ask that question but would be super interested in anyone's input.

Thanks Community!
Linus B.

 

[Samuel Tai]

Running a nested hypervisor is very unlikely to be possible, as the TrueNAS hypervisor is the upstream FreeBSD bhyve hypervisor, which doesn't yet expose VT-x and SVM to guests, according to https://forums.freebsd.org/threads/about-nested-virtualization-for-bhyve.78949/.

 

[Patrick M. Hausen]

You can run bhyve in ESXi but not vice versa.

 

[derlinus]

Thanks guys,
this is a bit discouraging but I am also a bit confused cause there is conflicting info:
@Samuel: The link you found mentions bhyve does not support VT-x as a final verdict, closing the thread.
But if I didn't misunderstand the text here bhyve does support and even require VT-x: https://wiki.freebsd.org/bhyve/pci_passthru
Also here: https://wiki.freebsd.org/bhyve#Q:_Does_bhyve_support_VT-d_PCI_device_pass-through.3F

Does that mean bhyve needs vt-x as a host, but can't provide it to guests ?
Need to get my head around this, but worst case i'll go as far as testing it.

Thanks again for your help!
Linus B.

 

[Samuel Tai]

Does that mean bhyve needs vt-x as a host, but can't provide it to guests ?

Precisely, which is what I posted above.

 

[derlinus]

Gotcha ! Thanks Samuel, great to know !

 

[derlinus]

FIX: Run Truenas Scale, then install VMware in KVM

Detail:

Quick update on the subject: It works but only on the Linux based version - Truenas Scale.

I am now running TNas on a Supermicro A2SDi-16C-HLN4F, I pass through 3 of the 4 on board NICs to VMware 7.03.
(VMware 7.x is very picky and MUST have a supported NIC - For security I didn't want to modify the VMware installer or use community drivers)
For the NAS itself I use a dedicated 10G NIC, had to be a PCIe 3.0 1x
Performance of the NAS is great and I get about 400 to 500Mbyte/s up and down with my fastest 5G client.

By using VMware I have the ease of Network configuration - I use to run my PFsense as VM.
I can also run OVAs for the ease of use to test stuff and it's isolated properly from my precious data.

Parts List:
Mainboard A2SDi-16C-HLN4F.
RAM: Samsung 32G, M393A1G40DB0-CPB / DDR4 Synchronous Registered (Buffered) 2133 MHz
OS drive: Intenso 3835440 250GB M.2
10G NIC: Qnap QXG-10G1T - Single-Port, 5-Speed 10 GbE (NBASE-T - RJ45) -chip: AQC107 NBase-T/IEEE
Storage SDDs: 8 x SAN Disk SDSSDH3-2T00-G25
Sata Power Splitting y cable: ean 4250580122333
CPU FAN: NF-A6x25 (custom mounted)
Case: Fractal Design, Node 304 (slightly modded to fit the 8 SSDs)
USB 3 Type A Male to 20pin header female - Thank you SuperMicro :-( - for USB3 support in the pc case, ASIN ‏ : ‎ B07X5522VQ)
PSU: Be Quiet, Pure Power, BQT L7-350W

Switch:
MikroTik CRS305-1G-4S+IN

Thanks Community for your excellent help and thanks IX Systems for the awesome product!