Running virtual machines in a virtualized TrueNAS host - Should this be possible, in theory?

[fsociety3765]

Hi all,

I run TrueNAS in a Proxmox virtual machine.

While I do not plan on utilizing TrueNAS for virtual machines, for reasons that are irrelevant, I needed to set up an Ubuntu Server VM directly on TrueNAS as I needed a Linux host to have direct access to the storage within TrueNAS.

I have Proxmox set up correctly to allow nested hardware-assisted virtualization with the TrueNAS guest CPU type set to "host". I ran through the wizard to create the virtual machine in TrueNAS. Linux, Ubuntu Server 20.04, 1 core, 2GB RAM, 1TB hard disk, DHCP for the network. It created the virtual machine. However, as soon as I tried to start it up, TrueNAS froze and became unresponsive. I waited a bit but it was clear it wasn't going to recover. I could no longer access the web UI, SSH, and it wouldn't respond to pings. In Proxmox, instead of the green play icon on the VM, I had a yellow warning icon with a message that said "internal error". I wasn't able to gracefully reboot or shutdown TrueNAS. I had to force stop it. Tried to boot up again and it ran through the boot sequence but shortly afterward it froze again and I had to force stop the VM again. I decided at that point it wasn't worth any further risk and I turned the CPU type back to the Proxmox default which TrueNAS will not allow virtual machines to run with. TrueNAS then booted with the nested VM in a stopped state and I was able to delete it.

All a bit scary... Haha.

Is there something I may have done wrong? While I fully understand that running TrueNAS in a VM is not highly recommended, as well as nested virtualization in general, should this have worked in theory?

Thanks,

FS

 

[Samuel Tai]

bhyve, which is the virtualization layer within TrueNAS Core, is still a relatively immature hypervisor, and hasn't been extensively tested in a nested virtualization environment. See this report from one of the bhyve developers: https://tips.graphica.com.au/freebsd-virtualisation-with-bhyve/.

In short, once bhyve matures enough, it should work, but no one knows how long that maturation process will take, given the very small installed base and developer pool for bhyve.

 

[Migsi]

I don't have clues about nested virtualization, but it might be worth to check if there are no other viable solutions.

I needed a Linux host to have direct access to the storage within TrueNAS.

I'm not sure what you mean by having "direct" access from within a VM. I just checked my VM's and there is, at least no obvious (except pcie passthrough) option to access disks directly. When adding a "disk" to the VM, it requires it to be a zvol somewhere on a pool, when adding a "raw file" you theoretically could pass through a whole dataset or even a pool, but I'm not sure what the implications are. As long as you only require to access a filesystem that is managed by truenas, you could share it via nfs and mount it in proxmox. That could provide what you need with minimal overhead while saving you from bothering with nested virtualization.

 

[boistordu]

I’m going to add my 2 cents here.
I’m running truenas in a vm which also virtualized another Ubuntu vm trough Bhyve.

it’s running smoothly.
The bare metal host is a fedora server 33.
the hypervisor is kvm-qemu.
the nested virtualization us running smoothly.
There are some efi error at the start of Ubuntu that I can see through vnc but nothing really stopping the virtualization from happening.

The only problem I get for the moment is to reach the truenas host from the Ubuntu vm. It’s working with the ping but not with other protocols wether it s ssh or nfs, it might be due to the setup or just an error I made maybe I don’t know.
But for the rest It didn’t crash or anything so I m pretty happy about the result for the moment.

 

[Syptec]

This works well with nested or un nest Hypers in VMWare and works as a 2nd party store to run, store and sync zfs snapshots. Actually really well.

 

[Whattteva]

I don't get it. Why would you need to nested virtualization when you're already running Proxmox? My TrueNAS CORE VM has nothing but NFS, SMB, and SSH services. It does make that NFS share available for other VM's on the Proxmox. It's setup to boot first with a 90 second delay before other VM's boot. There shouldn't be any need to do any nesting.