Running XCP-ng nested under KVM

[jimp]

I have seen many threads talking about running TrueNAS virtualized as a guest under hypervisors like XenServer/XCP-ng and Proxmox. However, I'm trying to do it the other way around.

I want TrueNAS to be on bare metal so I don't have to pass through my HBA and use iSCSI as if XCP-ng is talking to a separate system, and with XCP-ng as a guest inside TrueNAS, all the virtual disks will be backed by ZFS naturally. I'm trying to use the virtualization (KVM) offered by TrueNAS SCALE as well, but I'm also needing to migrate some existing XCP-ng VMs unchanged.

The setup:

Code:

TrueNAS SCALE
    XCP-ng 8.2 (free version of XenServer)
        Linux VMs (mostly CentOS 7 and Ubuntu 20), Windows VMs later (10 or newer)

I have successfully installed XCP-ng as a KVM guest when the CPU is set to passthrough (otherwise Windows guest support disables). So far, so good. Except now that I have transferred a VM to XCP-ng, the NIC for the guest VM never hears the DHCP broadcast for my router. The XCP-np host receives an IP address, but the guests under it timeout.

What do I need to change on the TrueNAS server to allow DHCP of my router to reach the guest VMs of the XCP-ng VM? I'm not very experienced with KVM, but I'm guessing there is something I need to enable or a bridge to setup or something.

 

[jimp]

Can't get virtual machine to work...

Hi all, I am trying to get an Ubuntu Server virtual machine running on SCALE. I have never used virtualization on TrueNAS before but I have a need to use it now. I have configured it all via the wizard etc and it appears to start up, but I can't access the display via VNC or SPICE. It doesn't...

www.truenas.com

@fsociety3765 If you don't use a bridge inside SCALE you'll get a macvtap connection with it's known limitations.

That didn't explain the limitations, but it did point me in the right direction. I wasn't using a bridge, so it must have been a macvtap connection. That worked for the VM receiving an IP, but once I switched to a bridge setup with the NIC I originally assigned to the VM (the main NIC of the system) and then assigned the bridge to the XCP-ng VM, the nested guest was able to receive an IP via DHCP.

It seems nested virtualization in TrueNAS SCALE might only have networking with a bridge at this time.

 

[probain]

Just wanted to mention that I'm doing the same thing.
Also, I noticed that the performance and behaviour with the VirtIO for the NIC (in truenas kvm). Delievered better results. Otherwise I had troubles with DHCP-renewals and stuff. Which was weird.

 

[jimp]

Nice, thank you for the feedback. I tried AHCI for disk and the e1000 NIC at first, but I switched to VirtIO on both and I haven't observed any issues in XCP-ng or the VMs running under it.

 

[probain]

Hmm, aparently the DHCP-renewal issues reapeered even with the VirtIO-drivers. Feels like there is something more underlying with either the device or the drivers. Since this affects XCP-guest i.e. XOA too.

Trying with static IP (still VirtIO) om the XCP-host now.

 

[probain]

Hmm, aparently the DHCP-renewal issues reapeered even with the VirtIO-drivers. Feels like there is something more underlying with either the device or the drivers. Since this affects XCP-guest i.e. XOA too.

Trying with static IP (still VirtIO) om the XCP-host now.

Even more problems. Basically xcp hangs in lots of interesting ways. Escpecially when trying to start a new guest VM other than XOA. This also shows up mostly after rebooting xcp once after initial install and setup.

Nesting hypervisors, is probably not recommended for a good reason . So I'm moving forward and leaving this for future endevours.

Just being curious. How is your experience so far? And what is your underlying hardware setup?

 

[jimp]

Well, I said it was working fine before, but I just realized it has actually crashed a few times. XCP-ng Center has logged some crash dumps. I'm not sure if it is an issue caused by KVM (maybe OOM?) or if the ACHI+e1000 setup would work better. When it is running, it works fine. But sometime later when I'm not looking it seems to crash/reboot and I don't yet know how to diagnose that.

My hardware is an ASUS B450 Strix mobo with 32gb Micron ECC ram, Ryzen 3950X, and ASUS 10g NIC. For storage I have a 4-port LSI SAS HBA plus Intel 20 port SAS expander. The only questionable component is the 10g NIC, which did not work well in TrueNAS Core and I suspect might not have all the driver support for solid virtualization support in Linux (just based on community sentiment against it).

Nothing I'm doing with XCP is really mission critical, but I also don't want it crashing every few days either. I'm going to try the non-VirtIO setup and look into how to convert/import these Xen VMs into TrueNAS/KVM.

 

[jimp]

It crashed again. I have yet to find a log of the crash on the KVM host or XCP. I haven't switched from VirtIO yet, but I just stopped using dynamic memory range in Xen since I read that feature is deprecated. Perhaps I'm running low on memory and it is crashing when it tries to scale up.