VM hangs if rebooted from guest with a passthru device.

[troonas]

So I have two VMs: one for running `docker` containers and another one acting as my router (OpenWRT). The router VM has a dual gigabit Intel PCIe NIC passed to it. I did this through the tunables interface (i.e. `pptdevs 5/0/0 5/0/1` and `vmm_load YES`).

If I reboot the `docker` VM from within the VM (i.e. via SSH), it reboots fine.

If I do the same from the router VM though, it seems to just hang once it shuts down. TrueNAS thinks it's running in the UI, but the VNC console cannot connect. If I stop the VM in the TrueNAS UI, it will immediately stop. Starting it again works no problem.

Anyone know how to fix this? Normally I wouldn't care very much, but the router VM is kinda critical and fixing issues with it not running involves me needing to physically go over to the computer, heh

 

[Patrick M. Hausen]

Does OpenWRT support a serial console? If yes, activate it. You can see the serial device in the VM UI and attach to it with

Code:

cu -l /dev/<console-device>

in a root shell via SSH.

Then check where it hangs and try to find out why the Linux guest will not completely reset the VM.

 

[troonas]

Thanks for the response! The serial console in the UI conveniently works right out of the box in OpenWRT without having to change any configuration in the VM. I tried rebooting via the serial console and this is the last thing it reports before freezing:

Code:

[ 1977.343758] reboot: Restarting system
[ 1977.345042] reboot: machine restart
[ 1977.347901] ACPI MEMORY or I/O RESET_REG.

So it seems like the guest OS is essentially telling the hardware to reboot... but somehow TrueNAS isn't seeing that and everything is stuck in a limbo state.

Any other ideas here?

 

[Patrick M. Hausen]

Google combinations like "bhyve", "Linux guest", "ACPI MEMORY or I/O RESET_REG" ...

I'll take it to the upcoming bhyve production users meeting but cannot promise anything. I am not actively involved in development. My suspicion is that there might be a setting inside of Linux controlling how exactly the reset is performed, that will work.

 

[troonas]

Hmm, I found some advice to add `acpi=off` to the boot config, but that caused the VM to hang on boot... so hard that only rebooting the host fixed it.

A workaround I've been considering is just not passing the ethernet card into the OpenWRT VM and instead bridging the ethernet ports in (i.e. one for WAN and one for LAN). The LAN one actually carries a few different tagged VLANs and I was hoping to create untagged VLAN interfaces from that bridge in TrueNAS for my jails and docker VM. The TrueNAS interface, however, doesn't allow creating VLAN interfaces from a bridge... is that something that's even possible with FreeBSD?

 

[Patrick M. Hausen]

Any interface that is a member of a bridge cannot carry tagged frames. What works in FreeBSD is to create one bridge per VLAN to assign to jails or VMs.

 

[troonas]

Okay, so now what I've tried is the following:

I have two physical ethernet interfaces called igb0 and igb1 and I have added them as NICs through the TrueNAS UI to my OpenWRT router VM.

igb0 is connected to my modem and works; OpenWRT gets WAN IPv4/6 addresses. OpenWRT is exposing multiple VLANs over igb1. The physical interface corresponding to igb1 is connected to a managed switch and all hosts connected via that are working as expected. So far so good.

On TrueNAS, I have configured three VLANs whose parent interface is igb1: vlan10, vlan20, and vlan30. For each vlanX I have configured a corresponding bridgeX. The IP of the TrueNAS host is configured on bridge20. Those bridges are connected to various jails/VMs.

Now the issue is, everything connected to those bridges is not reachable from OpenWRT and vice-versa. Have I cheated in some way here by having igb1 act as a VLAN trunk? I can see that under the hood, TrueNAS has configured a "secret" bridge0 and the OpenWRT VM's igb1 connection is really a vnet0 -> bridge0 -> igb1 connection.

I'm confused though... doesn't that mean bridge0 is carrying tagged frames from vnet0 to igb1, which isn't supposed to be possible?