bhyve VMs autostart conflicts with iocage jail bridge configuration

[zizzithefox]

I am posting this before sending a bug report because I would like to know if somebody else has recently had this problem. I searched the forum but found nothing recent, so.

Currently running TrueNAS Core 12-U3.1 but tested same behaviour with U4.1 here. Situation is a bunch of iocage jails (7) configured with these settings:

• VNET on
• Autoconfigure IPv6
• vnet_default_interface : igb2:casa
• IPv4 interface: vnet0
• IPv4 Address: set manually for all the jails to some LAN address
• network/interfaces: vnet0:bridge0
• IPv6: autoconfigure
• Autostart: on

Then I also have 3 bhyve virtual machines.

If I set autostart to OFF for the virtual machines, everything is fine, the jails get their interface on bridge0, the IP address I want, networking inside the jails works and they are reachable. I can then start the virtual machines, which get their own interfaces on bridge0 and everything works fine.

If I set autostart to ON for any of the virtual machines, the VMs themselves are always working fine but once every two reboots my jails are unreachable on the network because bridge0 does not work anymore and VMs are on another bridge1 that is created automatically (while the jails are on bridge0 because it's in their settings). I have to switch the jails manually to use either bridge1. If I restart, everything is still on bridge0 and jails are working and so on.

To clarify, suppose I leave the configuration for the jails to bridge0 and it is working now because I have VM autostart off and I started them manually. Now I turn autostart on for a VM.

Reboot - jails not working (but work if I set bridge1 on jails)

Reboot - jails working (on bridge0)

Reboot - jails not working (but work if I set bridge1 on jails)

Reboot - working (on bridge0)

...

As a workaround I am currently using an init script to automatically start the virtual machines… but it is odd. It started probably around 12 U2 or maybe with U3 but I am not sure since I don't restart the server that much so I didn't detect the problem before U3.

I tried to put an RC tunable with iohyve_flags to (net=igb2 kmod=1) as stated in a previous (older) post, but to no avail.

 

[Patrick M. Hausen]

You need to statically create your bridge interface in the network settings, then

• move all ip address configuration from the physical to the bridge
• set vnet_default_interface to "none" for your jails
• assign bridge0 to the jails and the VMs

The best way to get there if some downtime is possible

• set autostart to off for all jails and VMs
• reboot, so you won't have an automatically created bridge interface
• create a new bridge interface with your physical as the only member
• remove IP (v4 and v6) configuration from physical
• assign IP configuration to the new bridge
• do the test, then save dance
• reboot just to make sure
• then fix and start your jails and VMs individually