VM Networking/Bridge Problem

[Ctiger256]

I just upgraded to 11.1-U3 and one of my VMs will not start. I have two network interfaces active in different vlans igb1 and igb2. All the VMs bridged to igb2 start fine, but the one bridged to igb1 will not. It worked fine before the upgrade, but not now. I have rebooted several times with no luck. If I switch it's NIC over to igb1 it boots fine.

Here's what middleware.log shows:

[2018/03/21 11:06:28] (DEBUG) VMService.run():155 - ====> NIC_ATTACH: igb1
[2018/03/21 11:06:28] (DEBUG) VMService.run():231 - Starting bhyve: bhyve -H -w -c 4 -m 16384 -s 0:0,hostbridge -s 31,lpc -l com1,/dev/nmdm9A -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -s 3,ahci-hd,/dev/zvol/magstorage/vms/windows10 -s 4,virtio-net,tap2,mac=00:a0:98:11:f7:66 9_windows10
[2018/03/21 11:06:30] (DEBUG) VMService.run():250 - windows10: Unhandled ps2 keyboard command 0x02

[2018/03/21 11:06:34] (DEBUG) VMService.run():250 - windows10: Unhandled ps2 keyboard command 0x02

[2018/03/21 11:07:03] (DEBUG) VMService.run():155 - ====> NIC_ATTACH: igb1
[2018/03/21 11:07:03] (DEBUG) VMService.run():231 - Starting bhyve: bhyve -H -w -c 4 -m 16384 -s 0:0,hostbridge -s 31,lpc -l com1,/dev/nmdm9A -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -s 3,ahci-hd,/dev/zvol/magstorage/vms/windows10 -s 4,virtio-net,tap3,mac=00:a0:98:11:f7:66 9_windows10
[2018/03/21 11:07:03] (DEBUG) VMService.run():250 - windows10: vm_reinit: Device busy

[2018/03/21 11:07:03] (INFO) VMService.run():265 - ===> Powered off VM: windows10 ID: 9 BHYVE_CODE: 1
[2018/03/21 11:07:03] (WARNING) VMService.destroy_vm():275 - ===> Destroying VM: windows10 ID: 9 BHYVE_CODE: 1
[2018/03/21 11:07:07] (DEBUG) VMService.run():250 - windows10: vm_run error -1, errno 25

[2018/03/21 11:07:07] (INFO) VMService.run():265 - ===> Powered off VM: windows10 ID: 9 BHYVE_CODE: 1
[2018/03/21 11:07:07] (WARNING) VMService.destroy_vm():275 - ===> Destroying VM: windows10 ID: 9 BHYVE_CODE: 1

The console also shows that bridge1 is created and each of the tap interfaces (tap2 ant tap3 respectively) tries to get created.

The igb1 interface itself is fine, i.e., I can ping to and from it.

Any thoughts or help greatly appreciated. Happy to post more info to help debug.

 

[dlavigne]

There's at least one open bug wrt VMs and VLANs and I know the dev is in the middle of reworking this code to be less flaky.

 

[dlavigne]

There's at least one open bug wrt VMs and VLANs and I know the dev is in the middle of reworking this code to be less flaky with VLANs.

 

[Ctiger256]

Will keep an eye, though in this case the VLANs are not created on the FreeNAS box. I just have two interfaces plugged into two untagged ports on different VLANs on my switch. So from the FreeNAS box's perspective each interface should be just sitting on a different subnet.