SOLVED FreeBSD 13.2 as bhyve guest - reboot not working

[Patrick M. Hausen]

Hi all,

I just installed an OPNsense in bhyve with two network interfaces passed through. This part in general worked suprisingly easy and well given the heat that bhyve frequently receives.

I had two issues, one I could solve myself:

1. shutdown -p now or shutdown via OPNsense UI or "STOP" in the TrueNAS UI would shut down the virtual machine to the point that all filesystems were unmounted cleanly, but the machine would not "power off. It would hang with the last message being "ix1 changed state to DOWN".
   I could fix that one with hw.efi.poweroff=0 inside the VM's OS.
2. Unfortunately that does not fix things for shutdown -r now, not even for reboot. Like above the system shuts down but then the final reset does not happen and the VM stays in that state until I click "POWER OFF" in the TrueNAS UI and then "START".
   I tried each possible combination of values for hw.acpi.handle_reboot and hw.acpi.disable_on_reboot without anything changing.

Any hints welcome.

Kind regards,
Patrick

 

[Patrick M. Hausen]

Switched from PCIe passthrough to bridged networking (in TrueNAS) and VirtIO (in OPNsense). With TrueNAS and the OPNsense VM sharing all 4 cores I can reach near gigabit throughput. No tunables necessary.

So using PCIe passthrough in bhyve can still have undesired side effects. Pity.

 

[Patrick M. Hausen]

I can confirm that a FreeBSD 13.2 guest works as expected in that regard with TrueNAS-13.1-MASTER-202310040714-bf3192b.

@morganL
Woohoo! Bhyve improvements incoming!

 

[Patrick M. Hausen]

End result:

Code:

                         TrueNAS CORE                
        ┌────────────────────────────────────────────┐
        │                                            │
        │                           OPNsense VM      │
        │       ┌ ─ ─ ─ ─ ─ ┐   ┌──────────────────┐ │
        │    ┌───────────┐      │                  │ │
        │ ┌──┴────────┐  │  │   │  LAN        WAN  │ │
        │ │           │  │      │┌─────┐    ┌─────┐│ │
        │ │ VMs/jails │  │  │   ││ ix0 │    │ ix1 ││ │
        │ │           │  ├ ─    │└─────┘    └─────┘│ │
        │ │           ├──┘      │   ▲          ▲   │ │
        │ └────────┬──┘         └───┼──────────┼───┘ │
        │          │                │          │     │
        │          │                │   PCIe   │     │
        │          │                │   pass   │     │
        │ ┌────────┴─────────┐      │   thru   │     │
        │ │                  │      │          │     │
        │ │     bridge0      │      │          │     │
┌────┐  │ │┌─────┐    ┌─────┐│   ┌──┴──┐    ┌──┴──┐  │
│IPMI├──┼─┼┤ ix0 │    │ ix1 ││   │ ix2 │    │ ix3 │  │
└────┘  │ │└──┬──┘    └──┬──┘│   └──┬──┘    └──┬──┘  │
        │ └───┼──────────┼───┘      │          │     │
        └─────┼──────────┼──────────┼──────────┼─────┘
              │          │          │          │     
              ▼          └──────────┘          ▼     
                                                     
          to laptop                        to uplink 

With this mainboard in this case.

With very low profile (VLP) memory modules I was able to cram 2 SATA-DOMs and 2 SATA SSDs, 1 M.2, 1 2.5" in there. With three Noctua coolers it's whisper quiet, not even "closet in the living room" but right on your desk. Not noticable. CPU and drive temperatures around 45 Celsius.

Complete lab to lug around to conferences, workshops, bar camps, ...

 

[victort]

End result:

Code:

                         TrueNAS CORE               
        ┌────────────────────────────────────────────┐
        │                                            │
        │                           OPNsense VM      │
        │       ┌ ─ ─ ─ ─ ─ ┐   ┌──────────────────┐ │
        │    ┌───────────┐      │                  │ │
        │ ┌──┴────────┐  │  │   │  LAN        WAN  │ │
        │ │           │  │      │┌─────┐    ┌─────┐│ │
        │ │ VMs/jails │  │  │   ││ ix0 │    │ ix1 ││ │
        │ │           │  ├ ─    │└─────┘    └─────┘│ │
        │ │           ├──┘      │   ▲          ▲   │ │
        │ └────────┬──┘         └───┼──────────┼───┘ │
        │          │                │          │     │
        │          │                │   PCIe   │     │
        │          │                │   pass   │     │
        │ ┌────────┴─────────┐      │   thru   │     │
        │ │                  │      │          │     │
        │ │     bridge0      │      │          │     │
┌────┐  │ │┌─────┐    ┌─────┐│   ┌──┴──┐    ┌──┴──┐  │
│IPMI├──┼─┼┤ ix0 │    │ ix1 ││   │ ix2 │    │ ix3 │  │
└────┘  │ │└──┬──┘    └──┬──┘│   └──┬──┘    └──┬──┘  │
        │ └───┼──────────┼───┘      │          │     │
        └─────┼──────────┼──────────┼──────────┼─────┘
              │          │          │          │    
              ▼          └──────────┘          ▼    
                                                    
          to laptop                        to uplink

With this mainboard in this case.

With very low profile (VLP) memory modules I was able to cram 2 SATA-DOMs and 2 SATA SSDs, 1 M.2, 1 2.5" in there. With three Noctua coolers it's whisper quiet, not even "closet in the living room" but right on your desk. Not noticable. CPU and drive temperatures around 45 Celsius.

Complete lab to lug around to conferences, workshops, bar camps, ...

How do you get uplink? Cellular?

 

[Patrick M. Hausen]

Whatever is provided at the event? Bring a small AP to bridge to the conference WiFi, for example. Or ask for a wired connection.

 

[oliver.epper]

I have the same problem (neither shutdown nor reboot working) with a FreeBSD 14 guest. Can anyone point me to what made it work with FreeBSD 13.2? Or even better tell me how to make it work with FreeBSD 14?

 

[Patrick M. Hausen]

I have the same problem (neither shutdown nor reboot working) with a FreeBSD 14 guest. Can anyone point me to what made it work with FreeBSD 13.2? Or even better tell me how to make it work with FreeBSD 14?

Nothing you can do now. My tests showed that it will probably work just fine in the next release of TrueNAS CORE due in April if I am not mistaken.