Working VMs Stopped Working

[Paul Crook]

I had 2 bhyve Ubuntu VMs working, was able to VNC to them and see them on the network. At some point in the past couple of months (I don't know exactly when), they stopped being accessible. The VM screen in the GUI says they're running, but I can't connect to them via serial, VNC nor the network. I upgraded to 11.2 Stable sometime after they were working, but I don't know if the upgrade was related to the failure. When I start one, I see this in middlewared.log:

Code:

[2019/02/01 01:52:51] (INFO) VMService.__set_guest_vmemory():901 - ===> Setting ARC FROM: 124148584448 TO: 119853617152
[2019/02/01 01:52:51] (DEBUG) VMService.run():179 - ====> NIC_ATTACH: igb0
[2019/02/01 01:52:51] (DEBUG) VMService.run():264 - Starting bhyve: bhyve -A -H -w -c 4 -m 4096 -s 0:0,hostbridge -s 31,lpc -l com1,/dev/nmdm6A -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -s 4,e1000,tap2,mac=00:a0:98:62:e3:54 -s 29,fbuf,vncserver,tcp=10.1.11.18:6330,w=1280,h=1024,,wait -s 30,xhci,tablet -s 3:0,ahci,hd:/dev/zvol/Kryz/pcrook-web-services-gh2rs 6_pcrook web services
[2019/02/01 01:52:51] (DEBUG) VMService.run():277 - ==> Start WEBVNC at port 6230 with pid number 43355
[2019/02/01 01:52:51] (DEBUG) VMService.run():286 - pcrook web services: 31/01/2019 18:52:51 Listening for VNC connections on TCP port 6330
[2019/02/01 01:52:51] (DEBUG) VMService.run():286 - pcrook web services: 31/01/2019 18:52:51 Listening for VNC connections on TCP6 port 6330
[2019/02/01 01:52:51] (WARNING) middlewared.devd_listen():93 - Failed to parse devd message: !system=DEVFS subsystem=CDEV type=CREATE cdev=vmm/6_pcrook web services

When I stop it, I see:

Code:

[2019/02/01 01:58:50] (DEBUG) VMService.stop():426 - ===> Soft Stop VM: pcrook web services ID: 6 BHYVE_CODE: None
[2019/02/01 01:58:50] (INFO) VMService.run():311 - ===> Error VM: pcrook web services ID: 6 BHYVE_CODE: -15
[2019/02/01 01:58:50] (ERROR) VMService.running():438 - ===> VMM pcrook web services is running without bhyve process.
[2019/02/01 01:58:50] (DEBUG) VMService.__teardown_guest_vmemory():338 - ===> Give back guest memory to ARC: 4294967296
[2019/02/01 01:58:50] (WARNING) VMService.destroy_vm():318 - ===> Destroying VM: pcrook web services ID: 6 BHYVE_CODE: -15
[2019/02/01 01:58:50] (WARNING) middlewared.devd_listen():93 - Failed to parse devd message: !system=DEVFS subsystem=CDEV type=DESTROY cdev=vmm.io/6_pcrook web services.framebuffer

[2019/02/01 01:58:50] (DEBUG) VMService.kill_bhyve_web():405 - ==> Killing WEBVNC: 43355
[2019/02/01 01:58:50] (ERROR) asyncio.default_exception_handler():1266 - Task exception was never retrieved
future: <Task finished coro=<VMSupervisor.run() done, defined at /usr/local/lib/python3.6/site-packages/middlewared/plugins/vm.py:107> exception=CallError('VM pcrook web services failed to start: 31/01/2019 18:52:51 Listening for VNC connections on TCP port 6330\n31/01/2019 18:52:51 Listening for VNC connections on TCP6 port 6330',)>

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/vm.py", line 315, in run
    raise CallError(f'VM {self.vm["name"]} failed to start: {output}')
middlewared.service_exception.CallError: [EFAULT] VM pcrook web services failed to start: 31/01/2019 18:52:51 Listening for VNC connections on TCP port 6330
31/01/2019 18:52:51 Listening for VNC connections on TCP6 port 6330
[2019/02/01 01:58:50] (WARNING) middlewared.devd_listen():93 - Failed to parse devd message: !system=DEVFS subsystem=CDEV type=DESTROY cdev=vmm/6_pcrook web services

I tried to create a new Ubuntu guest VM but can't VNC to it to interact with the installation process. In that case, I see:

Code:

[2019/02/01 02:10:41] (INFO) VMService.__set_guest_vmemory():901 - ===> Setting ARC FROM: 128443551744 TO: 124148584448
[2019/02/01 02:10:41] (DEBUG) VMService.run():179 - ====> NIC_ATTACH: igb0
[2019/02/01 02:10:41] (DEBUG) VMService.run():264 - Starting bhyve: bhyve -A -H -w -c 4 -m 4096 -s 0:0,hostbridge -s 31,lpc -l com1,/dev/nmdm7A -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -s 4,virtio-net,tap2,mac=00:a0:98:78:7e:2b -s 29,fbuf,vncserver,tcp=10.1.11.18:6257,w=1024,h=768,,wait -s 30,xhci,tablet -s 3:0,ahci,cd:/mnt/Kryz/OS Installation ISO Files/ubuntu-18.04-server-amd64.iso,hd:/dev/zvol/Kryz/testvm-lpg9fh 7_testvm
[2019/02/01 02:10:41] (DEBUG) VMService.run():277 - ==> Start WEBVNC at port 6157 with pid number 50300
[2019/02/01 02:10:41] (DEBUG) VMService.run():286 - testvm: 31/01/2019 19:10:41 Listening for VNC connections on TCP port 6257
[2019/02/01 02:10:41] (DEBUG) VMService.run():286 - testvm: 31/01/2019 19:10:41 Listening for VNC connections on TCP6 port 6257

I have no idea where to go from here to troubleshoot this problem. Can anyone suggest some ways forward?

Thanks in advance,

Paul Crook
OS Version: FreeNAS-11.2-RC2
CPU: Intel(R) Xeon(R) CPU E5-2650L v4 @ 1.70GHz (28 cores)
RAM: 128 GiB

 

[dlavigne]

Were you able to make any headway with this?