SOLVED FreeNAS 11 Bhyve VM won't Start

[kgmoney]

Hi,

I'm trying to setup a VM per https://doc.freenas.org/11/vms.html. I'm running on an AMD and the output of

Code:

grep POPCNT /var/run/dmesg.boot

is

Code:

Features2=0x802009<SSE3,MON,CX16,POPCNT>

Here is the full CPU info:

Code:

CPU: AMD Phenom(tm) II X6 1090T Processor (3192.06-MHz K8-class CPU)
  Origin="AuthenticAMD"  Id=0x100fa0  Family=0x10  Model=0xa  Stepping=0
  Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0x802009<SSE3,MON,CX16,POPCNT>
  AMD Features=0xee500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM,3DNow!+,3DNow!>
  AMD Features2=0x37ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,SKINIT,WDT>
  SVM: NP,NRIP,NAsids=64
  TSC: P-state invariant, performance statistics

The only thing I see in /var/log/messages when I try to start the vm is the following:

Code:

Jun 30 14:19:40 freenas tap3: Ethernet address: 00:bd:4a:eb:88:03
Jun 30 14:19:40 freenas kernel: tap3: promiscuous mode enabled

I create the VM in the web interface and everthing looks good, however when I try to start it, the status remains 'STOPPED'. Are there logs somewhere that I could use for troubleshooting? Anyone else have issues with AMD? ...I've seen it work on an Intel board.

Thanks,
Kevin

 

[scrappy]

Have you enabled virtualization support in your BIOS? It may be disabled by default.

 

[nemisisak]

Agree with scrappy. In theory the processor should be supported as the processor has population count and nrip features available therefore should support virtualisation. BIOS could therefore be blocking the VM.

 

[kgmoney]

Thanks for posting guys. I double checked the bios and virtualization support is enabled. Just for fun I disabled it and checked dmesg again...

Code:

CPU: AMD Phenom(tm) II X6 1090T Processor (3192.06-MHz K8-class CPU)
  Origin="AuthenticAMD"  Id=0x100fa0  Family=0x10  Model=0xa  Stepping=0
  Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0x802009<SSE3,MON,CX16,POPCNT>
  AMD Features=0xee500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM,3DNow!+,3DNow!>
  AMD Features2=0x37ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,SKINIT,WDT>
  SVM: (disabled in BIOS) NP,NRIP,NAsids=64
  TSC: P-state invariant, performance statistics

There you can see where is now states it is disabled. So I renabled it and I'm still stuck :(

Is there documentation somewhere about controlling the VMs from the command line? ...I wonder if that might yield more info on why it's not working.

 

[kgmoney]

So I figured out that I should be looking for iohyve rather than bhyve. When I do the iohyve list command, it finds no VMs. Is it safe to assume, that using the iohyve commands from the shell should work with VMs create from the GUI? If so, it seems that maybe the GUI isn't actually creating the VMs at all! I'm tempted to try a reinstall of freenas, but it doesn't seem like that should make any difference.


EDIT: Looks like that is not the case. I tried iohyve list on a freenas (intel based) with a vm running and it didn't list anything either.

 

[Ellimist]

I have the exact same issue. I'm not sure what I'm supposed to see on the command line but I get the same result from dmesg and also the same result for iohyve.

I went looking for documentation for running and creating VM's on the command line in freenas 11 and ended up here. An info would be good as freenas 11's documentation is somewhat lacking and mixed up with corral documentation at the moment.

 

[BigDave]

An info would be good as freenas 11's documentation is somewhat lacking and mixed up with corral documentation at the moment.

:(
I was under the impression the documentation is very well done.
Are you speaking of section 2.6 in particular?
http://doc.freenas.org/11/freenas.html

 

[Ellimist]

No this is in reference to section 14. Its great for GUI only but provides no troubleshooting.

If we look at the active directory section (9.1 and its subsection) there is plenty in there to help you troubleshoot when the GUI goes wrong. For Vm's theres nada and the issue we are facing is that nothing is even displayed when you click the start button.

 

[breeegz]

I'm running into the same issue. VM's won't start, stays in the "STOPPED" state.

I'm running a dual XEON CPU and virtualization is turned on in BIOS. /var/log/messages is the same as listed above when I try to start it. Ethernet device turns to promiscuous mode and nothing more.

 

[kgmoney]

I've not thoroughly tested, but it seems to be working for me now after updating to 11.1. I'll mark it solved unless I find otherwise.

 

[Ellimist]

So in 11.1 VM's are launching fine. I built a Windows VM but it locks up for some reason after a while. Not sure why yet. Haven't looked into it.

 

[breeegz]

I've not thoroughly tested, but it seems to be working for me now after updating to 11.1. I'll mark it solved unless I find otherwise.

Did you do anything to get it to start working after other than update? I'm on a new install of 11.1-Stable and not able to start VM's. I'm still hunting for a setting I missed, a command that sheds light on what is broken, or a error log that points me in the right direction.

 

[breeegz]

iohyve list doesn't list anything, although I'm not sure that it is supposed to.
iohyve version is "v0.7.9 2017/06/26 I Believe in You, You Can Do the Thing Edition"
iohyve getall [vm-name] gives "no such file or directory"
zfs list shows that my ZVOL is created and listed
-if I try to browse the folder structure, I don't see it listed under /mnt/Volume1/
bhyve help doesn't seem to have anything useful for my issue, looks like it's mostly used to set up new VM's. I want to run an already created VM or discover why the GUI is failing to launch/start the VM's
dmesg shows my Ethernet address and "tap0: promiscuous mode enabled" every time I try to start my VM

I don't mean to hijack someone else's thread, but from what I can tell from the above posts, I'm running into the same issue as the OP.

 

[xlf]

trying to get a grip on this as well.

From what i figured the last 2 days: iohyve and the gui vms seem to work on completely separate planets. They do not know of each other.

So, it's either try to get the vm running with the gui, or try to get things done with iohyve, but not see the results in the gui.

 

[breeegz]

My output:

Code:

root@freenas:~ # grep VT-x /var/run/dmesg.boot
  VT-x: HLT,PAUSE

(2x XEON X5365, https://ark.intel.com/products/30702/Intel-Xeon-Processor-X5365-8M-Cache-3_00-GHz-1333-MHz-FSB)

This doesn't match what the guide said was a minimum requirement (EPT & UG), so I guess that means that I'm SOL when it comes to the FreeNAS version of virtualization, aka bhyve. I think that there should be some sort of error message if a VM tries to start and fails, it would help with issues like the ones listed in this thread and the others like it.

Thanks for your time and effort.

 

[ViciousXUSMC]

Just got done moving my FreeNAS from bare metal to ESXi and thought I broke my VMs because I did a snapshot of my ZVols and moved it from one pool to another because I took my pool where the VM disks where and made that my ESXi Local Datastore.

Thanks to this thread I was able to connect the dots that VTx may not be enabled by default, sure enough I found it is a setting for the VM and was able to resolve it. So just a heads up for anybody else.

 

[Chris L]

Just in case it was not directly clear running

Code:

grep VT-x /var/run/dmesg.boot

you will get output such as

Code:

[root@ltnas ~]# grep VT-x /var/run/dmesg.boot								   
  VT-x: (disabled in BIOS) HLT,PAUSE											
  VT-x: (disabled in BIOS) HLT,PAUSE

in my case it VT-x was disabled in BIOS as shown above

 

[createcsoft]

this was my problem at /var/log/middlewared.log, and saw that I move a file that was on divices

[2018/08/22 17:33:57] (DEBUG) VMService.run():231 - Starting bhyve: bhyve -H -w -c 2 -m 2048 -s 0:0,hostbridge -s 31,lpc -l com1,/dev/nmdm51A -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -s 3,virtio-net,tap0,mac=00:a0:98:31:63:85 -s 29,fbuf,vncserver,tcp=0.0.0.0:5951,w=1024,h=768,, -s 30,xhci,tablet -s 4,ahci-hd,/dev/zvol/HD4TB/ws2012m1_ws2012m1_clone0 -s 5,ahci-cd,/mnt/HD4TB/ISO/virtio-win-0.1.96.iso 51_ws2012m1_clone0
[2018/08/22 17:33:57] (DEBUG) VMService.run():250 - ws2012m1_clone0: bhyve: Could not open backing file: /mnt/HD4TB/ISO/virtio-win-0.1.96.iso: No such file or directory

 

[threatbutt]

The title on this is misleading; there is no solution offered beyond playing update roulette.

createcsoft was onto something though-- there is a lot of information to be found in /var/log/middlewared.log. In my case, the reason why the VM was failing was right there in plain English:

Code:

[2018/10/12 20:59:43] (DEBUG) VMService.run():250 - Windows10: 12/10/2018 13:59:42 Listening for VNC connections on TCP6 port 5901
[2018/10/12 20:59:43] (DEBUG) VMService.run():250 - Windows10: ROM boot failed: unrestricted guest capability not available
[2018/10/12 20:59:43] (DEBUG) VMService.run():250 - Windows10: fbuf frame buffer base: 0x904600000 [sz 16777216]
[2018/10/12 20:59:43] (INFO) VMService.run():265 - ===> Powered off VM: Windows10 ID: 1 BHYVE_CODE: 1

...so my processor doesn't support UG, like breegz discovered.