VM management on web ui

[Chavell3]

HI everybody,

I have a fresh installation of TrueNAS and wanted to use the virtual machine feature.
The first try to install this VM was okay, but later on the VNC connection dropped and the VM was in a stale state.
I tried to stop the VM from the web ui, but unfortunatly got only an error and so the VM is not manageable... as it can't be stopped...

maybe does someone have some experience around the VM feature usage and this error?
See attached the error from the web ui, I get when I try to stop the VM.

Code:

Error: Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/middlewared/job.py", line 361, in run
    await self.future
  File "/usr/local/lib/python3.8/site-packages/middlewared/job.py", line 399, in __run_body
    rv = await self.middleware.run_in_thread(self.method, *([self] + args))
  File "/usr/local/lib/python3.8/site-packages/middlewared/utils/run_in_thread.py", line 10, in run_in_thread
    return await self.loop.run_in_executor(self.run_in_thread_executor, functools.partial(method, *args, **kwargs))
  File "/usr/local/lib/python3.8/site-packages/middlewared/utils/io_thread_pool_executor.py", line 25, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.8/site-packages/middlewared/schema.py", line 977, in nf
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/vm.py", line 1563, in stop
    vm.stop(vm_data['shutdown_timeout'])
  File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/vm.py", line 251, in stop
    self.domain.shutdown()
  File "/usr/local/lib/python3.8/site-packages/libvirt.py", line 2659, in shutdown
    if ret == -1: raise libvirtError ('virDomainShutdown() failed', dom=self)
libvirt.libvirtError: An error occurred, but the cause is unknown

Thanks.

BR
Chavell3

 

[Chavell3]

btw...

can someone tell me, what VM disk formats are supported for VM used with TrueNAS?

Chavell3

 

[Patrick M. Hausen]

Only raw format.

 

[Samuel Tai]

You can try killing the VM directly: pgrep -fl bhyve will yield the bhyve VMs running, along with their VM IDs and names. Then kill -9 <bhyve process ID> will terminate the rogue VM.

 

[Chavell3]

Thanks Samuel,
unfortunately if I run

pgrep -fl bhyve

I do not get any output.
But if I look from the Web ui the VM seems to be running...

 

[Samuel Tai]

OK, so the VM is actually down, but the web GUI hasn't caught up yet. Try rebooting to clear the GUI.

 

[Chavell3]

@Samuel Tai
yes rebooting the machine fixes the issue, for short term... but after some time the issue occurs again... so is there maybe a longterm solution that I can start boot the VM?

 

[Chavell3]

I also tried to start the VM manually via cli,

vm list
vm start vmName

but unfortunatly the VM is not listed...

 

[Chavell3]

should I maybe open a case at https://jira.ixsystems.com/projects/NAS/issues/NAS-103664?filter=allopenissues
?

 

[Samuel Tai]

Look in /var/log/middlewared for any lines containing bhyve. There may be an indication there why this VM keeps crashing. Also, what guest is running in the VM, and how did you configure the VM in TrueNAS?

 

[Chavell3]

I wanted to install a windows machine with 2 Cores and 5Gb RAM.
My Server uses a 2 Cores(4 Threads) CPU and has 16GB of RAM.
If I look in your named log file I only find the following(for bhyve):

Code:

libvirt.libvirtError: internal error: Child process (/usr/sbin/bhyvectl --destroy --vm=1_CR_Client) unexpected exit status 1

 

[Samuel Tai]

Try setting these loader tunables to reflect your CPU:

Code:

hw.vmm.topology.cores_per_package=2
hw.vmm.topology.threads_per_core=2

These take effect on reboot. This lets bhyve know your CPU supports HyperThreading.

 

[Chavell3]

currently I try to check multiple things, coming back soon.

 

[Chavell3]

I found from the documentation, that at

/var/log/vm/vmname.

there should be some VM logs... but there aren't...
additionally if tried the command

vm list

but initially I got an output that some settings e.g.

$vm_enable

in rc.conf is not set...

so I set those commands, I found:

vm_enable="YES"
vm_dir="/mnt/data/"
vm_list=""
vm_delay="5"

But actually i don't no, where to set the "vm_dir" because I cannot found any information about that...
Another but, now that I set those commands in rd.conf the command

vm list

works...
but it does not list any vm's.

any idea how I can fix that?
Thanks

 

[Samuel Tai]

Actually, you should back out of your loader.conf changes; those are for CLI management of bhyve VMs that are incompatible with the GUI. For CLI status compatible with the GUI, you should use API calls via midclt call vm.query to get a list of defined VMs by ID and whether they're running or not.

 

[Chavell3]

and is there also a command to start or stop a vm via midctl?
I guess that could be enough for me...

 

[Samuel Tai]

Yes, midclt call vm.start <VM ID> and midclt call vm.stop <VM ID>. The IDs are the integer values listed in the "id" fields from the midclt call vm.query output.

 

[Chavell3]

yes, I found that too... but unfortunatly the vm is no stopable... :(
or I mean, it just not stops

 

[Samuel Tai]

Try midclt call vm.stop <VM ID> "force":true.

 

[Chavell3]

the return from your command was 4743
is midclt call vm.query representing the web state of the vm?
because if I run this command, the vm is still in running state.