Problem shutting down VM in TrueNAS-SCALE-22.02.3

[joeystacks]

What happened?:

1. System fails to restart using GUI - MUST USE CONSOLE
2. System fails to allow VM to restart - ERROR: FAILED - Requested operation is not valid: domain is not running

Any recent changes?: Yes. Recently swapped network cards around in the system and had to repair the network bridge.

Full error:
Error: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/middlewared/job.py", line 411, in run await self.future File "/usr/lib/python3/dist-packages/middlewared/job.py", line 448, in __run_body rv = await self.middleware.run_in_thread(self.method, *([self] + args)) File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1208, in run_in_thread return await self.run_in_executor(self.thread_pool_executor, method, *args, **kwargs) File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1205, in run_in_executor return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs)) File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run result = self.fn(*self.args, **self.kwargs) File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1276, in nf return func(*args, **kwargs) File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1146, in nf res = f(*args, **kwargs) File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_lifecycle.py", line 73, in stop self._stop(vm_data['name'], vm_data['shutdown_timeout']) File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_supervisor.py", line 70, in _stop self.vms[vm_name].stop(shutdown_timeout) File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/supervisor/supervisor_base.py", line 213, in stop self.domain.shutdown() File "/usr/lib/python3/dist-packages/libvirt.py", line 2905, in shutdown raise libvirtError('virDomainShutdown() failed') libvirt.libvirtError: Requested operation is not valid: domain is not running

OS/System Info:

OS Version:TrueNAS-SCALE-22.02.3
Product:PowerEdge T420
Model:Intel(R) Xeon(R) CPU E5-2420 v2 @ 2.20GHz (x2)
Memory:94 GiB

Requests for help.

Thanks.

 

[morganL]

What happened?:

1. System fails to restart using GUI - MUST USE CONSOLE
2. System fails to allow VM to restart - ERROR: FAILED - Requested operation is not valid: domain is not running

Any recent changes?: Yes. Recently swapped network cards around in the system and had to repair the network bridge.

Full error:
Error: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/middlewared/job.py", line 411, in run await self.future File "/usr/lib/python3/dist-packages/middlewared/job.py", line 448, in __run_body rv = await self.middleware.run_in_thread(self.method, *([self] + args)) File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1208, in run_in_thread return await self.run_in_executor(self.thread_pool_executor, method, *args, **kwargs) File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1205, in run_in_executor return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs)) File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run result = self.fn(*self.args, **self.kwargs) File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1276, in nf return func(*args, **kwargs) File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1146, in nf res = f(*args, **kwargs) File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_lifecycle.py", line 73, in stop self._stop(vm_data['name'], vm_data['shutdown_timeout']) File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_supervisor.py", line 70, in _stop self.vms[vm_name].stop(shutdown_timeout) File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/supervisor/supervisor_base.py", line 213, in stop self.domain.shutdown() File "/usr/lib/python3/dist-packages/libvirt.py", line 2905, in shutdown raise libvirtError('virDomainShutdown() failed') libvirt.libvirtError: Requested operation is not valid: domain is not running

OS/System Info:

OS Version:TrueNAS-SCALE-22.02.3
Product:PowerEdge T420
Model:Intel(R) Xeon(R) CPU E5-2420 v2 @ 2.20GHz (x2)
Memory:94 GiB

Requests for help.

Thanks.

Is networking working.. all IP addresses and DNS?

 

[joeystacks]

Is networking working.. all IP addresses and DNS?

Yes. Everything is functioning (other vms and containers working). I have temporarily resolved it with reducing the over utilization of the RAM and it seems to have resolved it for some time. Used to Proxmox where over utilization wasn't a problem. Is TrueNas's implementation for VMs significantly different that over utilization on a system resource is a problem?

 

[joeystacks]

Yes. Everything is functioning (other vms and containers working). I have temporarily resolved it with reducing the over utilization of the RAM and it seems to have resolved it for some time. Used to Proxmox where over utilization wasn't a problem. Is TrueNas's implementation for VMs significantly different that over utilization on a system resource is a problem?

Update: Even with the changes to utilization, I'm still having this issue. VM shows black screen when connecting to console via web, cannot RDP to device, ping is dead with 100% packet loss to the VM.

 

[joeystacks]

Update: Even with the changes to utilization, I'm still having this issue. VM shows black screen when connecting to console via web, cannot RDP to device, ping is dead with 100% packet loss to the VM.

Also: Other VM's will shut down and have no issues with it. This one however will not and gives me the domain error even though it's not connected to a domain.