VM failure libvert QEMU driver error

[kuruption]

One of my VM's won't come back up after upgrading to 22.02.2.1 (I don't know if it's related, it just happened after the reboot to this version).

I have three VM's, two of them booted fine (one Windows, one Linux)... this third Linux VM won't start. I am seeing the following libvert and QEMU errors in my logs when I start this VM.

Any help to fixing this would be appreciated.

messages​

Jul 23 08:16:44 nas.home.sundland.net kernel: audit: type=1400 audit(1658578604.338:26): apparmor="DENIED" operation="change_profile" info="label not found" error=-2 profile="libvirtd" name="libvirt-713871d2-b8c7-485d-b51d-f0800e799d71" pid=2287741 comm="rpc-worker"
Jul 23 08:16:45 nas.home.sundland.net middlewared[1880]: libvirt: QEMU Driver error : internal error: Process exited prior to exec: libvirt: error : unable to set AppArmor profile 'libvirt-713871d2-b8c7-485d-b51d-f0800e799d71' for '/usr/bin/qemu-system-x86_64': No such file or directory

middlewared.log​

[2022/07/23 08:16:42] (DEBUG) middlewared.__set_guest_vmemory():22 - Setting ARC from 128723941376 to 94364203008
[2022/07/23 08:16:45] (DEBUG) VMService.teardown_guest_vmemory():53 - Giving back guest memory to ARC: 128723941376
[2022/07/23 08:17:15] (WARNING) middlewared._loop_monitor_thread():1663 - Task seems blocked:
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1133, in nf
res = await f(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/pci.py", line 99, in passthrough_device
cp = await run(get_virsh_command_args() + ['nodedev-dumpxml', device], check=False)
File "/usr/lib/python3/dist-packages/middlewared/utils/__init__.py", line 50, in run
proc = await asyncio.create_subprocess_exec(*args, **kwargs)
File "/usr/lib/python3.9/asyncio/subprocess.py", line 236, in create_subprocess_exec
transport, protocol = await loop.subprocess_exec(
... + 10 lines below ...
[2022/07/23 08:17:38] (WARNING) middlewared._loop_monitor_thread():1663 - Task seems blocked:
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1133, in nf
res = await f(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/pci.py", line 99, in passthrough_device
cp = await run(get_virsh_command_args() + ['nodedev-dumpxml', device], check=False)
File "/usr/lib/python3/dist-packages/middlewared/utils/__init__.py", line 50, in run
proc = await asyncio.create_subprocess_exec(*args, **kwargs)
File "/usr/lib/python3.9/asyncio/subprocess.py", line 236, in create_subprocess_exec
transport, protocol = await loop.subprocess_exec(
... + 10 lines below ...

auth.log​

Jul 23 08:16:44 nas.home.sundland.net audit[2287741]: AVC apparmor="DENIED" operation="change_profile" info="label not found" error=-2 profile="libvirtd" name="libvirt-713871d2-b8c7-485d-b51d-f0800e799d71" pid=2287741 comm="rpc-worker"

 

[kuruption]

Ok my googlefoo kicked in and found a solution from https://estada.ch/2020/12/21/unable-to-start-kvm-virtual-machines-with-libvirt-and-apparmor/

Since things disappear from the internet, the answer is to go under /etc/apparmor.d/libvert and delete the offending profile file... in my case it is 'libvirt-713871d2-b8c7-485d-b51d-f0800e799d71'. After deleting, I could restart the VM.