Virtual Machine non-functional after upgrade from Bluefin

[ShakataGaNai]

Updated from Bluefin to Cobia (TrueNAS-SCALE-23.10.1.3) and it's not been super smooth. Most of the issues I've slowly been working out but the one I'm not sure what to do with at all is Virtual Machines. I have a single very small VM that I need for VPN routing and after the upgrade it will not start at all.

Attempting to start it provides the error message:

[EFAULT] internal error: process exited while connecting to monitor: Failed to open /dev/null for OFD lock probing: Operation not permitted 2024-01-29T19:06:55.393190Z qemu-system-x86_64: -blockdev {"driver":"file","filename":"/dev/null","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}: 'file' driver requires '/dev/null' to be a regular file

More info...

Code:

 Error: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/supervisor/supervisor.py", line 182, in start
    if self.domain.create() < 0:
       ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/libvirt.py", line 1373, in create
    raise libvirtError('virDomainCreate() failed')
libvirt.libvirtError: internal error: process exited while connecting to monitor: Failed to open /dev/null for OFD lock probing: Operation not permitted
2024-01-29T19:06:55.393190Z qemu-system-x86_64: -blockdev {"driver":"file","filename":"/dev/null","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}: 'file' driver requires '/dev/null' to be a regular file

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 201, in call_method
    result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1342, in _call
    return await methodobj(*prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 177, in nf
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 44, in nf
    res = await f(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_lifecycle.py", line 54, in start
    await self.middleware.run_in_thread(self._start, vm['name'])
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1254, in run_in_thread
    return await self.run_in_executor(self.thread_pool_executor, method, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1251, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_supervisor.py", line 68, in _start
    self.vms[vm_name].start(vm_data=self._vm_from_name(vm_name))
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/supervisor/supervisor.py", line 191, in start
    raise CallError('\n'.join(errors))
middlewared.service_exception.CallError: [EFAULT] internal error: process exited while connecting to monitor: Failed to open /dev/null for OFD lock probing: Operation not permitted
2024-01-29T19:06:55.393190Z qemu-system-x86_64: -blockdev {"driver":"file","filename":"/dev/null","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}: 'file' driver requires '/dev/null' to be a regular file

I've also attached the output from "Download Logs". One odd thing I've noticed is that the datasets includes two datasets for this VM (which I don't recall being there before upgrade).

 

[Kris Moore]

That log looks like it has some clues to an issue in the config. Can you open a bug ticket and attach the log / debugs to it? We'll need to investigate to see if its something we can solve on upgrade for you.

 

[rhuneai]

Did you get any resolution with this?
I just upgraded from Bluefin to Cobia and have the same issue with my Proxmox Backup Server VM.

 

[ShakataGaNai]

Did you get any resolution with this?
I just upgraded from Bluefin to Cobia and have the same issue with my Proxmox Backup Server VM.

No. Unfortunately, in my tinkering, I blew up my VM config by accident, so I couldn't do any additional diagnostics for the team.