Single GPU passthrough and the qemu/libvirt 'romfile' option

[Neuromante]

Hello everyone, i have used trueNAS core (and way back FreeNAS) before for storage purposes. When i found out that trueNAS SCALE was a thing, i wanted to try it out (especially the containerization ad virtualization capabilities). My workflow up to now involved Proxmox as the hypervisor, so i wanted to try and switch to SCALE replicating the VMs and infrastructure i currently have there.

Most of my VMs have multiple pci-e devices passed through (USB controllers, audio controllers, TPM modules etc.) and every one of them (concurrency is not a thing) has the same GPU assigned. The aforementioned GPU is the main display boot device and indeed it suffers from VBIOS shadowing by the kernel.

This is resolved by either moving it to a secondary slot (and using another card as the boot device) or by dumping the vbios when the card is unshadowed and passing it to qemu (libvirt in SCALE's case). In Proxmox i chose the latter approach to save on power and pci-e lanes being used.

Now to the point of this post, i can't find a consistent way of passing the romfile= option to a VM that i previously created in SCALE and fully tested as working. The VM in question is a Arch Linux VM using a desktop environment (so it has a login screen that turns on a dislay automatically).

I've tried to edit the VM xml configuration file in /etc/libvrt/qemu/ path, but by doing so when i start the VM from the webUI the operation fails citing:

failed to connect to Libvirt

The error persists even when the following offending line is remove or commented in the VM's configuration xml:

Code:

  <rom file='/root/rx480-vbios.bin'/>

For reference, here's the full traceback:

Code:

Error: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 139, in call_method
    result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self,
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1207, in _call
    return await methodobj(*prepared_call.args)
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1004, in nf
    return await f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_lifecycle.py", line 24, in start
    await self.middleware.run_in_thread(self._check_setup_connection)
  File "/usr/lib/python3/dist-packages/middlewared/utils/run_in_thread.py", line 10, in run_in_thread
    return await self.loop.run_in_executor(self.run_in_thread_executor, functools.partial(method, *args, **kwargs))
  File "/usr/lib/python3/dist-packages/middlewared/utils/io_thread_pool_executor.py", line 25, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/connection.py", line 44, in _check_setup_connection
    self._check_connection_alive()
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/connection.py", line 39, in _check_connection_alive
    raise CallError('Failed to connect to libvirt')
middlewared.service_exception.CallError: [EFAULT] Failed to connect to libvirt

Is the romfile option being considered for inclusion in the webUI? if so, should i move to nightly releases to test some more? Im currently running TrueNAS-SCALE-21.06-BETA.1.

Thanks for the support, keep up the good work!

 

[morganL]

Romfile option doesn't seem to be in Jira or on the roadmap. Make it a suggestion and see how many other people want it.