Device Passthrough Error Operation Not Permitted

[LoneStarfish]

I'm trying to get a VM up and running but having difficulty with passing through my USB controller. I finally got it separated out to a IOMMU group where it's all alone, however I now get a new error:

Code:

Error: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/supervisor/supervisor_base.py", line 165, in start
    if self.domain.create() < 0:
  File "/usr/lib/python3/dist-packages/libvirt.py", line 1353, in create
    raise libvirtError('virDomainCreate() failed')
libvirt.libvirtError: internal error: process exited while connecting to monitor: 2022-09-04T22:23:30.894130Z qemu-system-x86_64: -device vfio-pci,host=0000:02:00.0,id=hostdev0,bus=pci.0,addr=0x7: vfio 0000:02:00.0: failed to setup container for group 2: Failed to set iommu for container: Operation not permitted

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 176, in call_method
    result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self)
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1293, in _call
    return await methodobj(*prepared_call.args)
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1272, in nf
    return await func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1140, in nf
    res = await f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_lifecycle.py", line 39, in start
    await self.middleware.run_in_thread(self._start, vm['name'])
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1208, in run_in_thread
    return await self.run_in_executor(self.thread_pool_executor, method, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1205, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
  File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_supervisor.py", line 62, in _start
    self.vms[vm_name].start(vm_data=self._vm_from_name(vm_name))
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/supervisor/supervisor_base.py", line 174, in start
    raise CallError('\n'.join(errors))
middlewared.service_exception.CallError: [EFAULT] internal error: process exited while connecting to monitor: 2022-09-04T22:23:30.894130Z qemu-system-x86_64: -device vfio-pci,host=0000:02:00.0,id=hostdev0,bus=pci.0,addr=0x7: vfio 0000:02:00.0: failed to setup container for group 2: Failed to set iommu for container: Operation not permitted

I have no idea what to make of it and haven't been able to resolve with any of my googling and guide reading.
The system I'm on is an old Dell T7400 (Xeon E5430) that should be fully capable of virtualization, and I've made sure to enable all of the virtualization features in the bios. Running TrueNAS-SCALE-22.02.3

As always, I'm sure I've left things out so let me know if there are any other specs or info you'd like me to get. Thanks in advance to any of you kind folk that may help out

 

[ChrisRJ]

The system I'm on is an old Dell T7400 (Xeon E5430) that should be fully capable of virtualization,

This platform has limitations when it comes to virtualization. I don't remember the details (sorry), but it was mentioned a couple of times here.

 

[LoneStarfish]

This platform has limitations when it comes to virtualization. I don't remember the details (sorry), but it was mentioned a couple of times here.

I haven't been able to find any reference to it in my searching, but I did fear I'd run into issues with the age of the hardware sooner or later. My goal was to use this as a proving ground for my first-time setup before spending money on a proper rig since I had this old workstation laying around. It's also been recommended that I run something like proxmox as my hypervisor for VMs with truenas running underneath, so I may give that a shot as well as a last-ditch effort to tide me over 'til new hardware.

 

[ChrisRJ]

IIRC Sandybridge CPUs/boards or newer should work here. You definitely don't need something really new. Just not that old ;-)

 

[LoneStarfish]

I have had some small amount of success with my proxmox experiment, but not enough to keep this hardware viable very long. It's still having the same issue passing through the full PCI device (confirming it's a hardware limitation) however it is able to pass through individual USB devices (not hot-swappable sadly). This may push up my hardware upgrade timeline after all.

As far as what hardware I upgrade to - I see a couple routes. Obviously, there are plenty of new-enough server options on eBay for not much money, but for the sake of efficiency (noise, heat, power) as well as ensuring I have the performance for some of my near future project ideas I may just go ahead and spend the money on building something much newer. Current thinking is maybe a high core count 5th-gen Ryzen 7 or 9 taking advantage of the price drops with the newest releases. Decisions, decisions...

 

[ChrisRJ]

The first thing is to get clear on the use-case. For a NAS-only scenario you typically do not need more than 4 cores, unless we are talking serious business use (and at least 5 figures money). Also, relative to the power drawn by HDDs the CPU may(!) be less relevant. If, as an example, you are running 24 disks, the overall consumption will likely be around 300 W, so saving 15 W on a newer CPU may not be worth the extra spending.

Spending money on "future use" is usually not something that pays off in my experience. Anyway, if you share your plans we can probably offer some advice.

 

[LoneStarfish]

The NAS use is actually pretty low on my priorities, as I really don't have a need for a ton of data storage. I'm more interested in having higher speed storage than capacity, so I'm leaning towards something in the ballpark of 3-5 sata ssds in my "final" deployment which will have the side effect of being more power efficient. Primary use is for a few VMs running a 3d print server, homeassistant, and likely a windows instance for design simulations. The simulation case is where I'm most concerned with cpu performance (single and multi-threaded) and also benefits heavily from having fast storage.