Struggling to passthrough a VIA-based USB3 PCI card to a VM (Debian, for Home assistant)

flotueur · Oct 10, 2023

Hi all

After struggling to enable a PCI card in TrueNAS (see details here) I am now unable to pass the PCI card to the VM as the "Add device" / "PCI Passthrough Device" is appearing empty, preventing me from selecting my device.

Initially after a fresh install of TrueNAS, the "PCI Passthrough Device" drop down wasn't empty and I was able to choose my PCI USB3 card. The VM couldn't start because of that and I fixed it by running "iommu=soft" to grub using "midclt call system.advanced.update '{"kernel_extra_options": "iommu=soft"}'".
Now the drop down is completely empty and wouldn't load. I don't even know where to look for the logs.

I would appreciate your help here as I lack the skills to continue on my own!

Note that it is possible to select the Conbee stick from the USB passthrough sub menu but the VM (Home Assistant) has trouble using it.
Ultimately I feel this is better to passthrough the controller so that i can also add additional USB devices to the VM.

Here are some details:

OS Version:TrueNAS-SCALE-22.12.3.3
Motherboard: Supermicro a2sdi-4c-hln4f
Model:Intel(R) Atom(TM) CPU C3558 @ 2.20GHz
Memory:31 GiB
USB 3 controller: VIA Technologies, Inc. VL805 USB 3.0 Host Controller (rev 01)
"iommu=soft" added to grub

My lspci:

Code:

root@freenas[~]# lspci
00:00.0 Host bridge: Intel Corporation Atom Processor C3000 Series System Agent (rev 11)
00:04.0 Host bridge: Intel Corporation Atom Processor C3000 Series Error Registers (rev 11)
00:05.0 Generic system peripheral [0807]: Intel Corporation Atom Processor C3000 Series Root Complex Event Collector (rev 11)
00:06.0 PCI bridge: Intel Corporation Atom Processor C3000 Series Integrated QAT Root Port (rev 11)
00:0a.0 PCI bridge: Intel Corporation Atom Processor C3000 Series PCI Express Root Port #1 (rev 11)
00:10.0 PCI bridge: Intel Corporation Atom Processor C3000 Series PCI Express Root Port #6 (rev 11)
00:11.0 PCI bridge: Intel Corporation Atom Processor C3000 Series PCI Express Root Port #7 (rev 11)
00:12.0 System peripheral: Intel Corporation Atom Processor C3000 Series SMBus Contoller - Host (rev 11)
00:13.0 SATA controller: Intel Corporation Atom Processor C3000 Series SATA Controller 0 (rev 11)
00:14.0 SATA controller: Intel Corporation Atom Processor C3000 Series SATA Controller 1 (rev 11)
00:15.0 USB controller: Intel Corporation Atom Processor C3000 Series USB 3.0 xHCI Controller (rev 11)
00:16.0 PCI bridge: Intel Corporation Atom Processor C3000 Series Integrated LAN Root Port #0 (rev 11)
00:17.0 PCI bridge: Intel Corporation Atom Processor C3000 Series Integrated LAN Root Port #1 (rev 11)
00:18.0 Communication controller: Intel Corporation Atom Processor C3000 Series ME HECI 1 (rev 11)
00:1f.0 ISA bridge: Intel Corporation Atom Processor C3000 Series LPC or eSPI (rev 11)
00:1f.2 Memory controller: Intel Corporation Atom Processor C3000 Series Power Management Controller (rev 11)
00:1f.4 SMBus: Intel Corporation Atom Processor C3000 Series SMBus controller (rev 11)
00:1f.5 Serial bus controller [0c80]: Intel Corporation Atom Processor C3000 Series SPI Controller (rev 11)
01:00.0 Co-processor: Intel Corporation Atom Processor C3000 Series QuickAssist Technology (rev 11)
02:00.0 USB controller: VIA Technologies, Inc. VL805 USB 3.0 Host Controller (rev 01)
04:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 03)
05:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 30)
06:00.0 Ethernet controller: Intel Corporation Ethernet Connection X553 1GbE (rev 11)
06:00.1 Ethernet controller: Intel Corporation Ethernet Connection X553 1GbE (rev 11)
07:00.0 Ethernet controller: Intel Corporation Ethernet Connection X553 1GbE (rev 11)
07:00.1 Ethernet controller: Intel Corporation Ethernet Connection X553 1GbE (rev 11)

Regards,
Florian

flotueur · Oct 12, 2023

Any thoughts?
I am concerned that adding the iommu=soft wasn't a great idea to solve my problem but I don't understand it enough :(

NugentS · Oct 12, 2023

I have absolutely no idea I am afraid. You have wooshed past me

flotueur · Oct 12, 2023

Thanks - perhaps this question is easier: how to view the kernel options? I will attemp to remove the iommu=soft and restart my investigation from there...

flotueur · Oct 12, 2023

Ok, so I found the parameter details for IOMMU and changed the Kernel options to noforce:

Code:

midclt call system.advanced.update '{"kernel_extra_options": "iommu=noforce"}'

This enabled me to go back to a state where the card appeared in the PCI list.

I then selected it as a device and started my VM successfully.

For info, my VM is the latest Debian install "debian-12.1.0-amd64-netinst".

Now, in the VM, the card isn't showing.
The lspci lists:

Code:

00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:02.0 VGA compatible controller: Red Hat, Inc. QXL paravirtual graphic card (rev 05)
00:03.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (rev 03)
00:04.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host Controller (rev 03)
00:05.0 Communication controller: Red Hat, Inc. Virtio console
00:06.0 SCSI storage controller: Red Hat, Inc. Virtio block device
00:07.0 Unclassified device [00ff]: Red Hat, Inc. Virtio memory balloon

With the USB controller being very different then what i have. As a reminder, my USB controller is:

USB 3 controller: VIA Technologies, Inc. VL805 USB 3.0 Host Controller (rev 01)

*Confused pause*

After that, I restarted the VM and got the following error:

Code:

[EFAULT] internal error: Unknown PCI header type '127' for device '0000:02:00.0'

 Error: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/supervisor/supervisor.py", line 172, in start
    if self.domain.create() < 0:
  File "/usr/lib/python3/dist-packages/libvirt.py", line 1353, in create
    raise libvirtError('virDomainCreate() failed')
libvirt.libvirtError: internal error: Unknown PCI header type '127' for device '0000:02:00.0'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 204, in call_method
    result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self)
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1344, in _call
    return await methodobj(*prepared_call.args)
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1378, in nf
    return await func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1246, in nf
    res = await f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_lifecycle.py", line 46, in start
    await self.middleware.run_in_thread(self._start, vm['name'])
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1261, in run_in_thread
    return await self.run_in_executor(self.thread_pool_executor, method, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1258, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
  File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_supervisor.py", line 68, in _start
    self.vms[vm_name].start(vm_data=self._vm_from_name(vm_name))
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/supervisor/supervisor.py", line 181, in start
    raise CallError('\n'.join(errors))
middlewared.service_exception.CallError: [EFAULT] internal error: Unknown PCI header type '127' for device '0000:02:00.0'

Back to square 1, when I deduced that IOMMU=soft would be a good kernel option...

What am I missing?

flotueur · Oct 12, 2023

Note that my /etc/default/grub.d/truenas.cfg shows -

Code:

GRUB_DISTRIBUTOR="TrueNAS Scale"
GRUB_TIMEOUT=10
GRUB_CMDLINE_LINUX_DEFAULT="libata.allow_tpm=1 amd_iommu=on iommu=pt kvm_amd.npt=1 kvm_amd.avic=1 intel_iommu=on zfsforce=1 nvme_core.multipath=N iommu=noforce"
GRUB_TERMINAL_INPUT="console"
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX=""

Given my motherboard (A2SDi-4C-HLN4F) supports virtualization, do I really need IOMMU? Is that what is causing the conflict? Bit hesitant to remove kernel options in case i can't reboot the machine afterwards!

flotueur · Oct 17, 2023

Little up please :)
Hopefully someone think about potential way forwards!

sfatula · Oct 20, 2023

My guess is buggy firmware in the VL805, I'd look for firmware updates. You are not the only one with issues with this card.

flotueur · Oct 24, 2023

Thanks - I ended up trying with a different card with a different chipset but same problem... The VM starts for the first time then the second time it shows

"[EFAULT] internal error: Unknown PCI header type '127' for device '0000:02:00.0'"

Code:

 Error: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/supervisor/supervisor.py", line 172, in start
    if self.domain.create() < 0:
  File "/usr/lib/python3/dist-packages/libvirt.py", line 1353, in create
    raise libvirtError('virDomainCreate() failed')
libvirt.libvirtError: internal error: Unknown PCI header type '127' for device '0000:02:00.0'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 204, in call_method
    result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self)
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1344, in _call
    return await methodobj(*prepared_call.args)
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1378, in nf
    return await func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1246, in nf
    res = await f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_lifecycle.py", line 46, in start
    await self.middleware.run_in_thread(self._start, vm['name'])
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1261, in run_in_thread
    return await self.run_in_executor(self.thread_pool_executor, method, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1258, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
  File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_supervisor.py", line 68, in _start
    self.vms[vm_name].start(vm_data=self._vm_from_name(vm_name))
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/supervisor/supervisor.py", line 181, in start
    raise CallError('\n'.join(errors))
middlewared.service_exception.CallError: [EFAULT] internal error: Unknown PCI header type '127' for device '0000:02:00.0'

If that helps, the time i started the VM successfully, this appeared in TrueNAS logs:

Code:

Oct 24 11:55:43 freenas kernel: audit: type=1400 audit(1698144943.046:20): apparmor="DENIED" operation="capable" profile="libvirtd" pid=3804 comm="rpc-worker" capability=39  capname="bpf"

Is it relevant? What does that mean?

sfatula · Oct 24, 2023

Beyond me, I've never seen the issue nor have I ever had it. Someone else will need to chime in.

xyno · Oct 31, 2023

I ran into the same issues when doing a passthrough for a videocard to a VM
I cant tell out of my head the motherboard i was using however I had to passthrough allot more due some <magic hocus pocus weird stuff>

If the bus is shared or the lane, whatever people call it nowerdays, you might also need to forward those to the VM.
This was the solution for me, hopefully it works for you as well..

flotueur · Oct 31, 2023

@xyno Interesting, thanks for sharing! I will have a look into this...

I worked with someone on the discord group to bind the card so that it doesn't unmount everytime the VM restart but that didn't make the cut.
So the problem is still ongoing...

For now, I am using the USB ports for Truenas' purposes (like booting on a USB stick or saving/loading files) and using the MB PCI controller for the VM (the passthrough works in this case). But it took me to purshare some extra hardware to make it happen (I had my boot drive on USB before).

I will let everyone know what is the progress as I am going further... But please if you have some ideas keep posting!

Important Announcement for the TrueNAS Community.

Struggling to passthrough a VIA-based USB3 PCI card to a VM (Debian, for Home assistant)

flotueur

Dabbler

flotueur

Dabbler

NugentS

MVP

flotueur

Dabbler

flotueur

Dabbler

flotueur

Dabbler

flotueur

Dabbler

sfatula

Guru

flotueur

Dabbler

sfatula

Guru

xyno

Dabbler

flotueur

Dabbler

Similar threads