Virtual machine won't autostart after host reboot because passthorugh USB device changed it's "path"

[loty1825]

I'm running TrueNAS-SCALE-23.10.1.3 on a physical machine.

Machine specs:
Intel(R) Xeon(R) CPU E5506 @ 2.13GHz
24G ECC RAM
Supermicro X8DTL-i motherboard
Dell H310 Perc flashed to IT mode

I have a HP M1132 printer connected to it via a USB port. I run an Ubuntu Server virtual machine with USB device passthorugh so I can share the printer on my local network via CUPS. The virtual machine has 1CPU/Core/Thread, 1GB RAM, 10GB AHCI virtual hard drive, Intel e1000 virtual NIC and qemu-xhci USB controller for USB device passthrough.

The virtual machine is set to autostart after host reboot but it doesn't. It throws an error:

Error: Traceback (most recent call last):
File "/usr/lib/python3.11/xml/etree/ElementTree.py", line 1041, in _escape_attrib
if "&" in text:
^^^^^^^^^^^
TypeError: argument of type 'NoneType' is not iterable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_supervisor.py", line 45, in _check_add_domain
self._add(self._vm_from_name(vm_name)['id'])
File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_supervisor.py", line 16, in _add
self._add_with_vm_data(vm)
File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_supervisor.py", line 19, in _add_with_vm_data
self.vms[vm['name']] = VMSupervisor(vm, self.middleware)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/supervisor/supervisor.py", line 30, in __init__
self.update_domain()
File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/supervisor/supervisor.py", line 66, in update_domain
self.__define_domain()
File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/supervisor/supervisor.py", line 98, in __define_domain
vm_xml = etree.tostring(self.construct_xml()).decode()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/xml/etree/ElementTree.py", line 1098, in tostring
ElementTree(element).write(stream, encoding,
File "/usr/lib/python3.11/xml/etree/ElementTree.py", line 743, in write
serialize(write, self._root, qnames, namespaces,
File "/usr/lib/python3.11/xml/etree/ElementTree.py", line 906, in _serialize_xml
_serialize_xml(write, e, qnames, None,
File "/usr/lib/python3.11/xml/etree/ElementTree.py", line 906, in _serialize_xml
_serialize_xml(write, e, qnames, None,
File "/usr/lib/python3.11/xml/etree/ElementTree.py", line 906, in _serialize_xml
_serialize_xml(write, e, qnames, None,
[Previous line repeated 1 more time]
File "/usr/lib/python3.11/xml/etree/ElementTree.py", line 899, in _serialize_xml
v = _escape_attrib(v)
^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/xml/etree/ElementTree.py", line 1064, in _escape_attrib
_raise_serialization_error(text)
File "/usr/lib/python3.11/xml/etree/ElementTree.py", line 1018, in _raise_serialization_error
raise TypeError(
TypeError: cannot serialize None (type NoneType)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 201, in call_method
result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1342, in _call
return await methodobj(*prepared_call.args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 177, in nf
return await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 44, in nf
res = await f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_lifecycle.py", line 54, in start
await self.middleware.run_in_thread(self._start, vm['name'])
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1254, in run_in_thread
return await self.run_in_executor(self.thread_pool_executor, method, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1251, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_supervisor.py", line 67, in _start
self._check_add_domain(vm_name)
File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_supervisor.py", line 47, in _check_add_domain
raise CallError(f'Unable to define domain for {vm_name}: {e}')
middlewared.service_exception.CallError: [EFAULT] Unable to define domain for PrintServer: cannot serialize None (type NoneType)

Don't know how to tell it in proper terms, but the problem is after the host reboots it "forgets" the previous USB path.

For example. Before reboot the "path" is usb_4_2:

After the reboot it's usb 3_2:

When I select this "new" device. The machine starts without any issues.

So I'm looking for a way to force same usb path every time I reboot the host machine. How would I do that? Thanx.

 

[hrw]

I share the pain.

Thought that I can move HomeAssistant to TrueNAS instead of running it on separate machine. Turned out that I cannot because both Bluetooth and Zigbee USB devices (which I use in HA) block VM from booting on NAS reboot.

 

[hrw]

Solution: do not add USB devices to VM but add whole PCI USB controller instead.

PCI cards have fixed IDs so VM will work even after reboot.

 

[loty1825]

I was thinking more of some script or something that would run at startup and "keep connecting and discarding" USB devices until they appear at the desired path. But have no idea how to do that if it's even possible. But if passing through entire USB controller works I could live with that solution too I guess.

I don't have any USB controller card laying around to test the passthrough, so I tried to passthrough the onboard USB controller. I had to choose between 8 USB devices available.

My machine only has 2 physical USB ports at the back, the printer's connected to one of them. So I had to figure out which device that port belongs too. First I disconnected the printer and rebooted the TrueNAS host. Since there's no lsusb command available I used "dmesg | grep -i usb".

[ 2.163744] ACPI: bus type USB registered
[ 2.163858] usbcore: registered new interface driver usbfs
[ 2.163945] usbcore: registered new interface driver hub
[ 2.164044] usbcore: registered new device driver usb
[ 2.178184] ehci-pci 0000:00:1a.7: new USB bus registered, assigned bus number 1
[ 2.195598] ehci-pci 0000:00:1a.7: USB 2.0 started, EHCI 1.00
[ 2.195753] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.01
[ 2.195852] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 2.195948] usb usb1: Product: EHCI Host Controller
[ 2.196023] usb usb1: Manufacturer: Linux 6.1.63-production+truenas ehci_hcd
[ 2.196102] usb usb1: SerialNumber: 0000:00:1a.7
[ 2.196347] hub 1-0:1.0: USB hub found
[ 2.196962] uhci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 2
[ 2.197317] usb usb2: New USB device found, idVendor=1d6b, idProduct=0001, bcdDevice= 6.01
[ 2.197415] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 2.197509] usb usb2: Product: UHCI Host Controller
[ 2.197582] usb usb2: Manufacturer: Linux 6.1.63-production+truenas uhci_hcd
[ 2.197660] usb usb2: SerialNumber: 0000:00:1a.0
[ 2.197862] hub 2-0:1.0: USB hub found
[ 2.198490] uhci_hcd 0000:00:1a.1: new USB bus registered, assigned bus number 3
[ 2.198824] usb usb3: New USB device found, idVendor=1d6b, idProduct=0001, bcdDevice= 6.01
[ 2.198922] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 2.199017] usb usb3: Product: UHCI Host Controller
[ 2.199090] usb usb3: Manufacturer: Linux 6.1.63-production+truenas uhci_hcd
[ 2.199168] usb usb3: SerialNumber: 0000:00:1a.1
[ 2.199369] hub 3-0:1.0: USB hub found
[ 2.199994] uhci_hcd 0000:00:1a.2: new USB bus registered, assigned bus number 4
[ 2.200344] usb usb4: New USB device found, idVendor=1d6b, idProduct=0001, bcdDevice= 6.01
[ 2.201604] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 2.201698] usb usb4: Product: UHCI Host Controller
[ 2.201771] usb usb4: Manufacturer: Linux 6.1.63-production+truenas uhci_hcd
[ 2.201849] usb usb4: SerialNumber: 0000:00:1a.2
[ 2.202050] hub 4-0:1.0: USB hub found
[ 2.202603] uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 5
[ 2.202936] usb usb5: New USB device found, idVendor=1d6b, idProduct=0001, bcdDevice= 6.01
[ 2.203034] usb usb5: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 2.203128] usb usb5: Product: UHCI Host Controller
[ 2.203202] usb usb5: Manufacturer: Linux 6.1.63-production+truenas uhci_hcd
[ 2.203280] usb usb5: SerialNumber: 0000:00:1d.0
[ 2.203485] hub 5-0:1.0: USB hub found
[ 2.204098] uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 6
[ 2.204448] usb usb6: New USB device found, idVendor=1d6b, idProduct=0001, bcdDevice= 6.01
[ 2.204549] usb usb6: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 2.204644] usb usb6: Product: UHCI Host Controller
[ 2.204720] usb usb6: Manufacturer: Linux 6.1.63-production+truenas uhci_hcd
[ 2.204800] usb usb6: SerialNumber: 0000:00:1d.1
[ 2.205107] hub 6-0:1.0: USB hub found
[ 2.205839] ehci-pci 0000:00:1d.7: new USB bus registered, assigned bus number 7
[ 2.223747] ehci-pci 0000:00:1d.7: USB 2.0 started, EHCI 1.00
[ 2.224029] usb usb7: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.01
[ 2.224137] usb usb7: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 2.224231] usb usb7: Product: EHCI Host Controller
[ 2.224305] usb usb7: Manufacturer: Linux 6.1.63-production+truenas ehci_hcd
[ 2.224383] usb usb7: SerialNumber: 0000:00:1d.7
[ 2.224600] hub 7-0:1.0: USB hub found
[ 2.251888] hub 5-0:1.0: USB hub found
[ 2.279581] hub 6-0:1.0: USB hub found
[ 2.280392] uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 8
[ 2.280752] usb usb8: New USB device found, idVendor=1d6b, idProduct=0001, bcdDevice= 6.01
[ 2.280851] usb usb8: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 2.280946] usb usb8: Product: UHCI Host Controller
[ 2.281092] usb usb8: Manufacturer: Linux 6.1.63-production+truenas uhci_hcd
[ 2.281170] usb usb8: SerialNumber: 0000:00:1d.2
[ 2.283112] hub 8-0:1.0: USB hub found

So there's 8 USB hubs, I guess each one corresponding to one of the 8 USB devices. Let's connect the printer and to the dmseg | grep again:

[ 359.181296] usb 7-1: new high-speed USB device number 2 using ehci-pci
[ 359.383075] usb 7-1: New USB device found, idVendor=03f0, idProduct=042a, bcdDevice= 1.00
[ 359.383718] usb 7-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 359.384328] usb 7-1: Product: HP LaserJet Professional M1132 MFP
[ 359.384949] usb 7-1: Manufacturer: Hewlett-Packard
[ 359.385608] usb 7-1: SerialNumber: 000000000QH82P4TPR1a
[ 359.445334] usblp 7-1:1.0: usblp0: USB Bidirectional printer dev 2 if 0 alt 0 proto 2 vid 0x03F0 pid 0x042A
[ 359.447267] usbcore: registered new interface driver usblp

So this time the printer's connected to USB hub no. 7, port 1. Let's disconnect an reconect the printer to the same port.

[ 749.790525] usb 7-1: new high-speed USB device number 3 using ehci-pci
[ 749.992028] usb 7-1: New USB device found, idVendor=03f0, idProduct=042a, bcdDevice= 1.00
[ 749.992672] usb 7-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 749.993301] usb 7-1: Product: HP LaserJet Professional M1132 MFP
[ 749.993932] usb 7-1: Manufacturer: Hewlett-Packard
[ 749.994583] usb 7-1: SerialNumber: 000000000QH82P4TPR1a
[ 750.042043] usblp 7-1:1.0: usblp0: USB Bidirectional printer dev 3 if 0 alt 0 proto 2 vid 0x03F0 pid 0x042A

The printer's still connected to the same hub.

Now let's connect it to the other USB port:

[ 895.989903] usb 7-2: new high-speed USB device number 4 using ehci-pci
[ 896.191380] usb 7-2: New USB device found, idVendor=03f0, idProduct=042a, bcdDevice= 1.00
[ 896.192019] usb 7-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 896.192643] usb 7-2: Product: HP LaserJet Professional M1132 MFP
[ 896.193268] usb 7-2: Manufacturer: Hewlett-Packard
[ 896.193931] usb 7-2: SerialNumber: 000000000QH82P4TPR1a
[ 896.241680] usblp 7-2:1.0: usblp0: USB Bidirectional printer dev 4 if 0 alt 0 proto 2 vid 0x03F0 pid 0x042A

Still same hub number, but this time on port no. 2.

Connect printer to the other USB port again:

[ 1095.950733] usb 7-1: new high-speed USB device number 5 using ehci-pci
[ 1096.151622] usb 7-1: New USB device found, idVendor=03f0, idProduct=042a, bcdDevice= 1.00
[ 1096.152268] usb 7-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 1096.152890] usb 7-1: Product: HP LaserJet Professional M1132 MFP
[ 1096.153511] usb 7-1: Manufacturer: Hewlett-Packard
[ 1096.154130] usb 7-1: SerialNumber: 000000000QH82P4TPR1a
[ 1096.201627] usblp 7-1:1.0: usblp0: USB Bidirectional printer dev 5 if 0 alt 0 proto 2 vid 0x03F0 pid 0x042A

So it seems usb hub numbers that are assigned at system startup get retained untill next reboot. Conencting and disconnecting devices doesn't change them. At this particular startup the hub that both of my rear USB ports are connected to has hub no. 7 assigned and it's serial number is 0000:00:1d.7. Ok let's chose this device for the passthrough.

The machine still won't start. Now we get this:

Not that I have the slightest idea what iommu_groups are but googling around I learn where iommu device directories are. And the directory of group 20 lists 4 devices:

root@truenas[/mnt/pool1/share/samo]# ls /sys/kernel/iommu_groups/20/devices
0000:00:1d.0 0000:00:1d.1 0000:00:1d.2 0000:00:1d.7

So let's add the all to the VM like the error message asks us to:

Now the VM will start for sure right? Well no. Of course it has to complain about something else. Now we get this:

It complains about the only of those 4 devices I added that I realy care about because it's the one my printer's connected to. So it seems it doesn't have some permissions set properly.

One idea I got from googling:

dmesg | grep -e vfio
[ 2247.138243] vfio_iommu_type1_attach_group: No interrupt remapping support. Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform

Don't know what vfio is. But it seems it wants me to enable some unsafe interrupts whatever that is. Sure I would if I knew how to.

Some Debian documentation I found googling mentions creating a configuration file and editing it to enable unsafe interrupts. Let' try it:

nano /etc/modprobe.d/kvm_iommu.conf

Add options kvm allow_unsafe_assigned_interrupts=1 to the file:

Virtual machine still won't start which is expected. I'm pretty sure restarting some service is required to read the conf file. I have no idea which one so let's reboot. After reboot it still doesn't work with same error.

I also came across a command to check if unsafe interrupts are enabled.

root@truenas[/]# cat /sys/module/vfio_iommu_type1/parameters/allow_unsafe_interrupts

N

I guess it would say Y if enabled.

So the question now is how do I enable unsafe interrupts in TrueNAS Scale?

 

[Patrick M. Hausen]

One person to whom I recommended getting a reasonably cheap PCIe USB card for pass through in TN CORE did exactly that and reported success. I never tried it personally, but if you have a free slot in your system, I would again recommend going that way.

If you buy mail order at least in Europe you have by law the right to return the card if it doesn't work out. Two weeks time to test if I am not mistaken. Fair conduct (in my opinion) would be to keep the packaging, antistatic bags, cable ties etc. in pristine condition and return the goods exactly as received.

 

[loty1825]

I'm not realy fond of buying an expansion card because I'm trying to learn something. I could also buy a new network printer, a print server or say a Rasberry PI to configure it as a print server instead. But that's not the point. I'm trying to learn something here. And buying new stuff that will work doesn't give me a chance to learn how to make the existing setup work. So I'd rather keep learning with the current setup until I either make it work or learn that it's not possible for it to work.

 

[loty1825]

Some more googling suggests enabling unsafe interrupts in grub configuration file. But with scale there's no such file at the usual location. /etc/default/grub doesn't exist. There is /etc/default/grub.d directory that contains 4 configuration files. I tried editing the truenas.cfg but after reboot all my edits are gone so I guess this file gets overwritten at every system startup.

I've come across a command that is supposed to be used in TrueNAS scale instead of editing grub configuration files manually. midclt call system.advanced.update '{"kernel_extra_options": "firstoption=value secondoption=value thirdoption=value"}' is supposed to do the trick.

So to enable unsafe interrupts it should be formed like this:

midclt call system.advanced.update '{"kernel_extra_options": "vfio_iommu_type1.allow_unsafe_interrupts=1"}'

After running it I got this output:

{"id": 1, "consolemenu": true, "serialconsole": false, "serialport": "ttyS0", "s erialspeed": "9600", "powerdaemon": false, "swapondrive": 2, "overprovision": nu ll, "traceback": true, "advancedmode": false, "autotune": false, "debugkernel": false, "uploadcrash": true, "anonstats": true, "anonstats_token": "", "motd": "W elcome to TrueNAS", "boot_scrub": 7, "fqdn_syslog": false, "sed_user": "USER", " sysloglevel": "F_INFO", "syslogserver": "", "syslog_transport": "UDP", "kdump_en abled": false, "isolated_gpu_pci_ids": [], "kernel_extra_options": "vfio_iommu_t ype1.allow_unsafe_interrupts=1", "syslog_tls_certificate": null, "syslog_tls_cer tificate_authority": null, "consolemsg": false}

Rebooted the machine and thought to myself here goes nothing. But after restart the VM did auto start. I still have to do some testing to confirm but seems this solved it.

 

[loty1825]

After several TureNAS reboots sending test page to the printer worked every time without any user interaction. So if you want to passthrough an entire PCI USB controller this solution works.

But I'm still interested in how to do it by passing through just a single USB port. Just for the learning sake. So if someone has some idea I'd really appreciate it. Would it be possible using udev rules maybe? Like to write a set of rules that would tell my 0000:00:1d.7 USB hub should always be assigned hub number 7 for example? If I could force that, the virtual machine may be able to auto start even with USB passthrough.