TrueNAS Scale 22.12.3 - virtualization PCI passthrough broken?

hege

Dabbler
Joined
May 16, 2023
Messages
13
I have upgraded my Scale from 22.12.2 to 22.12.3 and my virtualized pfSense failed to come up. Checking on the Virtualization tab on the UI, it says that the PCI device is not available (I'm using Supermicro X10SDV Xeon-D1541 motherboard with the built-in Intel i350 Gigabit PCI device). Running lspci as root, shows the PCI network card, so it's not a PCI device ID mismatch. When I tried to edit the PCI device in the VM settings, the PCI dropdown doesn't load at all. It seems like the middleware is not able to enumerate PCI devices. Tried an extra reboot, just in case, but it still didn't fix the problem..

This seems like a pretty major bug. For now, I've reverted back to 22.12.2. Anyone else seeing similar behavior?
 

hege

Dabbler
Joined
May 16, 2023
Messages
13
Some logs from the failed attempt:



Code:
[2023/06/13 09:12:10] (INFO) InterfaceService.sync_interface():1725 - eno2 is not in interfaces database
[2023/06/13 09:12:11] (ERROR) middlewared.start_on_boot():71 - Failed to start VM pfsense: [EFAULT] VM will not start as PCI Device: pci_0000_08_00_1 device(s) are not available.

...

[2023/06/13 09:16:17] (WARNING) middlewared._loop_monitor_thread():1754 - Task seems blocked:
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1246, in nf
    res = await f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/pci.py", line 109, in passthrough_device
    cp = await run(get_virsh_command_args() + ['nodedev-dumpxml', device], check=False)
  File "/usr/lib/python3/dist-packages/middlewared/utils/__init__.py", line 60, in run
    proc = await asyncio.create_subprocess_exec(*args, **kwargs)
  File "/usr/lib/python3.9/asyncio/subprocess.py", line 236, in create_subprocess_exec
    transport, protocol = await loop.subprocess_exec(
  ... + 10 lines below ...
[2023/06/13 09:18:06] (WARNING) application.call_method():239 - Exception while calling vm.device.passthrough_device_choices(*[])
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 211, in call_method
    self._send({
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 136, in _send
    serialized = json.dumps(data)
  File "/usr/lib/python3/dist-packages/middlewared/client/ejson.py", line 39, in dumps
    return json.dumps(obj, cls=JSONEncoder, **kwargs)
  File "/usr/lib/python3.9/json/__init__.py", line 234, in dumps
    return cls(
  File "/usr/lib/python3.9/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.9/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib/python3/dist-packages/middlewared/client/ejson.py", line 18, in default
    return super(JSONEncoder, self).default(obj)
  File "/usr/lib/python3.9/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type generator is not JSON serializable
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,112
Code:
[2023/06/13 09:12:11] (ERROR) middlewared.start_on_boot():71 - Failed to start VM pfsense: [EFAULT] VM will not start as PCI Device: pci_0000_08_00_1 device(s) are not available.

Can you post an lspci -nn to go along with this? Ideally under 22.12.3 but I wouldn't see any reason why enumeration would have changed from 22.12.2
 

hege

Dabbler
Joined
May 16, 2023
Messages
13
Can you post an lspci -nn to go along with this? Ideally under 22.12.3 but I wouldn't see any reason why enumeration would have changed from 22.12.2

Full output os lspci -nn on 22.12.2 below. I will grab the debug bundle from 22.12.3 as well and submit a bug report.

Code:
root@truenas[~]# lspci -nn
00:00.0 Host bridge [0600]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DMI2 [8086:6f00] (rev 03)
00:01.0 PCI bridge [0604]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 1 [8086:6f02] (rev 03)
00:02.0 PCI bridge [0604]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 2 [8086:6f04] (rev 03)
00:02.2 PCI bridge [0604]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 2 [8086:6f06] (rev 03)
00:03.0 PCI bridge [0604]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 3 [8086:6f08] (rev 03)
00:03.1 PCI bridge [0604]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 3 [8086:6f09] (rev 03)
00:03.2 PCI bridge [0604]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 3 [8086:6f0a] (rev 03)
00:03.3 PCI bridge [0604]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 3 [8086:6f0b] (rev 03)
00:05.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Map/VTd_Misc/System Management [8086:6f28] (rev 03)
00:05.1 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D IIO Hot Plug [8086:6f29] (rev 03)
00:05.2 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D IIO RAS/Control Status/Global Errors [8086:6f2a] (rev 03)
00:14.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI [8086:8c31] (rev 05)
00:16.0 Communication controller [0780]: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 [8086:8c3a] (rev 04)
00:16.1 Communication controller [0780]: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #2 [8086:8c3b] (rev 04)
00:1a.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 [8086:8c2d] (rev 05)
00:1c.0 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 [8086:8c10] (rev d5)
00:1c.4 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 [8086:8c18] (rev d5)
00:1d.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 [8086:8c26] (rev 05)
00:1f.0 ISA bridge [0601]: Intel Corporation C224 Series Chipset Family Server Standard SKU LPC Controller [8086:8c54] (rev 05)
00:1f.2 SATA controller [0106]: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] [8086:8c02] (rev 05)
00:1f.3 SMBus [0c05]: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller [8086:8c22] (rev 05)
00:1f.6 Signal processing controller [1180]: Intel Corporation 8 Series Chipset Family Thermal Management Controller [8086:8c24] (rev 05)
02:00.0 System peripheral [0880]: Intel Corporation Xeon Processor D Family QuickData Technology Register DMA Channel 0 [8086:6f50]
02:00.1 System peripheral [0880]: Intel Corporation Xeon Processor D Family QuickData Technology Register DMA Channel 1 [8086:6f51]
02:00.2 System peripheral [0880]: Intel Corporation Xeon Processor D Family QuickData Technology Register DMA Channel 2 [8086:6f52]
02:00.3 System peripheral [0880]: Intel Corporation Xeon Processor D Family QuickData Technology Register DMA Channel 3 [8086:6f53]
03:00.0 Unassigned class [ff00]: Intel Corporation 82599EB 10-Gigabit Dummy Function [8086:10a6]
06:00.0 Non-Volatile memory controller [0108]: Sandisk Corp WD Black 2018/SN750 / PC SN720 NVMe SSD [15b7:5002]
07:00.0 Non-Volatile memory controller [0108]: Sandisk Corp Device [15b7:501e]
08:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
08:00.1 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
0a:00.0 PCI bridge [0604]: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge [1a03:1150] (rev 03)
0b:00.0 VGA compatible controller [0300]: ASPEED Technology, Inc. ASPEED Graphics Family [1a03:2000] (rev 30)
ff:0b.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link 0/1 [8086:6f81] (rev 03)
ff:0b.1 Performance counters [1101]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link 0/1 [8086:6f36] (rev 03)
ff:0b.2 Performance counters [1101]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link 0/1 [8086:6f37] (rev 03)
ff:0b.3 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link Debug [8086:6f76] (rev 03)
ff:0c.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6fe0] (rev 03)
ff:0c.1 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6fe1] (rev 03)
ff:0c.2 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6fe2] (rev 03)
ff:0c.3 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6fe3] (rev 03)
ff:0c.4 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6fe4] (rev 03)
ff:0c.5 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6fe5] (rev 03)
ff:0c.6 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6fe6] (rev 03)
ff:0c.7 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6fe7] (rev 03)
ff:0f.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6ff8] (rev 03)
ff:0f.4 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6ffc] (rev 03)
ff:0f.5 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6ffd] (rev 03)
ff:0f.6 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6ffe] (rev 03)
ff:10.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R2PCIe Agent [8086:6f1d] (rev 03)
ff:10.1 Performance counters [1101]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R2PCIe Agent [8086:6f34] (rev 03)
ff:10.5 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Ubox [8086:6f1e] (rev 03)
ff:10.6 Performance counters [1101]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Ubox [8086:6f7d] (rev 03)
ff:10.7 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Ubox [8086:6f1f] (rev 03)
ff:12.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Home Agent 0 [8086:6fa0] (rev 03)
ff:12.1 Performance counters [1101]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Home Agent 0 [8086:6f30] (rev 03)
ff:13.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Target Address/Thermal/RAS [8086:6fa8] (rev 03)
ff:13.1 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Target Address/Thermal/RAS [8086:6f71] (rev 03)
ff:13.2 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder [8086:6faa] (rev 03)
ff:13.3 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder [8086:6fab] (rev 03)
ff:13.4 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder [8086:6fac] (rev 03)
ff:13.5 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder [8086:6fad] (rev 03)
ff:13.6 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Broadcast [8086:6fae] (rev 03)
ff:13.7 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Global Broadcast [8086:6faf] (rev 03)
ff:14.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 0 Thermal Control [8086:6fb0] (rev 03)
ff:14.1 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 1 Thermal Control [8086:6fb1] (rev 03)
ff:14.2 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 0 Error [8086:6fb2] (rev 03)
ff:14.3 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 1 Error [8086:6fb3] (rev 03)
ff:14.4 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface [8086:6fbc] (rev 03)
ff:14.5 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface [8086:6fbd] (rev 03)
ff:14.6 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface [8086:6fbe] (rev 03)
ff:14.7 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface [8086:6fbf] (rev 03)
ff:15.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 2 Thermal Control [8086:6fb4] (rev 03)
ff:15.1 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 3 Thermal Control [8086:6fb5] (rev 03)
ff:15.2 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 2 Error [8086:6fb6] (rev 03)
ff:15.3 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 3 Error [8086:6fb7] (rev 03)
ff:1e.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit [8086:6f98] (rev 03)
ff:1e.1 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit [8086:6f99] (rev 03)
ff:1e.2 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit [8086:6f9a] (rev 03)
ff:1e.3 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit [8086:6fc0] (rev 03)
ff:1e.4 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit [8086:6f9c] (rev 03)
ff:1f.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit [8086:6f88] (rev 03)
ff:1f.2 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit [8086:6f8a] (rev 03)
 

GabHav

Cadet
Joined
Nov 9, 2022
Messages
1
It´s broken for me too. I went back to 22.12.2
 
Last edited by a moderator:
Joined
Apr 10, 2023
Messages
1
I have upgraded my Scale from 22.12.2 to 22.12.3 and my virtualized pfSense failed to come up. Checking on the Virtualization tab on the UI, it says that the PCI device is not available (I'm using Supermicro X10SDV Xeon-D1541 motherboard with the built-in Intel i350 Gigabit PCI device). Running lspci as root, shows the PCI network card, so it's not a PCI device ID mismatch. When I tried to edit the PCI device in the VM settings, the PCI dropdown doesn't load at all. It seems like the middleware is not able to enumerate PCI devices. Tried an extra reboot, just in case, but it still didn't fix the problem..

This seems like a pretty major bug. For now, I've reverted back to 22.12.2. Anyone else seeing similar behavior?
Yep, me too. I'm trying to pass a USB ConeBee II ZigBee stick to my HomeAssistant VM using PCI passthrough. It stopped working as soon as I updated Scale to 22.12.3. It's working again now that I've reverted to 22.12.2.
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,112

Trevor68

Contributor
Joined
Oct 15, 2021
Messages
130
My GTX 1650 has disappeared altogether, was this expected too?

root@truenas[~]# nvidia-smi
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,112
My GTX 1650 has disappeared altogether, was this expected too?

root@truenas[~]# nvidia-smi
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.
Definitely not intended, no. Can you run lspci -v and show me the relevant section of your NVIDIA card, such as the example below:

Code:
0b:00.0 3D controller: NVIDIA Corporation GP104GL [Tesla P4] (rev a1)
        DeviceName: pciPassthru0
        Subsystem: NVIDIA Corporation GP104GL [Tesla P4]
        Physical Slot: 192
        Flags: bus master, fast devsel, latency 248, IRQ 19
        Memory at fc000000 (32-bit, non-prefetchable) [size=16M]
        Memory at d0000000 (64-bit, prefetchable) [size=256M]
        Memory at e4000000 (64-bit, prefetchable) [size=32M]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Capabilities: [100] Virtual Channel
        Capabilities: [128] Power Budgeting <?>
        Capabilities: [420] Advanced Error Reporting
        Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
        Capabilities: [900] Secondary PCI Express
        Kernel driver in use: nvidia
        Kernel modules: nouveau, nvidia_current_drm, nvidia_current
 

Trevor68

Contributor
Joined
Oct 15, 2021
Messages
130
Maybe just me, should the nvidia-smi command still work if the gpu is isolated? I have mine isolated to passthrough to VM. Though I had to remove all the pci devices to start the VM now.

Anyhow here is the relevant section.

01:00.0 VGA compatible controller: NVIDIA Corporation TU116 [GeForce GTX 1650] (rev a1) (prog-if 00 [VGA controller]) Subsystem: Gigabyte Technology Co., Ltd TU116 [GeForce GTX 1650] Flags: fast devsel, IRQ 11, IOMMU group 1 Memory at a4000000 (32-bit, non-prefetchable) [disabled] [size=16M] Memory at 90000000 (64-bit, prefetchable) [disabled] [size=256M] Memory at a0000000 (64-bit, prefetchable) [disabled] [size=32M] I/O ports at 4000 [disabled] [size=128] Expansion ROM at a5000000 [disabled] [size=512K] Capabilities: [60] Power Management version 3 Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [78] Express Legacy Endpoint, MSI 00 Capabilities: [100] Virtual Channel Capabilities: [250] Latency Tolerance Reporting Capabilities: [258] L1 PM Substates Capabilities: [128] Power Budgeting <?> Capabilities: [420] Advanced Error Reporting Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?> Capabilities: [900] Secondary PCI Express Capabilities: [bb0] Physical Resizable BAR Kernel driver in use: vfio-pci Kernel modules: nouveau, nvidia_current_drm, nvidia_current 01:00.1 Audio device: NVIDIA Corporation TU116 High Definition Audio Controller (rev a1) Subsystem: Gigabyte Technology Co., Ltd TU116 High Definition Audio Controller Flags: bus master, fast devsel, latency 0, IRQ 10, IOMMU group 1 Memory at a5080000 (32-bit, non-prefetchable) [size=16K] Capabilities: [60] Power Management version 3 Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [78] Express Endpoint, MSI 00 Capabilities: [100] Advanced Error Reporting Kernel driver in use: vfio-pci Kernel modules: snd_hda_intel 01:00.2 USB controller: NVIDIA Corporation TU116 USB 3.1 Host Controller (rev a1) (prog-if 30 [XHCI]) Subsystem: Gigabyte Technology Co., Ltd TU116 USB 3.1 Host Controller Flags: fast devsel, IRQ 141, IOMMU group 1 Memory at a2000000 (64-bit, prefetchable) [size=256K] Memory at a2040000 (64-bit, prefetchable) [size=64K] Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+ Capabilities: [78] Express Endpoint, MSI 00 Capabilities: [b4] Power Management version 3 Capabilities: [100] Advanced Error Reporting Kernel driver in use: xhci_hcd Kernel modules: xhci_pci 01:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU116 USB Type-C UCSI Controller (rev a1) Subsystem: Gigabyte Technology Co., Ltd TU116 USB Type-C UCSI Controller Flags: bus master, fast devsel, latency 0, IRQ 11, IOMMU group 1 Memory at a5084000 (32-bit, non-prefetchable) [size=4K] Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [78] Express Endpoint, MSI 00 Capabilities: [b4] Power Management version 3 Capabilities: [100] Advanced Error Reporting Kernel driver in use: vfio-pci
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,112
Maybe just me, should the nvidia-smi command still work if the gpu is isolated? I have mine isolated to passthrough to VM. Though I had to remove all the pci devices to start the VM now.
Yes, nvidia-smi will still poll the hardware even if it's being claimed by vfio-pci.

From the sounds of things (isolated to VM and won't properly start) this might be related to the PCI device marking that is being tracked in NAS-122456 - hopefully we can address this quickly for you!
 

Trevor68

Contributor
Joined
Oct 15, 2021
Messages
130
Thanks, I have tried adding either the Nvidia or Intel (gen8) GPU's to the vm directly, but the setting does not save. I will wait for the hotfix.
 

fudi87

Dabbler
Joined
Jul 4, 2022
Messages
11
Same issue here, Pci passtrough dosent work after update 12.12.3 Vm running windows 10 with a 1050 Ti card.

Setup: Supermicro X12SCA-F with intel 1290P 64Gb ecc.

root@truenas[~]# lspci
00:00.0 Host bridge: Intel Corporation Device 9b33 (rev 05)
00:01.0 PCI bridge: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) (rev 05)
00:08.0 System peripheral: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model
00:12.0 Signal processing controller: Intel Corporation Comet Lake PCH Thermal Controller
00:14.0 USB controller: Intel Corporation Comet Lake USB 3.1 xHCI Host Controller
00:14.2 RAM memory: Intel Corporation Comet Lake PCH Shared SRAM
00:16.0 Communication controller: Intel Corporation Comet Lake HECI Controller
00:16.3 Serial controller: Intel Corporation Device 06e3
00:17.0 SATA controller: Intel Corporation Device 06d2
00:1b.0 PCI bridge: Intel Corporation Comet Lake PCI Express Root Port #17 (rev f0)
00:1b.4 PCI bridge: Intel Corporation Comet Lake PCI Express Root Port #21 (rev f0)
00:1c.0 PCI bridge: Intel Corporation Device 06b8 (rev f0)
00:1c.5 PCI bridge: Intel Corporation Device 06bd (rev f0)
00:1c.7 PCI bridge: Intel Corporation Device 06bf (rev f0)
00:1d.0 PCI bridge: Intel Corporation Comet Lake PCI Express Root Port #9 (rev f0)
00:1f.0 ISA bridge: Intel Corporation Device 0697
00:1f.3 Audio device: Intel Corporation Comet Lake PCH cAVS
00:1f.4 SMBus: Intel Corporation Comet Lake PCH SMBus Controller
00:1f.5 Serial bus controller [0c80]: Intel Corporation Comet Lake PCH SPI Controller
00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (11) I219-LM
01:00.0 VGA compatible controller: NVIDIA Corporation GP107 [GeForce GTX 1050 Ti] (rev a1)
01:00.1 Audio device: NVIDIA Corporation GP107GL High Definition Audio Controller (rev a1)
02:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/980PRO
03:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/980PRO
05:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-LM (rev 03)
06:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 04)
07:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 41)
08:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd Device a809
root@truenas[~]#
root@truenas[~]#
 

Dasos

Dabbler
Joined
Aug 12, 2016
Messages
17
Upgraded my install just a few minutes ago and broke all my virtualization PCI passthrough, too. Rolled it back until I hear more as its needed for several key machines I run.
 

Trevor68

Contributor
Joined
Oct 15, 2021
Messages
130
any word on the hotfix? I see the jira ticket is closed.
 
Top