Unable to isolate iGPU despite being in its own IOMMU group

bcat

bcat

Explorer
Joined
Oct 20, 2022
Messages
84
I have a NAS built on an ASRock Rock E3C246D4M-4L (full specs in my signature) with a Xeon E-2276G processor. I'm running the latest stable build of Cobia (23.10.0.1). This configuration leaves me with two GPUs: one from the AST2500 BMC and the other from the processor's iGPU.

The TrueNAS console displays on the BMC's GPU, and the VGA port on the motherboard mirrors that output. The iGPU itself has no video outputs. It's also in its own IOMMU group (Group 0) as verified via this script:

Code:
IOMMU Group 0:
        00:02.0 Display controller [0380]: Intel Corporation CoffeeLake-S GT2 [UHD Graphics P630] [8086:3e96]
IOMMU Group 1:
        00:00.0 Host bridge [0600]: Intel Corporation 8th Gen Core Processor Host Bridge/DRAM Registers [8086:3ec6] (rev 07)
IOMMU Group 2:
        00:01.0 PCI bridge [0604]: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 07)
        00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x8) [8086:1905] (rev 07)
        00:01.2 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x4) [8086:1909] (rev 07)
        01:00.0 Non-Volatile memory controller [0108]: Phison Electronics Corporation PS5013 E13 NVMe Controller [1987:5013] (rev 01)
        02:00.0 Non-Volatile memory controller [0108]: Seagate Technology PLC FireCuda 510 SSD [1bb1:5012] (rev 01)
        03:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 [144d:a808]
IOMMU Group 3:
        00:08.0 System peripheral [0880]: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model [8086:1911]
IOMMU Group 4:
        00:12.0 Signal processing controller [1180]: Intel Corporation Cannon Lake PCH Thermal Controller [8086:a379] (rev 10)
IOMMU Group 5:
        00:14.0 USB controller [0c03]: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller [8086:a36d] (rev 10)
        00:14.2 RAM memory [0500]: Intel Corporation Cannon Lake PCH Shared SRAM [8086:a36f] (rev 10)
IOMMU Group 6:
        00:15.0 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH Serial IO I2C Controller #0 [8086:a368] (rev 10)
        00:15.1 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH Serial IO I2C Controller #1 [8086:a369] (rev 10)
IOMMU Group 7:
        00:16.0 Communication controller [0780]: Intel Corporation Cannon Lake PCH HECI Controller [8086:a360] (rev 10)
        00:16.1 Communication controller [0780]: Intel Corporation Device [8086:a361] (rev 10)
        00:16.4 Communication controller [0780]: Intel Corporation Cannon Lake PCH HECI Controller #2 [8086:a364] (rev 10)
IOMMU Group 8:
        00:17.0 SATA controller [0106]: Intel Corporation Cannon Lake PCH SATA AHCI Controller [8086:a352] (rev 10)
IOMMU Group 9:
        00:1b.0 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #17 [8086:a340] (rev f0)
IOMMU Group 10:
        00:1b.4 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #21 [8086:a32c] (rev f0)
IOMMU Group 11:
        00:1c.0 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #1 [8086:a338] (rev f0)
IOMMU Group 12:
        00:1d.0 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #9 [8086:a330] (rev f0)
IOMMU Group 13:
        00:1d.1 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #10 [8086:a331] (rev f0)
IOMMU Group 14:
        00:1d.2 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #11 [8086:a332] (rev f0)
IOMMU Group 15:
        00:1d.3 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #12 [8086:a333] (rev f0)
IOMMU Group 16:
        00:1e.0 Communication controller [0780]: Intel Corporation Cannon Lake PCH Serial IO UART Host Controller [8086:a328] (rev 10)
IOMMU Group 17:
        00:1f.0 ISA bridge [0601]: Intel Corporation Cannon Point-LP LPC Controller [8086:a309] (rev 10)
        00:1f.4 SMBus [0c05]: Intel Corporation Cannon Lake PCH SMBus Controller [8086:a323] (rev 10)
        00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH SPI Controller [8086:a324] (rev 10)
        00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (7) I219-LM [8086:15bb] (rev 10)
IOMMU Group 18:
        05:00.0 Ethernet controller [0200]: Mellanox Technologies MT27500 Family [ConnectX-3] [15b3:1003]
IOMMU Group 19:
        07:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)
IOMMU Group 20:
        08:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)
IOMMU Group 21:
        09:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)
IOMMU Group 22:
        0a:00.0 PCI bridge [0604]: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge [1a03:1150] (rev 04)
        0b:00.0 VGA compatible controller [0300]: ASPEED Technology, Inc. ASPEED Graphics Family [1a03:2000] (rev 41)

The iGPU works fine for hardware transcoding on the TrueNAS host (e.g., with the Plex app), but now I'd like to pass it through to a VM (as I'm trying to migrate away from SCALE's k3s apps to a setup that's easier for me to maintain). As far as I can tell, this should work swimmingly, since the iGPU is the only device in its IOMMU group (which should be the ideal case).

However, when I try to "isolate" the iGPU in SCALE's advanced settings, I get this error: "0000:00:02.0 GPU pci slot(s) consists of devices which cannot be isolated from host."

Unfortunately, SCALE doesn't tell me what devices it thinks cannot be isolated, so this is quite hard to debug further. Any tips would be appreciated! (I'll likely file a bug in the Jira in a bit, but it's quite possible I'm making some silly mistake on my end, so I wanted to check first.)

Edit: Weirdly, the code in the middleware that ultimately results in this error being thrown doesn't seem to look at IOMMU groups at all! I'm admittedly quite new to hardware passthrough use cases, but that doesn't seem right to me. It seems to be disallowing isolation of GPU devices with certain PCIe siblings... but without regard to what IOMMU groups they're actually in. Maybe that's just a bug?
 
Last edited:
M

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,694
bcat said:
I have a NAS built on an ASRock Rock E3C246D4M-4L (full specs in my signature) with a Xeon E-2276G processor. I'm running the latest stable build of Cobia (23.10.0.1). This configuration leaves me with two GPUs: one from the AST2500 BMC and the other from the processor's iGPU.

The TrueNAS console displays on the BMC's GPU, and the VGA port on the motherboard mirrors that output. The iGPU itself has no video outputs. It's also in its own IOMMU group (Group 0) as verified via this script:

Code:
IOMMU Group 0:
        00:02.0 Display controller [0380]: Intel Corporation CoffeeLake-S GT2 [UHD Graphics P630] [8086:3e96]
IOMMU Group 1:
        00:00.0 Host bridge [0600]: Intel Corporation 8th Gen Core Processor Host Bridge/DRAM Registers [8086:3ec6] (rev 07)
IOMMU Group 2:
        00:01.0 PCI bridge [0604]: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 07)
        00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x8) [8086:1905] (rev 07)
        00:01.2 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x4) [8086:1909] (rev 07)
        01:00.0 Non-Volatile memory controller [0108]: Phison Electronics Corporation PS5013 E13 NVMe Controller [1987:5013] (rev 01)
        02:00.0 Non-Volatile memory controller [0108]: Seagate Technology PLC FireCuda 510 SSD [1bb1:5012] (rev 01)
        03:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 [144d:a808]
IOMMU Group 3:
        00:08.0 System peripheral [0880]: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model [8086:1911]
IOMMU Group 4:
        00:12.0 Signal processing controller [1180]: Intel Corporation Cannon Lake PCH Thermal Controller [8086:a379] (rev 10)
IOMMU Group 5:
        00:14.0 USB controller [0c03]: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller [8086:a36d] (rev 10)
        00:14.2 RAM memory [0500]: Intel Corporation Cannon Lake PCH Shared SRAM [8086:a36f] (rev 10)
IOMMU Group 6:
        00:15.0 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH Serial IO I2C Controller #0 [8086:a368] (rev 10)
        00:15.1 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH Serial IO I2C Controller #1 [8086:a369] (rev 10)
IOMMU Group 7:
        00:16.0 Communication controller [0780]: Intel Corporation Cannon Lake PCH HECI Controller [8086:a360] (rev 10)
        00:16.1 Communication controller [0780]: Intel Corporation Device [8086:a361] (rev 10)
        00:16.4 Communication controller [0780]: Intel Corporation Cannon Lake PCH HECI Controller #2 [8086:a364] (rev 10)
IOMMU Group 8:
        00:17.0 SATA controller [0106]: Intel Corporation Cannon Lake PCH SATA AHCI Controller [8086:a352] (rev 10)
IOMMU Group 9:
        00:1b.0 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #17 [8086:a340] (rev f0)
IOMMU Group 10:
        00:1b.4 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #21 [8086:a32c] (rev f0)
IOMMU Group 11:
        00:1c.0 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #1 [8086:a338] (rev f0)
IOMMU Group 12:
        00:1d.0 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #9 [8086:a330] (rev f0)
IOMMU Group 13:
        00:1d.1 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #10 [8086:a331] (rev f0)
IOMMU Group 14:
        00:1d.2 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #11 [8086:a332] (rev f0)
IOMMU Group 15:
        00:1d.3 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #12 [8086:a333] (rev f0)
IOMMU Group 16:
        00:1e.0 Communication controller [0780]: Intel Corporation Cannon Lake PCH Serial IO UART Host Controller [8086:a328] (rev 10)
IOMMU Group 17:
        00:1f.0 ISA bridge [0601]: Intel Corporation Cannon Point-LP LPC Controller [8086:a309] (rev 10)
        00:1f.4 SMBus [0c05]: Intel Corporation Cannon Lake PCH SMBus Controller [8086:a323] (rev 10)
        00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH SPI Controller [8086:a324] (rev 10)
        00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (7) I219-LM [8086:15bb] (rev 10)
IOMMU Group 18:
        05:00.0 Ethernet controller [0200]: Mellanox Technologies MT27500 Family [ConnectX-3] [15b3:1003]
IOMMU Group 19:
        07:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)
IOMMU Group 20:
        08:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)
IOMMU Group 21:
        09:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)
IOMMU Group 22:
        0a:00.0 PCI bridge [0604]: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge [1a03:1150] (rev 04)
        0b:00.0 VGA compatible controller [0300]: ASPEED Technology, Inc. ASPEED Graphics Family [1a03:2000] (rev 41)

The iGPU works fine for hardware transcoding on the TrueNAS host (e.g., with the Plex app), but now I'd like to pass it through to a VM (as I'm trying to migrate away from SCALE's k3s apps to a setup that's easier for me to maintain). As far as I can tell, this should work swimmingly, since the iGPU is the only device in its IOMMU group (which should be the ideal case).

However, when I try to "isolate" the iGPU in SCALE's advanced settings, I get this error: "0000:00:02.0 GPU pci slot(s) consists of devices which cannot be isolated from host."

Unfortunately, SCALE doesn't tell me what devices it thinks cannot be isolated, so this is quite hard to debug further. Any tips would be appreciated! (I'll likely file a bug in the Jira in a bit, but it's quite possible I'm making some silly mistake on my end, so I wanted to check first.)

Edit: Weirdly, the code in the middleware that ultimately results in this error being thrown doesn't seem to look at IOMMU groups at all! I'm admittedly quite new to hardware passthrough use cases, but that doesn't seem right to me. It seems to be disallowing isolation of GPU devices with certain PCIe siblings... but without regard to what IOMMU groups they're actually in. Maybe that's just a bug?
Click to expand...

Do you know if it worked in Bluefin? (could you test)

I don't know whether there's a restriction on iGPUs.. its not a hardware config we actively turn into a product.
 
bcat

bcat

Explorer
Joined
Oct 20, 2022
Messages
84
morganL said:
Do you know if it worked in Bluefin? (could you test)

I don't know whether there's a restriction on iGPUs.. its not a hardware config we actively turn into a product.
Click to expand...
Unfortunately, I don't have a particularly easy way of testing. I have two TrueNAS boxes at home with similar setups (C246 motherboards with headless iGPUs and ASPEED BMCs), both of which show this issue... but they're also both actively in use and have been on Cobia for a bit now, so I don't really want to downgrade them (as they're doing NAS things really well).

I might be able to bodge together a temporary Bluefin install on a USB drive to test (as this shouldn't require anything more than a boot pool) if that's helpful, but probably not for a few days. (Sorry to be a pain; it's just something I'll have to fit around work, as well as finding an extended time when nobody needs anything running on the box.)

For what it's worth, I don't think this is really iGPU specific, so much as a consequence of the fact the "Is this GPU safe to isolate?" logic seems to use heuristics rather than looking at the actual IOMMU group structure itself. But I'll do what I can to dig deeper soon.
 
Last edited:
tprelog

tprelog

Patron
Joined
Mar 2, 2016
Messages
297
Maybe I can help. I've encountered the same error and am back on Bluefin now.

I'll try again before updating to Cobia using @bcat tip mentioned here.
 
L

Lipsum Ipsum

Dabbler
Joined
Aug 31, 2022
Messages
22
bcat said:
Edit: Weirdly, the code in the middleware that ultimately results in this error being thrown doesn't seem to look at IOMMU groups at all! I'm admittedly quite new to hardware passthrough use cases, but that doesn't seem right to me. It seems to be disallowing isolation of GPU devices with certain PCIe siblings... but without regard to what IOMMU groups they're actually in. Maybe that's just a bug?
Click to expand...
You're correct. It's not looking at IOMMU groups. It's just finding the parent device, and then all the decedents (not just siblings of the GPU) of that parent.

I was frustrated with this and ultimately worked around the issue with SR-IOV which is supported with Intel's 12th and 13th gen CPUs. Those CPUs support up to 7 virtual GPUs which can be passed in simply as a PCI device to the guest VM. It sidesteps the whole IOMMU group issue. I don't believe SR-IOV is supported by the earlier iGPUs like your 9th gen Xeon E-2276G, only 12th and 13th gen. 5th-10th gen CPUs with an iGPU supported a predecessor though, GVT-g. I don't have any of those CPUs to test, but perhaps something similar can be used to virtualize the GPU.
 
bcat

bcat

Explorer
Joined
Oct 20, 2022
Messages
84
Lipsum Ipsum said:
You're correct. It's not looking at IOMMU groups. It's just finding the parent device, and then all the decedents (not just siblings of the GPU) of that parent.
Click to expand...
Ah, thanks for confirming I'm not crazy (or at least not about this). :)

I'll try and file a bug in Jira about this tonight, time permitting.
Lipsum Ipsum said:
I was frustrated with this and ultimately worked around the issue with SR-IOV which is supported with Intel's 12th and 13th gen CPUs. Those CPUs support up to 7 virtual GPUs which can be passed in simply as a PCI device to the guest VM. It sidesteps the whole IOMMU group issue. I don't believe SR-IOV is supported by the earlier iGPUs like your 9th gen Xeon E-2276G, only 12th and 13th gen. 5th-10th gen CPUs with an iGPU supported a predecessor though, GVT-g. I don't have any of those CPUs to test, but perhaps something similar can be used to virtualize the GPU.
Click to expand...
Interesting. I can try this, but it sounds like there might be some performance issues with GVT-g. And it seems like a bit of complexity I'd rather not have in my setup, given that "vanilla" passthrough should work fine with the GPU in its own IOMMU group.

Site note: I've considered acquiring a cheap NUC with an Intel iGPU purely to run Plex. TrueNAS SCALE is wonderful as a NAS, but the VM/app story seems a bit incomplete still. But for a homelab, I really liked the idea of having one less piece of hardware taking up space and using power, so I'm still interested in getting passthrough working if possible. :)
 
bcat

bcat

Explorer
Joined
Oct 20, 2022
Messages
84
For what it's worth, I think the newly added get_pci_ids_for_gpu_isolation function returns the list of PCIe devices that "Isolated GPU Device(s)" should actually be checking, so it may not be too hard of a change to get this working properly.
 
HoneyBadger

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,112
Following this thread as I've been poking in the mix of GPU isolation behavior in the past including the situations of "multiple identical GPUs" and "GPU mdevs" - let me know the Jira ticket number when it's submitted if you get a chance.
 
L

Lipsum Ipsum

Dabbler
Joined
Aug 31, 2022
Messages
22
bcat said:
Site note: I've considered acquiring a cheap NUC with an Intel iGPU purely to run Plex. TrueNAS SCALE is wonderful as a NAS, but the VM/app story seems a bit incomplete still. But for a homelab, I really liked the idea of having one less piece of hardware taking up space and using power, so I'm still interested in getting passthrough working if possible. :)
Click to expand...
Absolutely agree with you on all accounts. I've not had any issues with my other "generic" VMs, just the VM that's handling HTPC-related duties due to the iGPU passthrough for transcoding.

bcat said:
For what it's worth, I think the newly added get_pci_ids_for_gpu_isolation function returns the list of PCIe devices that "Isolated GPU Device(s)" should actually be checking, so it may not be too hard of a change to get this working properly.
Click to expand...
Interesting. I didn't study it real closely, but it definitely looks to be closer in the right direction. Looking forward to it's release in December.
 
bcat

bcat

Explorer
Joined
Oct 20, 2022
Messages
84
tprelog

tprelog

Patron
Joined
Mar 2, 2016
Messages
297
morganL said:
Do you know if it worked in Bluefin? (could you test)
Click to expand...
It's not working on Bluefin either ( 22.12.4.2 ) - I have the same error while trying to isolate the iGPU:

Code:
0000:00:02.0 GPU pci slot(s) consists of devices which cannot be isolated from host.
 
tprelog

tprelog

Patron
Joined
Mar 2, 2016
Messages
297
morganL said:
Anyone keen to test, can use a SCALE Coibia nightly at https://download.truenas.com/
Click to expand...

Any chance you have a link to an update file rather than ISO images?

EDIT:

I found TrueNAS-SCALE-23.10-MASTER-20231116-033818.update here, but I guess this is not the file I'm looking for.

Trying to Install Manual Update File results in the following error:

Code:
[EFAULT] Unable to downgrade from 23.10.0.1 to 23.10-MASTER-20231116-033818


EDIT 2:

I reverted to Scale-22.12.4.2 and applied the manual update file from there - Unfortunately, I still cannot isolate the iGPU on SCALE-23.10-MASTER-20231116-033818. It is the same error as before.
 
Last edited:
bcat

bcat

Explorer
Joined
Oct 20, 2022
Messages
84
tprelog said:
I reverted to Scale-22.12.4.2 and applied the manual update file from there - Unfortunately, I still cannot isolate the iGPU on SCALE-23.10-MASTER-20231116-033818. It is the same error as before.
Click to expand...
Maybe I'm just being dense (I'm sorry if so), but I still think there are two separate issues here:

1. VM startup failing after GPU device passthrough configured in VE settings (https://ixsystems.atlassian.net/browse/NAS-124416, marked fixed).
2. Isolate GPU device failing in system advanced settings (https://ixsystems.atlassian.net/browse/NAS-125243). This issue doesn't happen on VM startup. It's a bug that shows up even with no VMs configured at all.

I asked about this in a Jira comment as well. (Of course it's the weekend, and this is a low-priority bug, so no rush on any responses. :) )
 
M

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,694
tprelog said:
Any chance you have a link to an update file rather than ISO images?

EDIT:

I found TrueNAS-SCALE-23.10-MASTER-20231116-033818.update here, but I guess this is not the file I'm looking for.

Trying to Install Manual Update File results in the following error:

Code:
[EFAULT] Unable to downgrade from 23.10.0.1 to 23.10-MASTER-20231116-033818


EDIT 2:

I reverted to Scale-22.12.4.2 and applied the manual update file from there - Unfortunately, I still cannot isolate the iGPU on SCALE-23.10-MASTER-20231116-033818. It is the same error as before.
Click to expand...

It may not be in the nightly yet...lets check.
github.com

NAS-124416 / 23.10.1 / Add an endpoint to retrieve PCI devices which are to be isolated for VM GPU isolation (by sonicaj) by bugclerk · Pull Request #12479 · truenas/middleware

This PR adds a new endpoint which UI should use when trying to determine which PCI devices are to be created for isolation when user selects a GPU to be attached to a VM. Original PR: #12472 Jira U...
github.com github.com
 
bcat

bcat

Explorer
Joined
Oct 20, 2022
Messages
84
So I just made an interesting discovery: GPU isolation isn't actually needed for GPU passthrough to work.

I can create a VM, pass through the iGPU PCIe device, and see it inside the VM just fine. :) When the VM starts, the GPU device on the host is switched from the i915 driver to the vfio-pci driver automatically (by libvirt, I assume).

On the TrueNAS SCALE host:

Code:
$ ls -l /sys/class/pci_bus/0000:00/device/0000:00:02.0/driver
lrwxrwxrwx 1 root root 0 Nov 18 22:16 /sys/class/pci_bus/0000:00/device/0000:00:02.0/driver -> ../../../bus/pci/drivers/vfio-pci

$ cat /sys/class/pci_bus/0000:00/device/0000:00:02.0/driver_override
vfio-pci

In the Debian VM:

Code:
$ ls -l /sys/class/drm/card1/device/driver
lrwxrwxrwx 1 root root 0 Nov 18 22:22 /sys/class/drm/card1/device/driver -> ../../../bus/pci/drivers/i915

It seems that all the "isolate" functionality in SCALE's advanced settings does is apply that driver override on boot. That still sounds like a good thing to do (to make extra sure nothing else will try to use it, e.g., TrueNAS SCALE apps); however, it doesn't actually seem necessary. Until the GPU isolation bug is fixed, I'll test out this setup further (passing through the iGPU without isolating on boot).

@morganL, I think this helps confirm that my issue is separate from NAS-124416. The reporter in that issue wasn't able to start a VM with the GPU passed through at all, but I can definitely start a VM with the device passed through. The only part that's broken for me is "isolating" the GPU on boot in advanced settings (i.e., applying the vfio-pci as soon as SCALE starts rather than waiting till the VM starts).
 
Last edited:
bcat

bcat

Explorer
Joined
Oct 20, 2022
Messages
84
Circling back on this, I can confirm after additional testing that GPU passthrough of this iGPU to a VM does indeed work fine, even without "isolating" it in SCALE's advanced settings. Here's Plex inside the VM using hardware transcoding successfully:

1700611184898.png

And here's intel_gpu_top inside the VM confirming the GPU is indeed being used:

1700611265768.png

And here's the PCIe device configuration in the VM:

1700611351576.png

So it appears to be only the "System Settings > Advanced > Isolated GPU Device(s)" UI that's broken:

1700611398139.png

As mentioned in my last post, the TrueNAS SCALE "isolate" functionality appears to be designed to apply a vfio-pci driver override for the "isolated" GPU on boot (presumably to prevent the NAS or SCALE apps from using it). It'd still be nice to get that "isolation" functionality working since the GPU device itself works fine with VM passthrough; however, this doesn't appear to be strictly necessary to use it in a VM.
 
bcat

bcat

Explorer
Joined
Oct 20, 2022
Messages
84
morganL said:
It may not be in the nightly yet...lets check.
github.com

NAS-124416 / 23.10.1 / Add an endpoint to retrieve PCI devices which are to be isolated for VM GPU isolation (by sonicaj) by bugclerk · Pull Request #12479 · truenas/middleware

This PR adds a new endpoint which UI should use when trying to determine which PCI devices are to be created for isolation when user selects a GPU to be attached to a VM. Original PR: #12472 Jira U...
github.com github.com
Click to expand...
I just tried using the latest Cobia nightly I could find (TrueNAS-SCALE-23.10-MASTER-20231130-034218), and unfortunately, the bug I reported in NAS-125243 is not actually fixed. I'm using the same system as before---"Ivy" from my signature---with a different boot medium since I didn't want to install a nightly on my "prod" boot-pool. :)

1701457599703.png

I know my bug report (NAS-125243) was closed out as a duplicate of NAS-124416, but I am almost certain they are different bugs that coincidentally both have to do with GPUs and IOMMU groups. I understand the similarity is suspicious, but if someone could take another look at the specific bug I reported in this thread when there's time, I'd honestly really appreciate it. (I don't have permission to reopen it myself.)

To be clear, unlike in the other GPU passthrough bug, I am in fact able to configure and start a VM passing through the iGPU by simply adding the iGPU as a PCI device in VM settings. (I've been running a VM with the iGPU passed through for several days.)

I just can't add the iGPU to the "isolated GPU" list in SCALE's advanced settings. That's what makes my bug distinct/not a duplicate.

I do think the functionality added to fix NAS-124416 could be adapted to fix the bug I'm seeing as well as well. Basically, this logic to identify "critical" GPUs doesn't currently use get_pci_ids_for_gpu_isolation. Instead, the "System Settings > Advanced > Isolated GPU PCI Ids" page still uses a heuristic to identify PCI devices associated with the GPU. That heuristic is overly broad, but looking at the containing IOMMU group (like the VM settings pages now do) ought to resolve this.

I also get that this is likely a low priority bug. I'm not opposed to trying to fix it myself (the scale-build repo on Github is quite nice, yay open source!), but it's not entirely how to do this cleanly since I'm not very familiar with the design of the middleware (e.g., what gets a service class vs. just utility functions, etc.), and I figured it wouldn't hurt to update here once more first. And thanks in any case!
 
Last edited:
M

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,694
bcat said:
I just tried using the latest Cobia nightly I could find (TrueNAS-SCALE-23.10-MASTER-20231130-034218), and unfortunately, the bug I reported in NAS-125243 is not actually fixed. I'm using the same system as before---"Ivy" from my signature---with a different boot medium since I didn't want to install a nightly on my "prod" boot-pool. :)

View attachment 73006

I know my bug report (NAS-125243) was closed out as a duplicate of NAS-124416, but I am almost certain they are different bugs that coincidentally both have to do with GPUs and IOMMU groups. I understand the similarity is suspicious, but if someone could take another look at the specific bug I reported in this thread when there's time, I'd honestly really appreciate it. (I don't have permission to reopen it myself.)

To be clear, unlike in the other GPU passthrough bug, I am in fact able to configure and start a VM passing through the iGPU by simply adding the iGPU as a PCI device in VM settings. (I've been running a VM with the iGPU passed through for several days.)

I just can't add the iGPU to the "isolated GPU" list in SCALE's advanced settings. That's what makes my bug distinct/not a duplicate.

I do think the functionality added to fix NAS-124416 could be adapted to fix the bug I'm seeing as well as well. Basically, this logic to identify "critical" GPUs doesn't currently use get_pci_ids_for_gpu_isolation. Instead, the "System Settings > Advanced > Isolated GPU PCI Ids" page still uses a heuristic to identify PCI devices associated with the GPU. That heuristic is overly broad, but looking at the containing IOMMU group (like the VM settings pages now do) ought to resolve this.

I also get that this is likely a low priority bug. I'm not opposed to trying to fix it myself (the scale-build repo on Github is quite nice, yay open source!), but it's not entirely how to do this cleanly since I'm not very familiar with the design of the middleware (e.g., what gets a service class vs. just utility functions, etc.), and I figured it wouldn't hurt to update here once more first. And thanks in any case!
Click to expand...

Thanks for doing the work and thinking through the problem.

Sorry for the extra work, but suggest a new bug report and refer to your original one with this extra information. If you can demonstrate a fix, it may help the team work out how to make it "official".
 
bcat

bcat

Explorer
Joined
Oct 20, 2022
Messages
84
morganL said:
Thanks for doing the work and thinking through the problem.

Sorry for the extra work, but suggest a new bug report and refer to your original one with this extra information. If you can demonstrate a fix, it may help the team work out how to make it "official".
Click to expand...
Sorry for the long delay. Re-filed as https://ixsystems.atlassian.net/browse/NAS-125751, which has (hopefully clearer) details on what appears to the problem and a possible suggested fix.
 
You must log in or register to reply here.

Similar threads

M
IOMMU group is not viable using pci passthrough for GPU in 1st or 2nd pice slot
Replies
4
Views
4K
homer27081990
homer27081990
impovich
Truenas on XCP-NG SATA controller passthrough
Replies
3
Views
3K
impovich
impovich
Daneel
Unable to SAVE selected isolated GPU, TrueNAS-SCALE-23.10.1
Replies
6
Views
1K
kashiwagi
kashiwagi
flotueur
Struggling to passthrough a VIA-based USB3 PCI card to a VM (Debian, for Home assistant)
Replies
11
Views
2K
flotueur
flotueur
G
SOLVED Dual NIC PCI Passthrough - user not reading docs
Replies
9
Views
3K
R.Heaton
R
Top