unable to isolate gpu in TrueNAS 23.12

[truenasinterested]

Hi,
i have an Asrock E3C246D4U2-2T with Xeon D 2174G (Quicksync-capable P630). BIOS 2.40. Onboard adapter is enabled, iGPU Multi Monitor support activated.
P630 and Aspeed-2500 appear in the system (lspci+Webgui of TrueNAS). i915 kernel module is automatically loaded. P630 is IOMMU group2 (is alone in this group).
If i try to isolate the iGPU, i get error "0000:00:02.0 gpu pci slot(s) consists of devices which cannot be isolated from host.". This will also happen if i unload i915 kernel module. VT-d is enabled.

Project is to pass the Quicksync unit to a VM in TrueNAS for being used in hardware transcoding.

Can you please advise how to fix the issue?

 

[morganL]

It might be a known bug.. due for fixing in 23.10.1.

[NAS-124416] - iXsystems TrueNAS Jira

ixsystems.atlassian.net

 

[bcat]

Hmm, I somehow missed this thread, but I have a very similar issue (on similar hardware).

It might be a known bug.. due for fixing in 23.10.1.

[NAS-124416] - iXsystems TrueNAS Jira

ixsystems.atlassian.net

I'm not sure that bug is directly related. That issue seems to be about failed VM startup (due to the VM not automatically passing through the entire IOMMU group). But the issue that I (and the OP) are having doesn't show up under "Virtualization", but rather under "System Settings > Advanced > Isolated GPU Device(s)". The "GPU pci slot(s) consists of devices which cannot be isolated from host" error comes up before even trying to create a VM. :)

 

[truenasinterested]

It might be a known bug.. due for fixing in 23.10.1.

[NAS-124416] - iXsystems TrueNAS Jira

ixsystems.atlassian.net

Hi,

i thought the bug is fixed but i still have the same behavior. :(

 

[bcat]

Hi,

i thought the bug is fixed but i still have the same behavior. :(

Which behavior?

1. Unable to configure/start a VM with a GPU passed through.
2. Unable to configure an "isolated" GPU (regardless of whether or not VMs with passed-through GPUs start up successfully).

These are distinct bugs, and the first (should be) fixed in 23.10.1, but the second is not.

 

[southern.cross]

Hello,

I have TrueNAS Scale 23.10.1 running as a VM in XCP-ng 8.2. I have successfully passed my NVIDIA GTX-1060 gpu through to the TrueNAS Scale VM using the directions found here.

In the console of the TrueNAS Scale VM I get the proper/expected output from running nvidia-smi. I've taken that as a successful passthrough from XCP-ng to TrueNAS Scale.

Now, I'm trying to make the GPU available to the Jellyfin app installed on TrueNAS Scale VM. I need this for the transcoding capabilities. In the TrueNAS Scale WebUI I go to SYSTEM SETTINGS --> ADVANCED --> ISOLATED GPU DEVICE(S) --> CONFIGURE to configure the gpus. As you can see in this image... TrueNAS Scale is not accepting my GPU as isolated.

It is a second confirmation that the GPU is getting passed from XCP-ng to TrueNAS Scale but I can't get it passed along to the Jellyfin app.

Some details about my system:
- CPU: Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz
- GPU: GP106 [GeForce GTX 1060 6GB]
- Motherboard: MSI, (,MS-7A71,)
- BIOS: American Megatrends Inc. (1.50)

I had to configure both the Intel Virtualization and VT-d features in the BIOS to get the XCP-ng passthrough to work (as expected).

I also found the following guidance about properly configuring BIOS for passthrough/isolation. I did find a setting related 4G decoding that I enabled in my BIOS but the problem persisted. The rest of the recommended settings are not available in my BIOS.

Anybody see what I've missed here?

Thanks in advance.

 

[bcat]

Hello,

I have TrueNAS Scale 23.10.1 running as a VM in XCP-ng 8.2. I have successfully passed my NVIDIA GTX-1060 gpu through to the TrueNAS Scale VM using the directions found here.

In the console of the TrueNAS Scale VM I get the proper/expected output from running nvidia-smi. I've taken that as a successful passthrough from XCP-ng to TrueNAS Scale.

Now, I'm trying to make the GPU available to the Jellyfin app installed on TrueNAS Scale VM. I need this for the transcoding capabilities. In the TrueNAS Scale WebUI I go to SYSTEM SETTINGS --> ADVANCED --> ISOLATED GPU DEVICE(S) --> CONFIGURE to configure the gpus. As you can see in this image... TrueNAS Scale is not accepting my GPU as isolated.

If you just want to expose the GPU to a TrueNAS app (i.e., a container running in TrueNAS via Kubernetes), you don't need to "isolate" the GPU. In fact, you should not do so. "Isolating" a GPU is done when you want to pass a GPU through from TrueNAS to a VM running inside TrueNAS.

 

[southern.cross]

@bcat
Thanks for your feedback.

I ended up deleting my TrueNAS Scales VM and reinstalling (and subsequently reinstalling Jellyfin) and that allowed me to get the GPU passed all the way through to the Jellyfin container.

Now the problem is I can't get a single video to playback... transcoding is failing: https://forum.jellyfin.org/t-playback-error-client-isn-t-compatible?pid=12851#pid12851

 

[southern.cross]

Playback problem is now solved!

Thanks to @Xstar97TheNoob from the TrueChart support group,

Discord - A New Way to Chat with Friends & Communities

Discord is the easiest way to communicate over voice, video, and text. Chat, hang out, and stay close with your friends and communities.

discord.com

I had to stop using ACLs. It's documented here:

Dataset and Share Setup Guide | TrueCharts

This guide provides the recommended setup for dataset permissions, shares and application data storage.

truecharts.org