Can't use Quadro P1000 graphics card on apps - Lenovo S30/E5-2680v2/9211-8i

[dgotc]

Hello All,

I've searched and read all I could find on this issue and don't really have a solution, so I'm starting my own thread with my hardware. I have three apps (Jellyfin, iSpy and Code Project AI docker) that I'd like to use this Nvidia Quadro P1000 on for encoding/decoding. While the card is recognized by the OS, it is not being used. How do I get it to be used by the apps?

Hardware:
Motherboard: Lenovo S30
CPU: E5-2860v2
Ram: 96GB DDR3 ECC
HBA: Fujitsu 9211-8i (LSI) running in IT mode
Graphics Card 1: some old thing for VGA output so that the OS uses that and the P1000 is free for other uses
Graphics Card 2: Nvidia Quadro P1000
Boot Drive: Crucial M400 240GB SSD
Zpools: 8x HGST HUS724040ALE640

Software:
OS: TrueNAS-SCALE-23.10.0.1
Nvidia Driver: 535.54.03
Jellyfin: truecharts 15.0.21
iSpy: truecharts 6.0.21
Code Project AI: docker of whatever is latest stable as off 11/8/23(US)

So, when I run nvidia-smi it sees the card and reports model and temperature. In the app setup I add the GPU "Allocate 0 nvidia.com/gpu GPU" In the respective app I configure to use the GPU, then test and the quality of images or speed of transcodes does not change. Also nvidia-smi uses no processes and has no activity whatsoever.

 

[Netto Hikari]

Same problem here. I upgraded to Cobia yesterday after holding off for a while, as I usually do with "it just works applicances" like TrueNAS.

Unfortunately, after upgrading, I can no longer assign GPUs to applications, which is a bummer, as my family and I use this machine mainly for consuming content through Jellyfin on a daily basis.

The card is detected by the OS:

It can't be assigned to any application, however:

GPU support is enabled within kubernetes settings:

There seems to be a Jira ticket about this issue already, so we should probably just upvote it and wait: https://ixsystems.atlassian.net/bro...5MDhjNGQ5ODllMjZjMWRmYmQ3ZjE0MzEiLCJwIjoiaiJ9

 

[Kris Moore]

Does this only occur in TrueCharts apps or does it occur in official TrueNAS catalog ones as well?

 

[dgotc]

Does this only occur in TrueCharts apps or does it occur in official TrueNAS catalog ones as well?

I'm not sure this of this action and not sure if it relates to my original post. Thanks for looking into this, though. On my P1000 I can see nvidia.com/gpu 0, 1, 2, 3, 4, 5.

Anyways, I've managed to get just 1 app to use the gpu, which is far beyond anywhere I've gotten to yet. Now I have to document my configuration...

for truecharts iSpy the graphics are working with this:

 

[dgotc]

In an effort to use the GPU for several apps instead of just one I tried the following:
removed the "Select GPU" section from the truecharts setup (seen above) and added two Environment Variables:

NVIDIA_DRIVER_CAPABILITIES=all
NVIDIA_VISIBLE_DEVICES=all

This resulted in the following error:

 

[Netto Hikari]

not sure if it relates to my original post

I'm sorry, I didn't want to hijack your thread. To me, it seemed (and still seems) to be the exact same issue.

Does this only occur in TrueCharts apps or does it occur in official TrueNAS catalog ones as well?

It seems that this issue only occurs with TrueCharts apps. I didn't even think about TrueCharts being the issue here. I just checked Jellyfin from TrueNAS instead of TrueCharts and I now see the options to assign GPUs again.

I'll probably take my time and migrate to the TrueNAS version of the app, then.

 

[dgotc]

I'm sorry, I didn't want to hijack your thread. To me, it seemed (and still seems) to be the exact same issue.


It seems that this issue only occurs with TrueCharts apps. I didn't even think about TrueCharts being the issue here. I just checked Jellyfin from TrueNAS instead of TrueCharts and I now see the options to assign GPUs again.

I'll probably take my time and migrate to the TrueNAS version of the app, then.

No worries, if your idea is correct and using the truenas version solves the issue for you, please update this thread.

 

[dgotc]

Ok, I have an update. I'm running truecharts versions of iSpy and Jellyfin. When I select "Allocate 1 nvidia.com/gpu GPU" on both apps I find they're working. This wasn't the case with truenas 22, but 23 seems to have a change. I still can't get codeprojectai in a truenas docker to run, but this is progress.

 

[Mortorojo]

I am the one who posted the Jira ticket listed above. This does occur on both Truenas and TrueCharts apps. Odly enough I saw this thread and took another look at my setting. There is was 5 allocatable gpus, I assigned 1 to each app tested and all was working. Then rebooted truenas and the issue came back, 0 gpu's can be allocated. Not sure what triggered it to allow them, but something is very bugged at the moment. If you can assign them DO NOT RESTART.

 

[Mortorojo]

As just posted restarted, and gpu not allocatable. Just did a rollback on plex and there it was I could allocate the gpu, and not just for the app i rolled back, but all the apps. I assigned it and updated again, still working. I'd suggest trying this to fix the issue temporarily. But still DO NOT REBOOT TRUENAS.

 

[dgotc]

As just posted restarted, and gpu not allocatable. Just did a rollback on plex and there it was I could allocate the gpu, and not just for the app i rolled back, but all the apps. I assigned it and updated again, still working. I'd suggest trying this to fix the issue temporarily. But still DO NOT REBOOT TRUENAS.

Thanks Mortorojo,

I can confirm your findings. After a reboot the setting "Allocate 0 nividia.com/gpu GPU" is present and there are no other options in the drop-down like "Allocate 1..." or " ...2...", etc... With my installation I have to set both iSpy and Jellyfin to "Allocate1...". If I use 2 or 3 or 5 they don't load.

I did notice a few other behaviors:
1. Once rebooted and the above issue observed, if one manually stops each app that uses the gpu and manually starts them the "Allocate 1..." setting comes back. In fact all the 1 through 5 settings show up in the drop down.
2. When I have both apps using "Allocate 1..." and change one app to another number such as 2 or 3, the app I changed fails to start and the one running crashes (the pod fails). Setting both back to "Allocate 1..." and starting each app fixes it.

Moriorojo, what gpu do you have? Also, what apps are you using this with (charts, versions, etc...)? What are we doing differently that you can use the other Allocation numbers and I can only use 0 or 1 for all apps?

 

[Mortorojo]

Quadro P4000. All my apps are truecharts, but i tested with truenas native apps and had same results as mentioned above. Im only allocating 1 gpu on each app, using plex and tdarr. There response to my ticket claimes its an issue with the iommu groups and will be fixed in next update, 23.10.1, in Dec.

 

[Isma]

Quadro P4000. All my apps are truecharts, but i tested with truenas native apps and had same results as mentioned above. Im only allocating 1 gpu on each app, using plex and tdarr. There response to my ticket claimes its an issue with the iommu groups and will be fixed in next update, 23.10.1, in Dec.

perfect

Plex in Cobia TrueNAS-SCALE-23.10.0.1

Hello, I was looking for information about plex and hardware acceleration and even though I add the GPU to the app I have no way to select it to transcode, what am I doing wrong?

www.truenas.com

 

[dgotc]

Quadro P4000. All my apps are truecharts, but i tested with truenas native apps and had same results as mentioned above. Im only allocating 1 gpu on each app, using plex and tdarr. There response to my ticket claimes its an issue with the iommu groups and will be fixed in next update, 23.10.1, in Dec.

Awesome, thanks for the info. I'm running 23.10.0.1 and experiencing this issue, so we'll see what 23.10.1 does soon.

 

[dgotc]

Well, I have 23.10.1 installed. I installed a new custom code project ai container and it keeps crashing with the same error. It doesn't even allow me to allocate anything other than 0 GPU

 

[joace]

same here with latest 23.10.1, i don't have any crashing with apps, it just doesn't work, no process detected in nvidia-smi even with successful assignment, not sure other trick still needs to revealed.

 

[StartledApple]

Hi all, I'm also having this issue. Running the latest stable release of TruNAS SCALE when I try to allocate a GPU in the TruNAS version of Plex, I have no option other than 0. I am running a Ryzen 5 5600G on my system and I can allocate the integrated GPU to Plex, for what it's worth. Any solutions to allocation a P1000 (what I have) as the GPU?