GPU not showing up on apps setting

[nero10578]

My hardware specs:
Intel Xeon E5 2679 V4
Asus X99-E-WS 10G
256GB Samsung ECC REG DDR4
EVGA RTX 3060 12GB (for VM)
EVGA GTX 1660 Super (for apps)

I have a GTX 1660 Super that I use for the jellyfin truecharts apps for hw acceleration. This has been working fine until it suddenly didn't. No idea what happened but today jellyfin didn't want to transcode then when I tried to restart the app it says GPU not available. So I opened the settings and saw that the GPU is missing so I clicked the cross to delete the entry and tried to add it again but nothing shows up there.

I tried restarting the machine and also tried restarting the machine after removing the other isolated GPU which was my RTX 3060 that I passthrough to a VM. It still does not show up on the apps settings on the GPU section. I even just upgraded to the latest TrueNAS-SCALE-22.12.3.3 just in case that might help but nope.

The GTX 1660 Super shows up on the GPU isolation setting and also on nvidia-smi. So it seems to be working just fine, not to mention the RTX 3060 that works fine in the VM as well. I literally have no idea what caused this since I am away from home and did not mess with the hardware and software at all. Any help would be appreciated. Thank you.

 

[omega552003]

I have the exact same issue, something with the nvidia-container-plugin-daemon is crashing.

 

[omega552003]

I have the exact same issue, something with the nvidia-container-plugin-daemon is crashing.

GPU no longer available to apps

So I was able to use my Geforce GTX 1080 for Plex(TRUENAS charts) for trans-coding without any issues a couple months ago. I recently noticed that the GPU is nolonger being used by Plex and when I looked at GPU resources its blank, no selection menu. I've check and the host has the drivers...

www.truenas.com

 

[Djubei]

Had same issue with 1050ti. Plex app stuck on deploying until i turn off gpu settings. Simple reboot solved the problem:)

 

[omega552003]

The Apps

Had same issue with 1050ti. Plex app stuck on deploying until i turn off gpu settings. Simple reboot solved the problem:)

The Apps deploy fine, the GPU is just not available/usable to the apps.

 

[sos_nz]

Do you run any VMs? The GPU can be available to either the apps *or* VM's, but not both. I have an NVIDIA 1060. It was used for transcoding in Jellyfin, but then I set up a VM, and it's now stuck in limbo - unavailable to either VMs or apps. I can't recall the exact errors in shell, but there's a thread on here somewhere about it.

There's a setting under System Settings > Advanced "Isolated GPU Device(s):NVIDIA Corporation GP106 [GeForce GTX 1060 6GB]", which you can set on or off, to allow the GPU to be used for apps.

In my case, I haven't been able to 'unlock' my GPU, so I can no longer choose the GPU in Jellyfin's app settings, or in-app settings.

EDIT: here's a recent thread discussing the issue:

NVIDIA drivers won't load

My system won't load NVIDIA drivers. TrueNAS-SCALE-22.12.0 switched trains from CORE->ANGELFISH->BLUEFIN if that matters lspci |grep NVIDIA 01:00.0 VGA compatible controller: NVIDIA Corporation GP107GL [Quadro P400] (rev a1) 01:00.1 Audio device: NVIDIA Corporation GP107GL High Definition Audio...

www.truenas.com

 

[nero10578]

Do you run any VMs? The GPU can be available to either the apps *or* VM's, but not both. I have an NVIDIA 1060. It was used for transcoding in Jellyfin, but then I set up a VM, and it's now stuck in limbo - unavailable to either VMs or apps. I can't recall the exact errors in shell, but there's a thread on here somewhere about it.

There's a setting under System Settings > Advanced "Isolated GPU Device(s):NVIDIA Corporation GP106 [GeForce GTX 1060 6GB]", which you can set on or off, to allow the GPU to be used for apps.

In my case, I haven't been able to 'unlock' my GPU, so I can no longer choose the GPU in Jellyfin's app settings, or in-app settings.

EDIT: here's a recent thread discussing the issue:

NVIDIA drivers won't load

My system won't load NVIDIA drivers. TrueNAS-SCALE-22.12.0 switched trains from CORE->ANGELFISH->BLUEFIN if that matters lspci |grep NVIDIA 01:00.0 VGA compatible controller: NVIDIA Corporation GP107GL [Quadro P400] (rev a1) 01:00.1 Audio device: NVIDIA Corporation GP107GL High Definition Audio...

www.truenas.com

Still having the same issue. I have not put the 1660S in the isolated GPUs only my 3060 so not sure what's going on.

 

[Trevor68]

Thats easy, GPU's got kind of broke when 12.2.2 was released. My GTX1650 has only ever said this since.

root@truenas[~]# nvidia-smi
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

 

[sos_nz]

root@truenas[~]# nvidia-smi
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

Mine too - but it's a 1060.

 

[Trevor68]

Yeah but it works fine on my VM, so not really complaining, just a bit of a mystery.

 

[nero10578]

Thats easy, GPU's got kind of broke when 12.2.2 was released. My GTX1650 has only ever said this since.

root@truenas[~]# nvidia-smi
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

My GPU shows up in nvidia-smi just fine so I don't think that is the issue for me.

 

[Trevor68]

Mine always did too before this update

 

[sos_nz]

Mine always did too before this update

Bah! Raise my hopes up, just to get the same error message with 'nvidia-smi'

 

[killseeker]

I'm only new to TrueNas Scale over the past week or so, as I am looking to move from xpenology to TrueNas Scale, primarily for Plex GPU transcoding with my Plexpass.

I started off installing TrueNas Scale 22.12.3.3 Bluefin - However Plex would not recognise the GPU at all, even though it was correctly presented through to the Plex container and I could run nvidia-smi from within the container (no processes would ever start on the GPU, nor would I see transcoding (hw) in my plex dashboard windows.

I've since tried our the previous version of TrueNas Scale 22.02.4 Angelfish and succesfully enabled Plex with GPU transcoding. What is up with Bluefin and how do we go about getting someone to fix this bug so its possible to use the lastest version of TrueNas scale?

The only real difference I can see if the version of the nvidia driver between Angelfish(470.103.01) and Bluefin(515.65.01).

 

[nero10578]

I'm only new to TrueNas Scale over the past week or so, as I am looking to move from xpenology to TrueNas Scale, primarily for Plex GPU transcoding with my Plexpass.

I started off installing TrueNas Scale 22.12.3.3 Bluefin - However Plex would not recognise the GPU at all, even though it was correctly presented through to the Plex container and I could run nvidia-smi from within the container (no processes would ever start on the GPU, nor would I see transcoding (hw) in my plex dashboard windows.

I've since tried our the previous version of TrueNas Scale 22.02.4 Angelfish and succesfully enabled Plex with GPU transcoding. What is up with Bluefin and how do we go about getting someone to fix this bug so its possible to use the lastest version of TrueNas scale?

The only real difference I can see if the version of the nvidia driver between Angelfish(470.103.01) and Bluefin(515.65.01).

So it does seem like the "fix" is to downgrade truenas to angelfish to not have GPU issues. I chose to just run my GPU accelerated apps in docker containers in a ubuntu VM within truenas scale. Hopefully they won't bork virtualization either.

 

[scorpdevil]

Thats easy, GPU's got kind of broke when 12.2.2 was released. My GTX1650 has only ever said this since.

root@truenas[~]# nvidia-smi
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

Hi, I am experiencing the same issues. What do you mean by “make sure” you have the latest NVIDIA drivers?

If I understand correctly, apt-get is not allowed and running the update from TrueNas is not detecting
new updates.

Any comments would be highly appreciated.

Thank you.

 

[flammen]

I experience the same issue, no GPU available in the app diaogues.
I upgraded from a GTX660 to a RTX2060 super, thinking that the driver is not supported any more, which solved the problem for a while. However, now the problem is back after one of the latest updates.
I did not use the GPU for any VM, it is not isolated, and I have a second GPU for truenas itself.
nvidia-smi reports driver support:

Code:

admin@truenas[~]$ nvidia-smi
Mon Sep 18 19:06:51 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 515.65.01    Driver Version: 515.65.01    CUDA Version: 11.7     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:65:00.0 Off |                  N/A |
| 30%   34C    P0    41W / 184W |      0MiB /  8192MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                              
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Any help is much appreciated!

 

[sos_nz]

it is not isolated

Don't you have to isolate it to have it available / show up for use in apps? At least that's how it seems to work on my set up (with a single GPU).

 

[flammen]

I isolated the GPU, however, it still does not show up in the app's install or edit dialogue. It does not matter whether I try to edit or fresh install a new app.
Example from Jellyfin attached, I also checked the Truecharts version and Plex, they all look the same.
Are there specific bios settings needed for using a PCIe GPU with apps?

 

[bullrees]

Have this same issue and it seems many others also. Has there been any fix found?

TrueNAS-SCALE-22.12.3.3
Official Truenas: Plex 1.32.6.7557_1.7.56
GPU1: NVIDIA P2000
GPU2: iGPU UHD770 (13900 CPU)
I tried first with the native truenas scale apps but couldn't get it working. I installed Mint Linux as a VM in truenas and used the GPU isolation to pass the P2000 thru. Installed docker and portainer on the VM. Installed latest Plex from Linuxserver.io and all worked fine. P2000 was available and worked to transcode in plex. This is not really the setup I wanted with the additional pain of dataset permissions, accounts, etc. So I went back to the native apps route where I am stuck.

I can see both GPUs as an option in the Isolation Menu. I made sure they are not isolated.
link
Here is the output from nvidea-smi
link

When I try to add/edit the truenas plex (official) container/pod there are the words "GPU Configuration" but no way to add anything. No add button or way to edit.
Here is what it looks like:
link
Appreciate any pointers/assistance