Nvidia GPU not appearing for use with SCALE

[frogcrush]

I installed a Quadro 2000 (2014 model) GPU into my main server today running TrueNAS-SCALE-22.02.1 and expected it to show up in the GPU section of Apps so that I can use it with iSpy (performance is terrible in iSpy and I am hoping this will help). But, no matter what app container I try to use it never appears in the menu.

I confirmed it shows up with lspci:

root@truenas[~]# lspci | grep VGA
03:00.0 VGA compatible controller: NVIDIA Corporation GF106GL [Quadro 2000] (rev a1)
0b:00.0 VGA compatible controller: Matrox Electronics Systems Ltd. G200eR2

And I tried running nvidia-smi like I've seen posted before but it errors out:

NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

Any tips on getting the GPU to appear for use with Apps?

 

[Ppriorfl]

Did you ever get any help on this? My 3080 isn't able to be used with scale and Nvidia-smi returns: Unable to determine the device handle for GPU 0000:21:00.0: Unknown Error

 

[MountainMan]

I *think* the problem with his Quadro 2000 is that it's Fermi based so it would want the legacy 390.x driver. Scale has 460.91.03 currently.

With your 3080 I swear I saw a message like that come up in a machine learning forum I visit, and IIRC was solved with a newer driver. Right now Scale is a little behind currently as 470.x is available for Debian 11. Might be worth submitting a request in JIRA for updated drivers if there isn't one already.

 

[NugentS]

I have exactly your setup

root@NewNAS[~]# lspci | grep VGA
0c:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 30)
84:00.0 VGA compatible controller: NVIDIA Corporation GP106GL [Quadro P2000] (rev a1)

But when I type nvidia-smi I get
root@NewNAS[~]# nvidia-smi Wed Jun 29 08:11:03 2022 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 460.91.03 Driver Version: 460.91.03 CUDA Version: 11.2 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 Quadro P2000 Off | 00000000:84:00.0 Off | N/A | | 44% 30C P8 4W / 75W | 0MiB / 5059MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+

 

[MountainMan]

I have exactly your setup

You actually have a Quadro P2000 (GP106GL) where the op has the older Quadro 2000 (GF106GL). That threw me at first also, as I'm running a P2000 too.

 

[NugentS]

Oh yes. Oops - GP106GL vs GF106GL

 

[frogcrush]

Yes - it threw me for a loop when I got the card too :D

So am I stuck on the legacy driver (and therefor SOL) then?

 

[Ukhupacha]

Yes - it threw me for a loop when I got the card too :D

So am I stuck on the legacy driver (and therefor SOL) then?

I was wondering, did you manage to change the drivers to Legacy drivers?

 

[RealMoto]

Yes - it threw me for a loop when I got the card too :D

So am I stuck on the legacy driver (and therefor SOL) then?

Were you able to use your GPU? I have the same issue and GPU.

 

[Ashkaan]

Same issue here. I can't seem to get anywhere. I can't even tell which GPU I have at this point.

 

[Ashkaan]

Just realized that my card (T600) is TOO new for the 460 driver set. Can we please get updated drivers for TrueNAS?

I'm using TrueNAS SCALE 22.12-RC.1.

 

[Ashkaan]

Wait, nevermind. It's not too new. Why isn't it showing up?

Appendix A. Supported NVIDIA GPU Products

 

[Sasquatch]

P400 here,same problem, @Ashkaan did you figure it out?


root@truenas[~]# lspci | grep VGA 01:00.0 VGA compatible controller: NVIDIA Corporation GP107GL [Quadro P400] (rev a1) 06:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 21)


root@truenas[~]# nvidia-smi NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

 

[Ashkaan]

No, I didn't. I ended up pivoting completing and setting up a VM with my GPU as passthrough.

It's funny that it allows passthrough to a VM, but I can't use it in Kubernetes.

 

[Sasquatch]

Passtrough using VM config menu didn't work for me, had to add PCI device...
With VM it doesn't matter if there are drivers for the GPU, just passtrough capable CPU and GPU

 

[jhavron]

Passtrough using VM config menu didn't work for me, had to add PCI device...
With VM it doesn't matter if there are drivers for the GPU, just passtrough capable CPU and GPU

having trouble also..... i had it then ...... its gone

 

[aramis0221]

Same problem here running on a Quadro 6000. Does anybody knows how to activate the nvidia legacy drivers or something related??

 

[morganL]

Same problem here running on a Quadro 6000. Does anybody knows how to activate the nvidia legacy drivers or something related??

With all of these comments, it is important to say which version of SCALE you are running.

 

[aramis0221]

With all of these comments, it is important to say which version of SCALE you are running.

Truenas Scale 22.12.1

 

[jhavron]

Truenas Scale 22.12.1

OS Version:TrueNAS-SCALE-22.12.0
Product:PowerEdge T110 II
Model:Intel(R) Xeon(R) CPU E31220 @ 3.10GHz
Memory:16 GiB

root@truenasscale[~]# lspci | grep VGA
01:00.0 VGA compatible controller: NVIDIA Corporation GK208 [GeForce GT 630 Rev. 2] (rev a1)
05:03.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200eW WPCM450 (rev 0a)
root@truenasscale[~]#


Working on upgrading to TrueNAS-22.12.1 right now though.