GPU passthrough in apps doesn't see GPU

[Flipou83]

First post in this forum. I haven't figured out how to add a signature with my hardware yet.

So first of all my hardware is a Dell PowerEdge R720. I've got a mix of 24 x 4GB and 8 GB sticks that amounts to a total of 120 GB ECC RAM DDR3 at 8500 speed which is 1066 Mhz I believe, not sure about that. It is running the latest BIOS 2.9.0 from last year. My HBA is a Dell H310. I've got a dual 10 Gbe + dual 1 Gbe daughter card installed and I have an NVIDIA quadro P400 in it.
The disks and pools are:w 2 x 16 TB Seagate Exos SATA in mirror with 2 x 1 TB 970 EVO NVMe as cache on the main pool with 2 x 500 GB Western Digital Black NVMe in mirror as application data. 2 x 500 GB 960 Samsung 960 PRO NVMe in mirror as pool used for temporary transcode/download folders. I do like mirrors...
My software is TrueNAS SCALE 21.06 beta.

So the problem now is that any apps don't see a gpu available (says 0). When I installed 21.06 from scratch I saw the GPU available and I used it in my emby from truecharts without any issue. After a reboot the app emby wouldn't start so I removed the gpu and then it started again. But I could never put the gpu back in there because it is not listed in the apps configuration.
I am able to use the gpu for a VM tho without a problem. I am clueless what to try. I tried asking around on reddit and Discord without any success that's why I am making this post as a last resort... Hoping that 21.08 fixes the issue I guess.
I tried excluding the gpu from truenas in the advanced config and moving to other pcie slots without success.

Any help would be really appreciated.

Thank you community :)

 

[morganL]

I'm not sure it's possible to use the GPU for VM and for the containers..... could it be that it stopped working for you after you tried it with the VM?

If you can accept the risk of Nightlies, you could update to see if 21.08 fixes have improved things.. or wait for 21.08 and then file a bug.

 

[Flipou83]

When I'm thinking about what happened that's actually exactly what happened. Apps with GPU passthrough worked fine, then I created a VM without using a GPU passthrough worked great. Then after that reboot I never could get the GPU detected again in the apps even tho I deleted the VM and have no working VM for a while now.

 

[morganL]

It seems like KVM think it= now owns this resource and won't give it up.
Power cycle or fresh software install might work..... otherwise, report it as a bug.

 

[Flipou83]

That would make total sense to me that could be the issue!
The problem I have occurred like 3 weeks ago. During that time I've rebooted several times with the issue persisting.
Is there a way to disable the VM (KVM) altogether?
By software install you mean reinstall SCALE and restore settings?
How stable is the nightly?

 

[morganL]

That would make total sense to me that could be the issue!
The problem I have occurred like 3 weeks ago. During that time I've rebooted several times with the issue persisting.
Is there a way to disable the VM (KVM) altogether?
By software install you mean reinstall SCALE and restore settings?
How stable is the nightly?

It's stable for data.... but there will be bugs. Later this month will be better.

 

[Flipou83]

Can't wait for it!

Keep up the good work team

Thanks for the feedback.