Truenas scale, Plex, RTX 3050, drivers, keylase driver patch

[gsrcrxsi]

So there's a lot of points here that I would like some clarification on.

Some background. I've been running a Truenas Core system since it was called Freenas the original system was installed with Freenas 11, and upgraded along the way to truenas 12 (latest). I initially had Plex running on the Truenas system via the plugin, but wanted to utilize nvidia GPU transcoding and it seemed that GPU support in VMs or jails would never come (is it still a dead feature for core? will never be implemented?). So I spun off Plex into it's own system on totally separate hardware. Currently my plex server is running on a system with Ubuntu 20.04 server on bare metal. also with a GTX 1660, nvidia drivers installed, along with the keylase driver patch to unlock transcode streams.

So now that Scale has proper GPU support in containers and VMs being Linux-based, Id like try to move everything back to one box. I have truenas scale 22.02.0 (release) installed, and I have successfully migrated my operational VMs over as a test (using clonezilla to image then re-image), and I will soon start migrating my other media automation apps (transmission/*'arrs/etc). But I'm unsure how to proceed with Plex or if it's possible to do what I want.

First, drivers for the GPU. the new Scale system will have an RTX 3050 for transcoding. Nvidia lists that this GPU requires the 510 drivers. but from what I've read in other posts, it seems that Scale has 460 drivers included? is that still correct? is there any way to update them? Similarly, is it possible to have different nvidia drivers used inside the Plex container than the host has?

Next, the keylase patch. this patch is easy to apply on bare metal, you just install the drivers and run the patch. their github also lists directions for a docker install. but as I understand it, the Plex app is Kubernetes, not Docker. so has anyone got the keylase patch to work with the default Plex app?

I "think" these issues could be resolved by running Plex in a VM, but I would really prefer to have plex in a container and ideally in the default plex app provided by Truenas.

 

[Kris Moore]

We indeed have the 460 drivers included now, but expect to update those in nightly images later. Here's list of supported GPU's with that driver:

Linux x64 (AMD64/EM64T) Display Driver | 460.91.03 | Linux 64-bit | NVIDIA

Download the English (US) Linux x64 (AMD64/EM64T) Display Driver for Linux 64-bit systems. Released 2021.7.20

www.nvidia.com

 

[gsrcrxsi]

ok, but can it be updated? can a container run a different driver version? I would like to use the 3050 GPU.

 

[Kris Moore]

No, the package cannot be independently updated. I expect we will be updating the package soon in our nightly images though, and if you want to toss us a ticket on https://jira.ixsystems.com we can track if we should backport the update to 22.02.1 or next update.

 

[gsrcrxsi]

added NAS-114979

I listed it as a "suggestion" as "bug" didnt seem to fit, and it wouldnt let me create a new feature option.

 

[Kris Moore]

That'll work. I tossed a vote on it also, I suspect it'll happen in the relatively near future.

 

[gsrcrxsi]

so how about applying the driver patch within the plex app? possible? or would I need to just create a custom docker and do it that way?

 

[Kris Moore]

so how about applying the driver patch within the plex app? possible? or would I need to just create a custom docker and do it that way?

I don't know if thats possible. There is a kernel module which is loaded on the host-side.

 

[dirtyfreebooter]

FWIW, i applied the keylapse patch to RC.2, directly to the host and it worked and nothing appeared broken. i assumed i would have to re-apply it after each upgrade, but its takes 2 seconds. before i upgraded to release, i switched CPUs to one with Intel iGPU with quick sync, so didn't get a chance to see, but upgrade itself went fine, i.e. applying the patch to the host didn't break the upgrade process.

 

[gsrcrxsi]

FWIW, i applied the keylapse patch to RC.2, directly to the host and it worked and nothing appeared broken. i assumed i would have to re-apply it after each upgrade, but its takes 2 seconds. before i upgraded to release, i switched CPUs to one with Intel iGPU with quick sync, so didn't get a chance to see, but upgrade itself went fine, i.e. applying the patch to the host didn't break the upgrade process.

That’s a good point. Thanks. I didn’t think about applying the patch to the host. Did the effects propagate into the container/app? And you could have more than 3 transcodes on the GPU in the app?

 

[dirtyfreebooter]

That’s a good point. Thanks. I didn’t think about applying the patch to the host. Did the effects propagate into the container/app? And you could have more than 3 transcodes on the GPU in the app?

yea i got up 9 streams with a 1650 gtx in plex container. 10 streams resulted in gpu out of memory error. the 1650 i choose only has a 75w tdp and does not require pcie power so i felt 9 was great given those constraints.

https://www.amazon.com/dp/B08CYDH8MD/ref=cm_sw_r_cp_apip_IcDQeaz7wPzkR

 

[crkinard]

RTX 3050 for transcoding

Good lord man. All you need is a Quadro P2000. Single slot. No extra power connectors. Runs cooler. Multi decodes. Supports HEVC (H.265). Fully supported in SCALE currently (using one right now).

Hell, you can use a K2000 but no HEVC support. These things are like $50-$100 on eBay. Only need HEVC support if you're backing up 4k Blu-Ray with something like MakeMKV.

I really would not use any GTX card for HW decode unless you already have one laying around. EVEN THEN I would replace it as soon as the money is available.

Had issues trying to get HW decode working in the official plex docker. Using linxsever.io's. Basically just worked.

 

[gsrcrxsi]

A P2000 is the same price as a 3050 lol and the 3050 has a lot more VRAM and better encoder. The Turing (7th gen) NVENC encoder is much better than the pascal encoder for efficiency and picture quality. I’m using a GTX1660 now. And it’s also better than the P2000. The same Turing encoder and more VRAM. And when I bought it it was about half the cost of a P2000. VRAM basically limits how many transcodes you can do. More so than the encoder itself. Especially for 4k transcodes.

And I’d like to have the better hardware for future proofing when Plex adds more codecs to their transcoder. Transcoding is not just for backing up files. Plex transcodes on the fly for anything that needs it (clients that don’t have a 4k tv, or someone watching from their phone). The move from a 1660 to a 3050 is really just for the extra 2GB VRAM. even though Ampere does have a better decoder as well, it’s not the main reason for the upgrade. Honestly I’d go for a 3060 for 12GB if they weren’t so expensive right now.

But this thread isn’t about hardware choice. It’s about getting the right features into TrueNAS given a set of hardware. Even if I stuck to a Quadro level card, there will be new cards. The RTX A2000 for example also needs newer drivers than what TrueNAS Scale has.

 

[dirtyfreebooter]

yea, it would be neat to see TrueNAS SCALE has some sort ability to get things like this into production faster, but i also understand from stability perspective, this is a big departure from Core/FreeBSD paradigm with regard to releases. And this in particular is not just updating software, its a kernel driver, so it could have more consequences system wide. that being said, it would be interesting for ixsystems to figure something awesome out. maybe even include multiple drivers and let it be selectable in the advanced settings, so users not needing bleeding edge can stay on a more stable driver.

 

[Ixian]

No judgement regarding keylase on my part but anyone who thinks Ix Systems is going to allow switching multiple kernels/drivers or tactically supporting tools like keylase is stone-cold crazy :)

They aren't building an Unraid alternative. Frankly you'd have better luck going with Unraid (official ZFS support is in development) since they do allow this kind of customization (although you have to go off-community to get keylase support).

 

[gsrcrxsi]

Suggesting Unraid is just silly lol. I’ve been using the IX FreeNAS/TrueNAS product since FreeNAS 11. The only feature I’ve been left wanting is GPU support in jails. Since that’ll never happen it seems, and since Scale supports GPUs better in both containers and VMs, it’s a logical transition. The whole reason I spun off Plex into a bare metal system was to get Nvidia GPU transcoding support. have a fairly large ZFS pool already that will be migrated over once I migrate the apps that are currently in TNcore Jails over to Scale Docker/Apps. And I’d like to simplify the setup and roll Plex back into the same physical box.

It seems that since the keylase patch works it’s a non-issue. They don’t have to actively support it for it to work, and it seems like that’s what’s happened. They would have to actively block it somehow. And there’s really no reason to do that. The keylase patch is a fairly simple and low-risk patch. It’s not on the level of “switching multiple kernels/drivers” it applies a very small patch to the existing driver.

 

[dirtyfreebooter]

yea, i get that all things have to be weighed by effort needed and whether it fits their product goals, but having the ability to select between a stable nvidia driver and say a "prerelease" one, so that folks can help test and possibly get unblocked until the next release isn't crazy. i don't mean they should have every nvidia driver available.

or do nothing, NAS is a NAS i guess. the main difference with TrueNAS SCALE vs say something like Proxmox which is also debian based, you can do all the regular debian/apt stuff on Proxmox, so you can often unblock yourself at your risk of making stuff not work. SCALE's design means that almost all things related to the base system have to be done for you. *shrug*

 

[gsrcrxsi]

well for now, my workaround is to just GPU passthrough into a VM.

I imaged my physical plex system, then restored the image into a VM. and even though the base TrueNAS SCALE system only recognizes the GPU as a generic nvidia device, it still allowed me to pass it through to a VM. inside the VM then, I can install the proper drivers and keylase patch just fine. the RTX 3050 is recognized by nvidia-smi and Plex is working as intended now. it might be nice to get it into an app whenever the base system is updated with newer drivers, but for now it's working well and I was able to take down my physical plex system and roll it back into the TrueNAS system, which was my ultimate goal from the beginning.

but it seems a little buggy. sometimes the VM wont start when the GPU is passed to it. no error, just unresponsive VM after start (no VNC and no SSH). a few reboots brings it back though.

 

[raudraido]

What OS are you using for GPU passthrough. I tried RTX 3050 with linux and Win and had no luck. TrueNas Scale will be very unresponsive, laggy and also VM with installed nvidia drivers becomes almost unusable.

well for now, my workaround is to just GPU passthrough into a VM.

I imaged my physical plex system, then restored the image into a VM. and even though the base TrueNAS SCALE system only recognizes the GPU as a generic nvidia device, it still allowed me to pass it through to a VM. inside the VM then, I can install the proper drivers and keylase patch just fine. the RTX 3050 is recognized by nvidia-smi and Plex is working as intended now. it might be nice to get it into an app whenever the base system is updated with newer drivers, but for now it's working well and I was able to take down my physical plex system and roll it back into the TrueNAS system, which was my ultimate goal from the beginning.

but it seems a little buggy. sometimes the VM wont start when the GPU is passed to it. no error, just unresponsive VM after start (no VNC and no SSH). a few reboots brings it back though.

 

[gsrcrxsi]

TrueNAS SCALE is doing the passthrough into it's own VM. I had a lot of crashing too, until I manually changed the board to run the slot at PCIe 3.0. even though the board only supports PCIe 3.0, it seems the card was still trying to negotiate 4.0. so I forced it. I've had this issue with other 30-series cards.