Use intel gpu hardware encoding in plex kubernetes deployment

anael_l

Cadet
Joined
Dec 24, 2020
Messages
2
Since I have heard of Trunas Scale, I have been really enthusiastic about it.
I have good knowledge of kubernetes, and so I wanted to mess around with it.
I ended up installing it on my desktop, and now it's my NAS. (I know it's in beta..)

On version 20.10, I deployed plex as a Docker container. I passed through volumes, for media, and my intel gpu for hardware transcoding.
Now that version 20.12 is out, I switch the Plex deployment to the helm chart provided by IX.
It was easy to use the same volumes as for the Docker container. However, there is no mention for the intel gpu passthrough.
If you use an Nvidia GPU, it's easier because they already included the Nvidia device plugin.
Just deploy plex with the GUI, then go on the command line as root and run:
midclt call -job chart.release.update 'plex' '{"values":"gpuConfiguration": {"nvidia.com/gpu": 1}}'

For intel, we first need to install the device plugin for k8s.
On the truenas terminal, as root, you first need to gain control of the k8s cluster with kubectl
Code:
alias kubectl="k3s kubectl"
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml


Now you can run kubectl get node to see your node status

To deploy the device plugin, you should only have to run :
Code:
kubectl apply -k https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/gpu_plugin?ref=v0.19.0


However, it didn't work for me. I had to clone the git repo, and then apply the config

Code:
git clone https://github.com/intel/intel-device-plugins-for-kubernetes.git
cd intel-device-plugins-for-kubernetes
git checkout v0.19.0
kubectl apply -n kube-system apply -k deployments/gpu_plugin


Wait a minute or two, and check if it has work:
kubectl get nodes -o=jsonpath="{range .items[*]}{.metadata.name}{'\n'}{' i915: '}{.status.allocatable.gpu\.intel\.com/i915}{'\n'}"

You should see: "i915: 1"

Now, we need to pass the device to our Plex deployment. Unfortunately, we can't do it through the Helm chart, as the value we need to pass ({"values":{"gpuConfiguration": {"gpu.intel.com/i915": 1}}}) is not supported.
We therefore need to directly modify the k8s deployment

kubectl -n ix-plex edit deployment plex

Now find the line ressources: {} in spec:{containers:{..}} and replace it by:
Code:
resources:                                                             
  limits:                                                             
    gpu.intel.com/i915: "1"


The pod will restart, and your plex instance should now use hardware decoding and reduce your CPU usage !
Don't forget that you need a Plex Pass for this feature, and you need to enable it in the settings.

The deployment configuration will most likely be overwritten by any new update. But this is just a temporary solution, as I'm sure the IXsystems team will include this feature in future version.
 

ornias

Wizard
Joined
Mar 6, 2020
Messages
1,458
Automatic setting up of the Intel device plugin should be included next version.
See: https://jira.ixsystems.com/browse/NAS-108532

That being said:
Next version should also have this setting in the UI, the gpu UI elementis already in the file used to dynamically generate the install dialog options, but for the time being those are still hardcoded elsewhere. I expect next version having some of the dynamic dialog options, as Kris said he understood it was hindering community app development.
 
Last edited:

srtviperjr

Dabbler
Joined
Jan 30, 2013
Messages
20
for some reason the nvidia transcoding isn't working for me on a vanilla installation of 20.12. The command
midclt call -job chart.release.update 'plex' '{"values":"gpuConfiguration": {"nvidia.com/gpu": 1}}'
returns errors below:

Code:
truenas# midclt call -job chart.release.update 'plex' '{"values":"gpuConfiguration": {"nvidia.com/gpu": 1}}'
Status: (none)
Total Progress: [________________________________________] 0.00%
[chart_release_update] A dict was expected
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 372, in run
    await self.future
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 408, in __run_body
    rv = await self.method(*([self] + args))
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1020, in nf
    args, kwargs = clean_and_validate_args(args, kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 978, in clean_and_validate_args
    value = attr.clean(args[args_index + i])
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 639, in clean
    raise Error(self.name, 'A dict was expected')
middlewared.schema.Error: [chart_release_update] A dict was expected


Unclear if this is required, or it was resolved in the latest update.

There is a compatible video card present and the drivers are loading. Where should I look for errors?
 

anael_l

Cadet
Joined
Dec 24, 2020
Messages
2
for some reason the nvidia transcoding isn't working for me on a vanilla installation of 20.12. The command returns errors below:

Code:
truenas# midclt call -job chart.release.update 'plex' '{"values":"gpuConfiguration": {"nvidia.com/gpu": 1}}'
Status: (none)
Total Progress: [________________________________________] 0.00%
[chart_release_update] A dict was expected
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 372, in run
    await self.future
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 408, in __run_body
    rv = await self.method(*([self] + args))
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1020, in nf
    args, kwargs = clean_and_validate_args(args, kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 978, in clean_and_validate_args
    value = attr.clean(args[args_index + i])
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 639, in clean
    raise Error(self.name, 'A dict was expected')
middlewared.schema.Error: [chart_release_update] A dict was expected


Unclear if this is required, or it was resolved in the latest update.

There is a compatible video card present and the drivers are loading. Where should I look for errors?
My bad, I posted the wrong command

midclt call -job chart.release.update 'plex' '{"values":{"gpuConfiguration": {"nvidia.com/gpu": 1}}}'

See https://www.truenas.com/docs/hub/scale/dev-notes/#using-kubernetes for more info
 

srtviperjr

Dabbler
Joined
Jan 30, 2013
Messages
20
Thanks! Command runs now, but still no love for the HW encoding. Did you do anything else to get it to run?
 

dalnew

Dabbler
Joined
Dec 9, 2020
Messages
26
I believe there is an issue with current nvidia drivers and the Linux 5.9 kernel. I think it was working in TrueNAS 20.10, which was on kernel 5.8, but now that it's upgraded to 5.9 the current nvidia drivers packaged aren't compatible for transcoding. The current driver version is 450.80.02 but I believe you need the latest 455.x drivers Nvidia just released in Nov which added 5.9 compatibility. Unfortunately those drivers are still in the experimental repo for Debian and I have had a heck of a time trying to get them installed, which to be fair, might be a bad idea anyway :)

 

srtviperjr

Dabbler
Joined
Jan 30, 2013
Messages
20
thanks! for the time being I just loaded ubuntu and am doing it there, when scale gets released I will give it another shot.
 

Saberwolf

Explorer
Joined
Feb 7, 2021
Messages
63
yes i am currently running Turenas Scale nightly the only things you need to get Nvidia GPU hardware acceleration work is running 4 commands

apt update && apt install nvidia-cuda-dev nvidia-cuda-toolkit && [ ! -f /dev/nvidia-uvm ] && /usr/bin/nvidia-modprobe -c0 -u

midclt call -job chart.release.update 'plex' '{"values":{"gpuConfiguration": {"nvidia.com/gpu": 1}}}'

midclt call -job chart.release.update 'plex' '{"values":{"extraEnv": {"NVIDIA_DRIVER_CAPABILITIES": "all"}}}'

midclt call -job chart.release.redeploy 'plex'

we are making the assumption that you named your app plex if not change any where you see plex to your name of app

the first command installs missing cuda drivers and toolkit
the second line updates app to support gpu
the thrid line updates plex to add env varible
the forth line redeploys plex app all with out restarting server
 

bodly

Dabbler
Joined
Jan 1, 2021
Messages
16
yes i am currently running Turenas Scale nightly the only things you need to get Nvidia GPU hardware acceleration work is running 4 commands

apt update && apt install nvidia-cuda-dev nvidia-cuda-toolkit && [ ! -f /dev/nvidia-uvm ] && /usr/bin/nvidia-modprobe -c0 -u

midclt call -job chart.release.update 'plex' '{"values":{"gpuConfiguration": {"nvidia.com/gpu": 1}}}'

midclt call -job chart.release.update 'plex' '{"values":{"extraEnv": {"NVIDIA_DRIVER_CAPABILITIES": "all"}}}'

midclt call -job chart.release.redeploy 'plex'

we are making the assumption that you named your app plex if not change any where you see plex to your name of app

the first command installs missing cuda drivers and toolkit
the second line updates app to support gpu
the thrid line updates plex to add env varible
the forth line redeploys plex app all with out restarting server
Ahh, thanks. I was working on this for a couple of hours and had decided to just wait as my cpu can handle software transcoding of anything less than 4K with no issues. But It's nice to be able to use my Nvidia GPU. Currently transcoding a 57 Mbps 4K movie down to 2Mbps 720p and the cpu is basically idle.

Code:
gojira# nvidia-smi                    
Mon Feb  8 07:43:17 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Quadro P620         Off  | 00000000:83:00.0 Off |                  N/A |
| 40%   52C    P0    N/A /  N/A |   1374MiB /  2000MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A   3524419      C   ...diaserver/Plex Transcoder     1372MiB |
+-----------------------------------------------------------------------------+


I did not need to install nvidia-cuda-dev or nvidia-cuda-toolkit.
I'm on TrueNAS-SCALE-21.02-MASTER-20210205-112916
 

dalnew

Dabbler
Joined
Dec 9, 2020
Messages
26
I think all the drivers were updated to the latest 460 a couple weeks back, which bring 4.9/4.10 kernel compatibility.
 

Saberwolf

Explorer
Joined
Feb 7, 2021
Messages
63
i have not tryed to do it with out cuda because Plex reqires it under the latest pull from plexinc
look at my trouble ticket
 

Saberwolf

Explorer
Joined
Feb 7, 2021
Messages
63
https://jira.ixsystems.com/browse/NAS-109192

i will however update to the latest nightly and test without updating packages but if you run the most current plex docker image i am sure it is required let me test further i will get back to you on that.
 

Saberwolf

Explorer
Joined
Feb 7, 2021
Messages
63
i also like to use nvtop to look at gpu usage i prefer it
 

Saberwolf

Explorer
Joined
Feb 7, 2021
Messages
63
i can confirm that running the TrueNAS-SCALE-21.02-MASTER-20210208-115115 running plex app using the latest docker image pull from plexinc hardware acceleration will not work without cuda or toolkit you might have an app that has the built in requirements. but under the latest pull you can expect hw will not work. verify this buy loading plex up with latest tag and check the console search for code check warning, errors, info, and you will see what i see see attached img
1612808202810.png

1612808246316.png

1612808282303.png

1612808390877.png

the issue lies in the loaded .so files in ldconfig app is calling for them and can not find or have access to them but if you build the image with them in there and you have exposed the GPU to the container if you have to required apps installed it will work but the host system should be the one to contain all driver's and toolkits for gpu support in Docker i have added support to to proxmox witch is where i learned a lot about nvidia and how to get it into LXC or Docker k8s is a different ball all in itself it uses both api level resource allocation and docker you must meet all docker requirements and k8s before you can get it to work

what i look for is i do an
Code:
truenas# ls -la /dev/nvidia*
crw-rw-rw- 1 root root 195, 254 Feb  8 10:04 /dev/nvidia-modeset
crw-rw-rw- 1 root root 195,   0 Feb  8 10:04 /dev/nvidia0
crw-rw-rw- 1 root root 195, 255 Feb  8 10:04 /dev/nvidiactl

/dev/nvidia-caps:
total 0
drwxr-xr-x  2 root root     80 Feb  8 10:04 .
drwxr-xr-x 21 root root   4860 Feb  8 10:18 ..
cr--------  1 root root 245, 1 Feb  8 10:04 nvidia-cap1
cr--r--r--  1 root root 245, 2 Feb  8 10:04 nvidia-cap2
truenas#

the above is without cuda support added to host system below will be what is in the plex app itself

Code:
truenas# kubectl get -n ix-plex2 pods
NAME                     READY   STATUS                     RESTARTS   AGE
plex2-56f7749f76-zv5mh   0/1     UnexpectedAdmissionError   0          24h
plex2-66c494f884-79gjf   0/1     UnexpectedAdmissionError   0          19h
plex2-66c494f884-w64b2   1/1     Running                    0          23m
truenas# kubectl exec -n ix-plex2
truenas# kubectl exec -n ix-plex2 --stdin --tty plex2-66c494f884-w64b2 -- /bin/bash
root@truenas:/# ls -la /dev/nvidia*
crw-rw-rw- 1 root root 195, 254 Feb  8 18:04 /dev/nvidia-modeset
crw-rw-rw- 1 root root 195,   0 Feb  8 18:04 /dev/nvidia0
crw-rw-rw- 1 root root 195, 255 Feb  8 18:04 /dev/nvidiactl
root@truenas:/#


now i will add cuda support to host os with cuda toolkit you will notice nothing will change on the host system until you running the command to find nvidia-uvm witch is CUDA support it will also not be exposed to the app until your restart or redeploy it
Code:
[ ! -f /dev/nvidia-uvm ] && /usr/bin/nvidia-modprobe -c0 -u


more to come waiting for those two packages to install and i have doc's appointment
 

Saberwolf

Explorer
Joined
Feb 7, 2021
Messages
63
Code:
Processing triggers for glx-alternative-nvidia (1.2.0) ...
update-alternatives: warning: forcing reinstallation of alternative /usr/lib/nvidia because link group glx is broken
Processing triggers for libc-bin (2.31-9) ...
Processing triggers for initramfs-tools (0.139) ...
update-initramfs: Generating /boot/initrd.img-5.10.0-1-amd64
cryptsetup: ERROR: Couldn't resolve device
    boot-pool/ROOT/21.02-MASTER-20210208-115115
cryptsetup: WARNING: Couldn't determine root device
cryptsetup: WARNING: target 'md126' not found in /etc/crypttab
I: The initramfs will attempt to resume from /dev/dm-15
I: (/dev/mapper/md126)
I: Set the RESUME variable to override this.
truenas# ls -la /dev/nvidia*
crw-rw-rw- 1 root root 195, 254 Feb  8 10:04 /dev/nvidia-modeset
crw-rw-rw- 1 root root 195,   0 Feb  8 10:04 /dev/nvidia0
crw-rw-rw- 1 root root 195, 255 Feb  8 10:04 /dev/nvidiactl

/dev/nvidia-caps:
total 0
drwxr-xr-x  2 root root     80 Feb  8 10:04 .
drwxr-xr-x 21 root root   4860 Feb  8 10:18 ..
cr--------  1 root root 245, 1 Feb  8 10:04 nvidia-cap1
cr--r--r--  1 root root 245, 2 Feb  8 10:04 nvidia-cap2
truenas# [ ! -f /dev/nvidia-uvm ] && /usr/bin/nvidia-modprobe -c0 -u
truenas# ls -la /dev/nvidia*
crw-rw-rw- 1 root root 195, 254 Feb  8 10:04 /dev/nvidia-modeset
crw-rw-rw- 1 root root 241,   0 Feb  8 10:58 /dev/nvidia-uvm
crw-rw-rw- 1 root root 241,   1 Feb  8 10:58 /dev/nvidia-uvm-tools
crw-rw-rw- 1 root root 195,   0 Feb  8 10:04 /dev/nvidia0
crw-rw-rw- 1 root root 195, 255 Feb  8 10:04 /dev/nvidiactl

/dev/nvidia-caps:
total 0
drwxr-xr-x  2 root root     80 Feb  8 10:04 .
drwxr-xr-x 21 root root   4900 Feb  8 10:58 ..
cr--------  1 root root 245, 1 Feb  8 10:04 nvidia-cap1
cr--r--r--  1 root root 245, 2 Feb  8 10:04 nvidia-cap2
truenas#


the above just after install of cuda and tool kit you will see i ran my command to find folder and to run nvidia-modset and it added folders to /dev location

Code:
truenas# kubectl exec -n ix-plex2 --stdin --tty plex2-66c494f884-w64b2 -- /bin/bash
root@truenas:/# ls -la /dev/nvidia*
crw-rw-rw- 1 root root 195, 254 Feb  8 18:04 /dev/nvidia-modeset
crw-rw-rw- 1 root root 195,   0 Feb  8 18:04 /dev/nvidia0
crw-rw-rw- 1 root root 195, 255 Feb  8 18:04 /dev/nvidiactl
root@truenas:/#


is what the app looks like before restart.
after restart what app looks like and HW acceleration is working
Code:
Last login: Mon Feb  8 11:54:57 2021 from 192.168.x.x

        TrueNAS (c) 2009-2021, iXsystems, Inc.
        All rights reserved.
        TrueNAS code is released under the modified BSD license with some
        files copyrighted by (c) iXsystems, Inc.

        For more information, documentation, help or support, go here:
        http://truenas.com
Welcome to TrueNAS
truenas# k3s kubectl get -n ix-plex2 pods
NAME                     READY   STATUS                     RESTARTS   AGE
plex2-56f7749f76-zv5mh   0/1     UnexpectedAdmissionError   0          27h
plex2-66c494f884-79gjf   0/1     UnexpectedAdmissionError   0          23h
plex2-66c494f884-7gzhz   1/1     Running                    0          3h10m
truenas# k3s kubectl exec -n ix-plex2 --stdin --tty plex2-66c494f884-7gzhz -- /bin/bash
root@truenas:/# ls -la /dev/nvidia*
crw-rw-rw- 1 root root 195, 254 Feb  8 18:04 /dev/nvidia-modeset
crw-rw-rw- 1 root root 241,   0 Feb  8 18:58 /dev/nvidia-uvm
crw-rw-rw- 1 root root 241,   1 Feb  8 18:58 /dev/nvidia-uvm-tools
crw-rw-rw- 1 root root 195,   0 Feb  8 18:04 /dev/nvidia0
crw-rw-rw- 1 root root 195, 255 Feb  8 18:04 /dev/nvidiactl
root@truenas:/#
 

dalnew

Dabbler
Joined
Dec 9, 2020
Messages
26
So should these nvidia cuda packages be installed and configured by default or is this something we should be installing as an end user? It looks like a bunch of the nvidia stuff is installed by default, except for this cuda related stuff
 

bodly

Dabbler
Joined
Jan 1, 2021
Messages
16
As I said earlier, I did not have to install the packages. But I did have to do the modprobe to have /dev/nvidia-uvm* show up.

gojira# ls -l /dev/nvidia*
crw-rw-rw- 1 root root 195, 254 Feb 5 14:03 /dev/nvidia-modeset
crw-rw-rw- 1 root root 237, 0 Feb 8 05:52 /dev/nvidia-uvm
crw-rw-rw- 1 root root 237, 1 Feb 8 05:52 /dev/nvidia-uvm-tools
crw-rw-rw- 1 root root 195, 0 Feb 5 13:31 /dev/nvidia0
crw-rw-rw- 1 root root 195, 255 Feb 5 13:31 /dev/nvidiactl

/dev/nvidia-caps:
total 0
cr-------- 1 root root 240, 1 Feb 5 13:31 nvidia-cap1
cr--r--r-- 1 root root 240, 2 Feb 5 13:31 nvidia-cap2

I just pulled in the latest plexinc/pms-docker image and redeployed and verified that it is using the gpu to transcode.
 
Top