Plex stopped working (Kubernetes issue?)

[dealy663]

Hi Guys

My Plex install stopped working a couple of days ago. It kinda just hangs while deploying. After a bit of digging it seems like it can't find the pod. I'm definitely new to Kubernetes so not too familiar with troubleshooting this sort of thing. At least there are no pods to choose from when it prompts me for one on the shell menu for the Plex app.

I'm on TrueNAS-SCALE-22.02-RC.2, and I only have a data volume configured, no transcode or extra host path volumes.

I tried upgrading to: 1.25.6.5577_1.6.24 of Plex but that doesn't seem to have made any difference.
These events show up:
Scaled down replica set plexgps-5c7c9cb4c5 to 0

0/1 nodes are available: 1 node(s) had taint {ix-svc-start: }, that the pod didn't tolerate.

0/1 nodes are available: 1 node(s) had taint {ix-svc-start: }, that the pod didn't tolerate.

I get this info from Kubernetes:
root@TrueNAS[~]# k3s kubectl describe node ix-truenas
Name: ix-truenas
Roles: control-plane,master
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=ix-truenas
kubernetes.io/os=linux
node-role.kubernetes.io/control-plane=true
node-role.kubernetes.io/master=true
openebs.io/nodeid=ix-truenas
openebs.io/nodename=ix-truenas
Annotations: csi.volume.kubernetes.io/nodeid: {"zfs.csi.openebs.io":"ix-truenas"}
k3s.io/node-args:
["server","--flannel-backend","none","--disable","traefik,metrics-server,local-storage","--disable-kube-proxy","--disable-network-policy",...
k3s.io/node-config-hash: ANG5PHUHQSYCUD6IKAS5T3IF4XVUYKALZCPNZWJWBFBUIOWFJK5A====
k3s.io/node-env: {"K3S_DATA_DIR":"/mnt/StripeHD1/ix-applications/k3s/data/8233fb36dd068f36871f8aaeac65513a60bf051782a8d7fcf2890a672f17f2e0"}
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Sun, 19 Dec 2021 10:03:47 -0800
Taints: ix-svc-start:NoExecute
Unschedulable: false
Lease:
HolderIdentity: ix-truenas
AcquireTime: <unset>
RenewTime: Sat, 12 Mar 2022 08:56:47 -0800
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure False Sat, 12 Mar 2022 08:55:06 -0800 Sun, 19 Dec 2021 10:03:46 -0800 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Sat, 12 Mar 2022 08:55:06 -0800 Sun, 19 Dec 2021 10:03:46 -0800 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Sat, 12 Mar 2022 08:55:06 -0800 Sun, 19 Dec 2021 10:03:46 -0800 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Sat, 12 Mar 2022 08:55:06 -0800 Sat, 12 Mar 2022 08:33:38 -0800 KubeletReady kubelet is posting ready status. AppArmor enabled
Addresses:
InternalIP: 192.168.86.11
Hostname: ix-truenas
Capacity:
cpu: 16
ephemeral-storage: 93761152Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 32800580Ki
pods: 110
Allocatable:
cpu: 16
ephemeral-storage: 91210848595
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 32800580Ki
pods: 110
System Info:
Machine ID: 2568b4482d8241babdeb526c719d26c2
System UUID: 2e4f2079-2e45-0000-0000-000000000000
Boot ID: 097ed99a-270e-48f0-a24a-60939d723f7d
Kernel Version: 5.10.81+truenas
OS Image: Debian GNU/Linux 11 (bullseye)
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://20.10.11
Kubelet Version: v1.21.0-k3s1
Kube-Proxy Version: v1.21.0-k3s1
PodCIDR: 172.16.0.0/16
PodCIDRs: 172.16.0.0/16
Non-terminated Pods: (0 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
--------- ---- ------------ ---------- --------------- ------------- ---
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 0 (0%) 0 (0%)
memory 0 (0%) 0 (0%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-1Gi 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Starting 23m kubelet Starting kubelet.
Normal NodeHasSufficientMemory 23m kubelet Node ix-truenas status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 23m kubelet Node ix-truenas status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 23m kubelet Node ix-truenas status is now: NodeHasSufficientPID
Normal NodeAllocatableEnforced 23m kubelet Updated Node Allocatable limit across pods
Warning Rebooted 23m kubelet Node ix-truenas has been rebooted, boot id: 097ed99a-270e-48f0-a24a-60939d723f7d
Normal NodeNotReady 23m kubelet Node ix-truenas status is now: NodeNotReady
Normal NodeReady 23m kubelet Node ix-truenas status is now: NodeReady

And this on Pods:
root@TrueNAS[~]# k3s kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-7448499f4d-7nwfs 0/1 Pending 0 2d13h
kube-system openebs-zfs-controller-0 0/5 Pending 0 2d13h
ix-plexgps plexgps-597b9d98cb-24bkm 0/1 Pending 0 3m17


Any suggestions on how to get this sorted out, or at least where to go next in the troubleshooting process?

Thansk, Derek

 

[morganL]

I'd suggest upgrading to the SCALE Release version and if needed, reinstall Plex and point to the original volume.

If you have issues, then it can be treated as a bug.

 

[Saberwolf]

as suggested upgrade. also go into app and edit config and turn off gpu allocation set it to 0 if it deployes check to see if you have an isolated GPU that you are trying to allocate to Plex container

 

[Akeela]

Double check your time zone/clock for your scale system. I was running into a weird issues where Plex wouldn't deploy and would randomly crash because I had the wrong time zone selected.. I think if you have the wrong timezone selected a cert for Kubernetes is unable to authenticate and it just craps out.