Plex App generates console message "IPVS: rr: UDP 172.17.21.37:32412 - no destination available"

JamieMcH

Cadet
Joined
Jan 22, 2021
Messages
5
I have installed the Plex app (v1.0.0) from the TrueNAS SCALE catalog. All worked fine to start with, however after editing it's settings to add a "Data Hostpath" the app will no longer start. The console just continually generates the following message until the app is manually stopped again:
IPVS: rr: UDP 172.17.21.37:32412 - no destination available

Plex App Settings:
  • Image Repo - plexinc/pms-docker
  • Image Tag - plexpass
  • Image Pull Policy - Only pull image if not present on host
  • Plex Node Port - 32400
  • Host Network - Checked
  • Transcode Hostpath Enabled - Unchecked
  • Data Hostpath Enabled - Checked
  • Config Hostpath Enabled - Unchecked
Hardware Specs:
  • Motherboard - ASUS X79-DELUXE
  • CPU - Intel Core i7-4820K @ 3.70GHz
  • RAM - 32GB
  • Boot Drive - 1x Kingston HyperX
    • SH103S3/240G
  • Hard Drives - 3x 3TB in RAIDZ1
    • WDC_WD30EZRX-00D8PB0 - 3TB
    • WDC_WD30EZRX-00D8PB0 - 3TB
    • WDC_WD30EFRX-68EUZN0 - 3TB
  • Controller - Onboard SATA
  • Network Card - Onboard 1GbE
 

peter.m

Dabbler
Joined
Jan 1, 2021
Messages
41
These messages are most likely not your problem - the app isn't starting, but the underlying network CNI is setting up firewall rules for it, and ipvs is complaining that the container is not listening on the target. Getting the status of the container via kubectl will most probably show what's wrong underneath (I'm guessing the volume isn't mounted properly)
 

JamieMcH

Cadet
Joined
Jan 22, 2021
Messages
5
Thanks @peter.m for your reply.

I assume when you say "I'm guessing the volume isn't mounted properly" you are referring to the data volume for Data Hostpath. I have tested this by removing the Data Hostpath and the issue persists.

Also, I am new to TrueNAS and Containers, so please forgive my lack of knowledge. How do I run kubectl? I tried simply typing it into the shell which returned "zsh: command not found: kubectl"
 

peter.m

Dabbler
Joined
Jan 1, 2021
Messages
41
Right, in SCALE it's hidden under k3s. So every time you want to run kubectl, you do k3s kubectl. What you'll want to do is kinda like this
k3s kubectl --all-namespaces get pods, after which it'll list your plex pod. Then k3s --namespace <namespace of plex> describe <name of plex pod> and you'll be looking under status: {} what went wrong. You might need to also pass the kubeconfig config to kubectl, I don't remember the path - it's in the developer notes, and an export KUBECONFIG=<path_to_yaml> will suffice
 

JamieMcH

Cadet
Joined
Jan 22, 2021
Messages
5
So after some additional reading and your assistance I have managed to get an output that I think will help:

I ran k3s kubectl describe pod/<pod name> --namespace <namespace> which returned the following result.

Code:
Name:         Plexbox-76495b74c9-tdwvh
Namespace:    ix-plexbox
Priority:     0
Node:         ix-truenas/<hostIPAddress>
Start Time:   Mon, 25 Jan 2021 14:10:38 +1100
Labels:       app.kubernetes.io/instance=plexbox
              app.kubernetes.io/name=plex
              pod-template-hash=76495b74c9
Annotations:  rollme: m1rsG
Status:       Running
IP:           <hostIPAddress>
IPs:
  IP:           <hostIPAddress>
Controlled By:  ReplicaSet/plexbox-76495b74c9
Containers:
  plex:
    Container ID:   docker://4182d082345ac1b2799248da4e594800a08f6f944be516500d3c54be8cc1d943
    Image:          plexinc/pms-docker:plexpass
    Image ID:       docker-pullable://plexinc/pms-docker@sha256:34cf89222747c4b151e125f77a4409c60a5a317afe1fdb91317a64e5a0d5e69e
    Ports:          32400/TCP, 32469/TCP, 1900/UDP, 32410/UDP, 32412/UDP, 32413/UDP, 32414/UDP
    Host Ports:     32400/TCP, 32469/TCP, 1900/UDP, 32410/UDP, 32412/UDP, 32413/UDP, 32414/UDP
    State:          Running
      Started:      Mon, 25 Jan 2021 14:21:42 +1100
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Mon, 25 Jan 2021 14:10:44 +1100
      Finished:     Mon, 25 Jan 2021 14:21:40 +1100
    Ready:          False
    Restart Count:  1
    Liveness:       http-get http://:32400/identity delay=0s timeout=1s period=15s #success=1 #failure=5
    Readiness:      http-get http://:32400/identity delay=0s timeout=1s period=15s #success=1 #failure=5
    Startup:        http-get http://:32400/identity delay=5s timeout=1s period=15s #success=1 #failure=40
    Environment:
      TZ:                     Etc/UTC
      PLEX_CLAIM:             claim-<token>
      PMS_INTERNAL_ADDRESS:   http://plexbox:32400
      PMS_IMAGE:              plexinc/pms-docker:plexpass
      KUBE_NAMESPACE:         ix-plexbox (v1:metadata.namespace)
    Mounts:
      /config from config (rw)
      /config/library/Application Support/Plex Media Server/Logs from shared-logs (rw)
      /data from data (rw)
      /shared from shared (rw)
      /transcode from transcode (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-xjv48 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  data:
    Type:         HostPath (bare host directory volume)
    Path:         /mnt/Media
    HostPathType:
  config:
    Type:         HostPath (bare host directory volume)
    Path:         /mnt/Media/ix-applications/releases/plexbox/volumes/ix_volumes/ix-plex_config
    HostPath Type:
  transcode:
    Type:         HostPath (bare host directory volume)
    Path:         /mnt/Media/ix-applications/releases/plexbox/volumes/ix-volumes/ix-plex_transcode
    HostPath Type:
  shared:
    Type:         EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:    <unset>
  shared-logs:
    Type:         EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:    <unset>
  default-token-xjv48:
    Type:         Secret (a volume populated by a Secret)
    SecretName:   default-token-xjv48
    Optional:     false
QoS Class:        BestEffort
Node-Selectors:   <none>
Tolerations:      node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                  node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type      Reason      Age                    From                  Message
  Normal    Scheduled   14m                                          Successfully assigned ix-plexbox/plexbox-76495b74c9-tdwvh to ix-truenas
  Normal    Pulled      14m                    kubelet, ix-truenas   Container image "plexinc/pms-docker:plexpass" already present on machine
  Normal    Created     14m                    kubelet, ix-truenas   Created container plex
  Normal    Started     14m                    kubelet, ix-truenas   Started container plex
  Warning   Unhealthy   14m                    kubelet, ix-truenas   Startup probe failed: Get "http://<hostIPAddress>:32400/identity": dial tcp <hostIPAddress>:32400: connect: connection refused
  Warning   Unhealthy   4m46s (x39 over 14m)   kubelet, ix-truenas   Startup probe failed: Get "http://<hostIPAddress>:32400/identity": EOF

Do you know what could be causing the "connection refused in the "Events:" segment?
 

peter.m

Dabbler
Joined
Jan 1, 2021
Messages
41
Great, so what's happening here is kubernetes runs what are called probes to judge if the given pod is healthy or not. If the probe fails enough times, the pod gets killed and deemed unhealthy. In your case, it tries to curl http://<hostIPAddress>:32400/identity and doesn't get a reply. I don't know much about plex, so you tell me if that's correct or not. There may be something else going on in plex itself - if the container is made right, it should spew out some logs, you can get at them by doing kubectl --namespace <namespace> logs <name_of_pod>, in your case that would be k3s kubectl --namespace ix-plexbox logs plexbox-76495b74c9-tdwvh

btw you might want to get k9s, makes looking at this stuff much easier :) https://k9scli.io
 
Last edited:

JamieMcH

Cadet
Joined
Jan 22, 2021
Messages
5
Ok, so thinking there was an issue with TrueNAS, I decided to wait till 21.02 was released and try again.
I ended up having the same issue. But I realised what caused it this time.

In PMS under Settings > Network > Secure connections I changed the setting to "Required". This forced a restart of the pod.
As the pod is attempting to re-deploy it is testing connections to all of the ports configured. Two of the ports are UDP:32412 & UDP:32414 which are used for GDM network discovery. I suspect these ports are used for insecure connections, so are rejecting communications.

This is causing TrueNAS to continue displaying "Deploying" instead of "Active". However, if I navigate to PMS via a web browser using https (instead of TrueNAS default of launching the portal via http) then I am able to connect to the PMS portal. This tells me that PMS is in fact "Active", and not "Deploying".

The issue is that the local server console repeatedly outputs the lines below, and the TrueNAS portal shows "Deploying" instead of "Active". This also prevents ">_Shell" access as TrueNAS does not see an active pod.
Code:
IPVS: rr: UDP <Pod IP>:32412 - no destination available
IPVS: rr: UDP <Pod IP>:32414 - no destination available 


I don't know if this is a PMS Docker image issue, a Docker/K3s issue, or a TrueNAS issue. Any thoughts?
 

waqarahmed

iXsystems
iXsystems
Joined
Aug 28, 2019
Messages
136
@JamieMcH I suspect this is an issue of the Plex Catalog Item ( Plex Chart ) and it probably needs to account for this. Can you please create an issue at "jira.ixsystems.com" and outline the details of your issue with steps to reproduce as well ? ( Please do attach a debug of the system as well )

Thank you
 

JamieMcH

Cadet
Joined
Jan 22, 2021
Messages
5
@waqarahmed I tried to create a bug report as requested, but it keeps giving me an error:

We can't create this issue for you right now, it could be due to unsupported content you've entered into one or more of the issue fields. If this situation persists, contact your administrator as they'll be able to access more specific information in the log file.
 

waqarahmed

iXsystems
iXsystems
Joined
Aug 28, 2019
Messages
136
Thank you for letting me know @JamieMcH, can you please provide more details via email me to me ? ( waqar@ixsystems.com )

More details would be Project selected in the ticket, username and any other options which were chosen like bug/suggestion etc.
 

appliance

Explorer
Joined
Nov 6, 2019
Messages
96
Do you have a ticket for this? I'd like to fix it in the most recent TrueNAS/TrueCharts.
 

ArmadaFG

Cadet
Joined
Sep 9, 2023
Messages
1
Hi,
It always works very well when I use Freenas, but with this new installation of Truenas, impossible to make it work
 
Top