Logitech Media Server stuck in "Deployment" SCALE 22.12.2

[sfraser131]

Hi I tagged this onto an existing thread regarding migration over the weekend. Likely was not the best spot for it. So I decided to start a new thread. I upgraded from Angelfish to SCALE 22.12.2 on the weekend. I deleted my 3 App's QB, PLEX & LMS and removed the IX-Applications directory prior to the upgrade. After the upgrade I tried reinstalling the 3 app’s mentioned above. I ran into issues other's encountered while trying to re-install the APP's on 22.12.2, such as no config.json file found etc. I ended up downgrading to 22.12.1 which solved the missing json file issue. I then installed QB, PLEX and LMS. QB, and PLEX are working fine, but LMS is stuck in deployment . I have tried upgrading back to 22.12.2 . I currently have disabled Host Path Safety checks, but ran into the same LMS deployment issue prior to any SMB SHARES being created, or turned on for that matter. I have also tried to create the LMS app with out any external storage or host path configuration specified. I have also tried manually specifying the apps user/grp id . Each case i get the same result, LMS stuck in deploying (over 12 hours). The LMS APP log file is empty. This is my 2nd post to this forum, so I apologise if my terminology is off , or I have left out important information. I am happy to respond to questions. Hardware is listed below. Any suggestions are appreciated.

Motherboard: Supermicro X8DTH
CPU's Intel Xeon CPU X5650 @ 2.67Ghz
Memory: 96G ECC
Boot: 1x Kingston 120G SSD SA400S37120G
Storage: 6x WD 4T NAS WD-WCC7K4SJYX55
Controller: LSI SAS2008 PCI-Express Fusion-MPT
File System: ZFS RAIDZ2 6 Wide
Ethernet: Intel 82576 Gigabit

 

[Whattteva]

Yeap, I'm also experiencing that. It's been deploying for probably 4-5 days now and probably will stay like that forever. In addition, I also have 5 other apps that I keep clicking upgrade and they keep failing; so while they are still running, they're stuck at whatever version they currently are now. Thankfully, this SCALE installation is just for testing and I'm not using it for actual production use, so it's no big deal.

TL; DR: SCALE sux.

 

[Tim1962]

Anyone resolve this old thread issue? (Same here)

 

[Whattteva]

I fixed it by reinstalling everything fresh from scratch. Yep, good thing I don't run this in production. My condolences to people using this BETA software as their production setup.

 

[Tim1962]

Thanks, but what do you mean by "reinstalling everything from scratch". How far back did you have to go.

Currently I've discovered an unused HP Proliant with TN13, and upgraded that to Scale with a fresh HDD to play with, so not entirely fresh vanilla install here

 

[Whattteva]

Completely ultra fresh. I've mentioned earlier that this is not a production setup. I'm just running it experimentally because I wanted to see what all the fuss is with SCALE so it has nothing of value and I can always just wipe everything clean. I only trust my production data on CORE.

 

[NickF]

I think we are being a little too general here. @Tim1962 what was your process to migrate to SCALE?

 

[Tim1962]

Initially, "upgraded" in place to Scale from Core 13.something. All worked well and Synching and Ember Apps ran to completion and could access UIs. LMS stuck on deployment, via truecharts, and also as Docker image from lmscommunity. Tried Host storage and PVc without succcess
Tried removing config to vanilla settings (reset to defaults), same result
Just now, brand new vanilla install, no changes to config, and LMS was first thing to install. Stuck at deployment, for an hour so far...

As a reminder, this is an HP Microserver, working perfectly with Core with nothing unusual hardware wise. It's a "spare" so happy to play and wipe. Just glad I didn't do it on the main Core Server with fingers crossed

I think I've done everything that **I** can think of, and recording the details for community benefit. Happy to try other settings if anyone has ideas but I am out of ideas/depth. BW Tim

 

[Tim1962]

Looking at Truecharts website and discord, it seems they require minimum a pair of SSD ...

Given its just a testbed I'd only used one spinning disk :-(
Will have to decide if it's worth the investment just to find out. (Probably yes but hadn't planned on purchase right now ... Hmmm?)

Interestingly I'm not actually sure what Scale does in reality, but if you add ONE hdd it adds it as a stripe, which seems "interesting"

 

[Tim1962]

Well...
Bought and installed 2 1Tb SSD (Crucial) as a mirrored pair.
Changed apps pool
Refreshed the catalogs.
Installed LMS (at this stage completely vanilla settings) , and (after 2-3 minutes) completed and successfully accessed the UI

Now, I don't know if the data needs to be also on the SSD pair, but that's for another day.

It's on the Truecharts site etc, but it's a bit of a shame that (LMS) just freezes on install rather than working slowly (maybe install might have run after a few days!!!). It maybe in the SCALE docs somewhere but I missed it, but maybe SCALE Docs need to promote the need for faster storage for some apps....

Hope this may help someone else

 

[sfraser131]

Well...
Bought and installed 2 1Tb SSD (Crucial) as a mirrored pair.
Changed apps pool
Refreshed the catalogs.
Installed LMS (at this stage completely vanilla settings) , and (after 2-3 minutes) completed and successfully accessed the UI

Now, I don't know if the data needs to be also on the SSD pair, but that's for another day.

It's on the Truecharts site etc, but it's a bit of a shame that (LMS) just freezes on install rather than working slowly (maybe install might have run after a few days!!!). It maybe in the SCALE docs somewhere but I missed it, but maybe SCALE Docs need to promote the need for faster storage for some apps....

Hope this may help someone else

Tim, did you ever figure out if the "mirrored pair" fix the problem , or was it changing the apps pool?
I still have not been able to get LMS running successfully.

 

[Tim1962]

Tim, did you ever figure out if the "mirrored pair" fix the problem , or was it changing the apps pool?
I still have not been able to get LMS running successfully.

TBH, not really but strong suspicion it was move from spinning rust to SSD that was critical. I read somewhere in corner of t’internet that read/write speed was particularly critical but didn’t keep the reference ( might have just dreamt it!!!)

 

[mymedia]

I'm also a LMS user and new to the Truenas Scale, but I bought into the system because I have some knowledge on Kubernetes.
I also ran into the stuck deployment issue with the Truecharts LMS chart.

I dug a little bit deeper and found the underlying problem and a temporary workaround.
The chart uses the docker image that is build by the slimserver-platforms Dockerfile
That file points to the script that is run on the container start: start-container.sh
The chmod command in that script on the /lms folder takes about ~10 minutes on my system with WD disks that run on 5400 RPM.
The default startupProbe settings in the chart only wait for period * failure seconds = 300 seconds = 5 minutes

Startup: tcp-socket :9000 delay=10s timeout=2s period=5s #success=1 #failure=60

This means the the deployment will check if the pod is ready (lms reachable on configured port) after 5 minutes and will restart it. Thus results in the "stuck deployment" we experience.

The workaround is to edit the deployment via kubectl commands after it was created by the chart. Unfortunately this edit will be overriden if you make any changes to the chart via the "apps" UI, i.e. to change configuration or update.

The steps that worked for me are the following:

• Make sure the currently logged in user can run k3s with sudo
• Open the "System settings" -> "shell"
• Get some information about the system:
  • List the current namespaces with: sudo k3s kubectl get namespaces
    The namespace for the LMS should be ix-logitech-media-server
  • List the current pods in the namespace: sudo k3s kubectl get pods -n ix-logitech-media-server
    The LMS pod should have the Status Running, but Ready 0/1
  • List the deployment: sudo k3s kubectl get deployments -n ix-logitech-media-server
• In Kubernetes, the deployment creates the pods, so we have to edit the deployment with:
  sudo k3s kubectl edit deployment logitech-media-server -n ix-logitech-media-server
  This will open the file with the default editor, in my case this is vi.
• With vi, move to the line failureThreshold under startupProbe and change the 60 to i.e. 600
  Type i to enter the vi edit mode, and do the above mentioned change. Hit esc to get out of the edit mode again.
  Type :wq to exit write the changes and quit vi again.
• Changes to a Kubernetes deployment file will result in a redeployment of the pod. So you should see a terminating pod with
  sudo k3s kubectl get pods -n ix-logitech-media-server right after the edit.
• Wait now for the pod to get into running state. (took about 10 minutes with my disks)
  You can i.e. check that status with watch: watch -n 10 sudo k3s kubectl get pods -n ix-logitech-media-server
  Until you see something like this (name age and restarts my vary):
  NAME READY STATUS RESTARTS AGE
  logitech-media-server-55c5d4dc94-vsbp4 1/1 Running 0 132m

Hope this helps and can save some time. It took me about two full days to figure this all out.
Best regards,
Stefan

 

[mymedia]

For the sake of completeness, these are the Github issues I created to resolve this:
Truecharts requests to make the startProbe configurable in the chart UI

logitech-media-server - make startupProbe configurable to support systems with slow disks · Issue #13504 · truecharts/charts

Is your feature request related to a problem? On TrueNAS Scale systems with HDDs, the chart gets stuck in deployment because the chown used in the startup script of the Dockerfile takes longer that...

github.com

Slimdevices issue to maybe move the chown to the Dockerfile (if possible):

Docker - slow startup on systems with spinning disks results in stuck deployment on Kubernetes chart · Issue #44 · LMS-Community/slimserver-platforms

I run the LMS with the Kubernetes chart form Truecharts on my Truenas SCALE NAS with two spinning disks. The chown command in the start-container.sh script takes around 10 minutes to complete. In p...

github.com

 

[aspegren]

Newbie to Truenas. TrueNAS-SCALE-23.10.0, a i5 motherboard 24gb memory, a SSD Drive with the OS, and 4* 10Tb drives Loaded with my music library. Installed LMS 8.04 and later 8.05, but stuck in deployment, have had brief success with MyMedias tips above, but then LMS could not see my dedicated SMB share (visable from other pc's on my network) path is /mnt/Volume 1/Peters, in LMS /mnt was empty.

I have had several other NAS drives but this is a very different beast in so many ways.

Boiled down my issues are:
1. How to get Logitech Media server 8.05 up and running???
2. How to get LMS to see my share /mnt/Volume 1/Peters ???

 

[aspegren]

History reads

2023-11-04 13:45:15Startup probe failed: dial tcp 192.168.2.32:9099: connect: connection refused
2023-11-04 13:45:03Created container logitech-media-server
2023-11-04 13:45:03Started container logitech-media-server
2023-11-04 13:45:00Scaled up replica set logitech-media-server-58cb9f88d5 to 1
2023-11-04 13:45:00Created pod: logitech-media-server-58cb9f88d5-x9c2k
2023-11-04 13:45:00Successfully assigned ix-logitech-media-server/logitech-media-server-58cb9f88d5-x9c2k to ix-truenas
2023-11-04 13:45:00Container image "tccr.io/truecharts/logitechmediaserver:8.4.0@sha256:aec92fe36f1f31b8a5e929935840400170d7ed188555cbf3604e9f71ee5c4019" already present on machine
2023-11-04 13:43:49Scaled down replica set logitech-media-server-7fdcdcd68b to 0 from 1

 

[aspegren]

Extended failureThreshold to 600 above and got the app in Running state briefly, ut couldnt open web GUI, now back in Deploying state again.
History reads:
2023-11-04 14:19:25Startup probe failed: dial tcp 192.168.2.32:9099: connect: connection refused (which is the web giu address)
2023-11-04 14:19:14Started container logitech-media-server

 

[aspegren]

Logs says repeatedly:
2023-11-06 11:16:16.810086+01:00chown: changing ownership of '/lms/.dockerignore': Operation not permitted
2023-11-06 11:16:16.810095+01:00chown: changing ownership of '/lms/Changelog2.html': Operation not permitted
2023-11-06 11:16:16.810107+01:00chown: changing ownership of '/lms/Slim-Utils-OS-Custom.pm': Operation not permitted
2023-11-06 11:16:16.810118+01:00chown: changing ownership of '/lms/Changelog.html': Operation not permitted
2023-11-06 11:16:16.810137+01:00chown: changing ownership of '/lms/Changelog6.html': Operation not permitted
2023-11-06 11:16:16.810150+01:00chown: changing ownership of '/lms': Operation not permitted
2023-11-06 11:16:16.810418+01:00Starting Logitech Media Server on port 9001...
2023-11-06 11:16:17.817157+01:00Password: su: Authentication failure

 

[aspegren]

What have I missed????

 

[cap]

When I installed the LMS a few months ago, it ran as root = 0.
I deleted the app immediately.