All apps stuck on deploying after reboot, sometimes

[Whiskey]

After booting all my apps are sometimes stuck on deploying. Usually a reboot brings things back in order, but I can't find the reason for this.

I've been playing with Scale for a few months now, ever since I got an R620 with oodles of ram and cpu power to play with. However, because of the current electricity prices I'm not running the server 24/7 (yet). So I frequently need to boot up, and I have about a 50/50 chance that when the apps are started, they all stay stuck in the "deploying" state.

I'm not very familiar with k3s yet and don't really know how to troubleshoot this or what configuration to check. I've read through many forum threads, but haven't found the answer yet. I did configure the network settings (interface and gateway, as mentioned here).

Code:

k3s kubectl get pods -A

shows this when things are stuck:

I grabbed the k3s_daemon.log as well that I can share, if that helps. Looking for any advice that can help me track down the cause of the problem (or suggestions for fixing this of course!). Thanks.

 

[Whiskey]

Still haven't solved this, usually just reboot because I got other things I want to work on. But had another look just now and when I describe any pod, they all say this under events:

And a systemctl status k3s gives me this:

To be continued... any help still appreciated

 

[Daisuke]

Normally, pods are not deployed until the catalogs refresh is completed, which occurs on a server reboot and takes a while if you have TrueCharts catalog installed. Give it like 5-10min and everything should be up and running fine.

 

[Whiskey]

Thanks for the suggestion!
As it happens, I left things as they were and went for a walk with the dog. Its 1h30m later now and all apps are still stuck in deploying. So unfortunately this is not the answer (I also checked and the catalog had refreshed successfully in the meantime).

 

[Daisuke]

Are you running TrueCharts apps? If yes, reach at their Discord channel, they should have an answer in no time. I never saw this behaviour before, except once when there was an issue with the disks.

 

[Whiskey]

yes, I am running TrueCharts apps. But actually thought it would more likely be an k3s issue since all apps stay stuck, both an official one as well as the TrueCharts ones. But may jump on Discord anyway and see if they have any suggestions.

 

[mexe]

Hi, checking into this thread. I'm experiencing this issue as well. I also run TrueCharts and the cataloging on boot takes roughly 2-3 minutes. However, the 3-4 containers I have running get stuck on "Deploying" after system reboots and shutdowns. The matter does not seem to resolve itself as I had turned on my server, let it run over night without checking it, and found it this morning with its containers stuck in "Deploying".

Typically it takes a reboot or two to get the system to get the containers going as it should.

 

[sensor]

Facing this problem everytime after a reboot.
There are some apps that keep stuck in the 'deploying' mode. Frustrating! Reboots won't solve the problem.

 

[mexe]

Facing this problem everytime after a reboot.
There are some apps that keep stuck in the 'deploying' mode. Frustrating! Reboots won't solve the problem.

Are you on Bluefin or Anglefish?
Upgrading to Bluefin a month or so after its release fixed this issue and dramatically changed how fast it takes for cataloging and container deployment on boot.

 

[sensor]

Are you on Bluefin or Anglefish?
Upgrading to Bluefin a month or so after its release fixed this issue and dramatically changed how fast it takes for cataloging and container deployment on boot.

Bluefin (22.12.1) odd.. going to backup plex and reinstall I guess.

 

[best112]

I have the same issue, some apps will not deploy after restart. There was one app that needed an update and it was not able to complete the update because it said that it was not able to access the SMB server.
so I turned off the SMB server in truenas and started each app and they all deployed and the update worked.
after all apps deployed I turned on the SMB server again. Hope this helps

 

[mainstream]

I also have the same problem over and over. I figured that looking in the applications event logs, that there's a problem with network shares. When these are turned off, it seems to work again...

So turn of your SMB or NFS shares after restart and redeploy your applications is fixing it. Isn't there a better solutions for this?

 

[Patrick M. Hausen]

You might want to additionally check this:

Setting Inotify in TrueNAS SCALE

truecharts.org