Apps stuck on "DEPLOYING 0/2" only if they have storage attached

[izomiac]

There appear to be a lot of root causes for "stuck on deploying" with SCALE, and this is probably the third time I've had it happen, but usually deleting my apps, rebooting a few times, and recreating them fixes it. Not this time. I've isolated the problem a bit and noticed that attaching "Host Path Volumes" under storage seems to trigger the issue. Apps that require no attached storage or persistent configuration run just fine. I was able to install and deploy Plex from the TrueNAS catalog, TrueCharts, Plex's official docker, and LinuxServer.io, but only if I didn't attach my config or media datasets. Once I tried attaching either, it'd fail to deploy. Next I tried creating a new empty dataset "dockertest" and attach it to an unused location "/dockertest", and it still failed to deploy, which implies it isn't the usual headache of a corrupted config or permissions issue.

Unfortunately I'm not sure exactly what I did that caused this. I recently upgraded to BlueFin and messed around with my gateway and IP settings to implement VRRP. Before I had two TrueNAS VMs with one being the primary router, the second being a VPN gateway router, with some Apps set to use one as a gateway and some to use the other. That worked fine, but TrueNAS was complaining since it didn't have internet access on reboot, so I eventually got around to wiring in a redundant physical router and using VRRP so there's a constant internet connection if the VMs are down. Around this time I noticed Plex was inaccessible on my TV, and discovered that all my apps were stuck deploying. I've reverted every setting related to the split gateway that I'm aware of and only testing with new containers with default settings until I get this issue sorted out.

All I can get "k3s kubectl logs -n ix-plex pod/plex-ix-chart-7cb57ddb5f-xg2vw" to say is: "Error from server (BadRequest): container "ix-chart" in pod "plex-ix-chart-7cb57ddb5f-xg2vw" is waiting to start: CreateContainerError".

 

[tsaG]

You can try to start the app and then check the Application Events. Currently, it seems that all App get stuck if either the shared folder is also used for Rsync or SMB.

For example, for me its my Nextcloud App:
"Error: Error response from daemon: invalid volume specification: '/mnt/Thinktank/User_shares:/var/www/html/data': Invalid mount path. /mnt/Thinktank/User_shares. Following service(s) uses this path: `SMB Share, Rsync Module`."

The workaround for now is to disable the folders rsync and SMB function (it doesn't suffice if you turn off the service). I already created a ticket for it and it was set to "solved" but in the end it was not integrated. :D So, next Beta I guess.

Maybe you can check, scroll through your logs (it was not the last message in the log) and see if this helps.

 

[izomiac]

And it deployed. It looks like this is correct. I wasn't able to find "Application Events", but I found that message under /var/log/k3s_daemon.log. Now I'm left scratching my head as to why a new dataset attached to an unused location would stop it... but the next time I tried that test it deployed as well, so perhaps the initial test failure was a random fluke?

In any case, thanks. I never would have guessed the TrueNAS middleware implementation of Samba and Kubernetes would interact like that.

Edit: Now I found Application Events. I didn't realize you could click on the app tabs. That would have been really helpful if I'd known a couple weeks ago when I started working on this. =/