SOLVED How to Troubleshoot Container Stuck on Deploying?

[yottabit]

I am attempting to deploy the official Debian docker image. It has been stuck on 'Deploying' status for a half hour now, with no obvious activity on the system.

I had this problem before while deploying Plex, and found that my config dataset was causing it to hang. In this case for Debian, I have not attached any datasets to the config (yet).

How do I troubleshoot when stuck in the 'Deploying' state? This is incredibly frustrating to not have any idea what's going on... is progress being made? Was there an error? etc.

The 'application events' log for the container shows:

Code:

2021-09-16 11:46:02
Scaled up replica set debian1-ix-chart-57b9c9cbb5 to 1
2021-09-16 11:46:02
Created pod: debian1-ix-chart-57b9c9cbb5-8vxkg

Successfully assigned ix-debian1/debian1-ix-chart-57b9c9cbb5-8vxkg to ix-truenas
2021-09-16 11:46:12
Add eth0 [172.17.0.107/16] from ix-net
2021-09-16 11:46:18
Add net1 [172.16.42.156/24] from ix-debian1/ix-debian1-0
2021-09-16 11:46:18
Container image "debian:latest" already present on machine
2021-09-16 11:46:26
Created container ix-chart
2021-09-16 11:46:27
Started container ix-chart
2021-09-16 11:46:36
Back-off restarting failed container

So in this case there was an error. Great. Now what?

I miss iocage.

 

[Kris Moore]

Will need to grab some logs to find out, here's one way to do that:

# k3s kubectl get pods,svc,daemonsets,deployments,statefulset,sc,pvc,ns,job --all-namespaces -o wide

Locate your pod / container and then run:

# kubectl logs -n ix-debian1 pod/name_here


Might give us an idea of whats going on with this particular deployment. Also, why are you running the debian:latest image? You may need to provide an entrypoint argument, like this:

How can I create an Ubuntu container ("app") in TrueNAS Scale w/ SSH?

Hi guys - I have a bunch of python scripts that I run to organize the files on my media server. Instead of installing the scripts directly on the host, which would require adding a ton of python libraries or using a VM, I thought I could take full advantage of TrueNAS Scale and launch a simple...

www.truenas.com

 

[yottabit]

Aha! I'm even active in that thread! I missed comment #2 which has the solution.

One thing I read in that thread that gives me pause is that it seems on container start, any state in the container from the last session would be lost, so so that's why we attach datasets as mountpoints, etc. But then this causes an annoyance for me as it means for my use cases, I would have to reinstall all of the packages I need for my workflows on every launch. I can automate this with a script, but considering I need golang, google-cloud-sdk, jdupes, google-chrome, SSH, and several other heavy packages with a laundry list of dependencies, this would be pretty wasteful (time, bandwidth).

What I really want is a jail... which lxc provides in linux, but I'm unsure on the approach I should use for this with SCALE.

 

[yottabit]

May have spoken a little too soon. The container shows as 'Active', but I cannot shell into it:

Code:

Error from server: error dialing backend: dial tcp: lookup ix-truenas: no such host

I tried several of the usual shell paths: /bin/bash (default), /usr/bin/bash, /bin/sh ... nope. :-/

 

[yottabit]

Grabbed the container ID from `docker ps` and then used `docker exec -it <container_id> /bin/bash` and it worked.

Next I did `touch /home/testfile`, restarted the container, and then wept as I discovered my testfile was missing.

I guess I will need to use a VM until we get a jail-like solution in SCALE and/or I have enough time to create my own container with the userland I need.

Good learning experience.

 

[Kris Moore]

One thing I'd recommend... Make your own Docker image. Instead of using Debian stock and then customizing it in runtime. Roll your own Dockerfile, pull in debian, add the packages / config you want, and then run that. That way you only map in the storage that has your local configuration / state, and the packages / base get updated as you push updated to the docker image. I've moved to that model and I don't miss jails even one bit now :)

 

[yottabit]

Yes, definitely will take this approach. I just need to time to do it. This is similar to instantiation scripts I had created for iocage before snapping and cloning those datasets.

I will need to take some time to figure out how to add the new docker image to the SCALE local repo for deployment, unless I just CLI / cron it directly with docker I guess (bypassing the UI and k3s).

Thanks!

 

[ericfoss]

As a person who came here and didn't find what I needed, I thought I'd add what helped me. Once you have the namespace from above or what I use below, just call describe.

k3s kubectl get pods --all-namespaces
k3s kubectl describe pods -n <namespace>

 

[NugentS]

There are "jails" that can be used on Scale.
Its not official - but appears to work: https://gist.github.com/Jip-Hop/4704ba4aa87c99f342b2846ed7885a5d