Apps stuck deploying

[jspenc]

Hi, I've recently moved over to TrueNAS scale (TrueNAS-SCALE-22.02.3), main reason being I wanted access to all that container goodness and my better knowledge of linux vs FreeBSD.

I have some level of experience working with Docker, writing my own compose files, using portainer etc. I have a high level understanding of Helm and what it's setting out to achieve. That said I don't have a great deal of experience with debugging k8s or k3s issues and am a bit stumped by this one.

I have managed to get Plex setup on my install, this did require a bit of light wrangling as fetching of the image was timing out, so I manually cached it via docker. After that it's been solid.

I am now trying to setup additional charts and am encountering the below on each attempt.

Just to add there does seem to be a bug on the events view with the ordering of the application events getting jumbled up

This RPC error does appear to be consistent across both attempted installs of apps:

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create a sandbox for pod "sonarr-6b6db84b6f-l65cc": operation timeout: context deadline exceeded

output of kubectl get pods

 

[indivision]

Sounds like a network performance issue. What is the network setup?

 

[jspenc]

I don't believe I've configured any special network settings.

Using on of the onboard ethernet ports which is a "HPE Ethernet 1 Gb 2-port 332i adapter"

This is a dump of ifconfig

 

[indivision]

Sorry for not clarifying better. I was meaning more in the hardware sense? Wired connection? NIC card?

 

[Heavy]

Verify that you have the following settings set:

Truenas Scale GUI > apps > Available Applications > Settings > Advanced Settings

Make sure your Route v4 Interface and Route v4 Gateway are set.

 

[stavros-k]

Sounds like a network performance issue. What is the network setup?

How does this sound like a network issue?

This RPC error does appear to be consistent across both attempted installs of apps:

The error combined with the output of the "get pods" command, shows a potential issue with the storage provide (openebs).

I'd suggest making sure you are on the latest version of Scale, and if that's true.
Please create a bug report to iX system's Jira bug tracker. https://ixsystems.atlassian.net/

Don't forget to attach a debug, so they can trace what's going on.

You can get a debug from Settings > Advanced > Save Debug

 

[TAG_TEAM]

{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"ix-chart\" in pod \"centos-ix-chart-7d5dd75d8-v26p5\" is waiting to start: ContainerCreating","reason":"BadRequest","code":400}

I can't seem to get any docker images to run. Looks like a network problem with the code 400, Bad Request... I installed TrueNas Scale fresh to see if my update had gone wrong - same performance. I did set the Kubernetes settings (which were empty) but it made no difference even after a reboot. I was just trying to spin up a CentOS container and it gets stuck on "Deploying". The error I got out of the log by starting the container and immediately selecting logs from the menu, this (above) is all that is displayed.

I don't know if it is related, but I have as well the following in the task manager and I can't clear it and have reinstalled the truecharts repository with force create even after deleting and recreating the pool that was selected for the charts. I don't know if it is related.



The catalog.create job for truecharts has failed at 40%. I can't even dismiss the alert(s). Installing an Official Chart like netdata goes without issue. All Docker containers I have tried fail in the same way, stuck on deploying...

Incidentally here is the log from the applications menu.

 

[Heavy]

@TAG_TEAM
If you open task manager and hover your mous over the caution symbol it should tell you why it failed. I believe if you also click on the task it might bring up the same error message. What does it say?

 

[indivision]

How does this sound like a network issue?

They said that fetching Plex failed multiple times. It sounded like the bug occurs during a fetch/install operation. The cited bug mentioned timing out.

Definitely not suggesting it must be network issue and I could be misunderstanding something...

 

[stavros-k]

Looks like a network problem with the code 400, Bad Request

That's just the middleware that tries to open the pod logs, and because it does not exist, it fails with 400. And that is returned to the WebUI.

If it's ONLY an issue with storage provider, any Apps that do not use PVC, should install.

But at this point with the catalog create failing as well, there seems to be more than the storage problem.
I'd really suggest opening a bugreport to iX-Systems. Before we start hacking around.
Because after that the bug report won't help much.

They said that fetching Plex failed multiple times. It sounded like the bug occurs during a fetch/install operation.

I see, I guess I see few errors way to many times, so immediately exclude other factors.
But usually you would see an ErrImagePull or ImagePullBackoff, if it can't pull the image.

 

[indivision]

I see, I guess I see few errors way to many times, so immediately exclude other factors.
But usually you would see an ErrImagePull or ImagePullBackoff, if it can't pull the image.

You're probably right. I would trust your experience with it over my own.

 

[TAG_TEAM]

The messages in the task manager disappeared magically - perhaps because of a reboot. Here is the log output from the failed CentOS chart.

2022-09-07 2:58:46
Created container ix-chart
2022-09-07 2:58:41
Container image "centos:latest" already present on machine
2022-09-07 2:58:55
Back-off restarting failed container
2022-09-07 2:58:47
Started container ix-chart
2022-09-07 2:58:41
Add eth0 [172.16.0.24/16] from ix-net

Successfully assigned ix-centos/centos-ix-chart-6b8d775fc7-gn8hr to ix-truenas
2022-09-07 2:58:36
Created pod: centos-ix-chart-6b8d775fc7-gn8hr
2022-09-06 22:13:55
Scaled up replica set centos-ix-chart-6b8d775fc7 to 1
2022-09-07 2:58:18
Deleted pod: centos-ix-chart-6b8d775fc7-x8mbf
2022-09-07 2:46:08
Scaled down replica set centos-ix-chart-6b8d775fc7 to 0
2022-09-07 2:46:42
Back-off restarting failed container
2022-09-07 2:46:29
Container image "centos:latest" already present on machine
2022-09-07 2:46:34
Started container ix-chart
2022-09-07 2:46:33
Created container ix-chart
2022-09-07 2:46:29
Add eth0 [172.16.0.23/16] from ix-net

Successfully assigned ix-centos/centos-ix-chart-6b8d775fc7-x8mbf to ix-truenas
2022-09-07 2:46:23
Created pod: centos-ix-chart-6b8d775fc7-x8mbf
2022-09-07 2:46:08
Deleted pod: centos-ix-chart-6b8d775fc7-vzchg
2022-09-06 22:14:05
Created container ix-chart
2022-09-06 22:14:14
Back-off restarting failed container

Is it the consensus that I open a bug report with IX still? I don't want to pester them if someone in this community can provide an answer and a fix.

 

[stavros-k]

Is it the consensus that I open a bug report with IX still? I don't want to pester them if someone in this community can provide an answer and a fix.

It would have been better before the reboot, to catch the problem and fix it for everyone.

Most cases community can give you a fix (I had one that would have worked in your case) But those are temp fixes. As this will happen again, for you and others.

Happy that it works now for you.

But please in the future consider opening bug reports, this will help make the whole system more stable for everyone!

 

[TAG_TEAM]

No, it still does not allow me to deploy any Docker images they still fail on "Deploying." The only thing that changed was the task manager errors are no longer displayed. I didn't read your post until after I rebooted the machine and once I saw your message when I went to check the "catalog.create" error in the task manager it was gone along with the history.

Since I cannot deploy any docker images I probably need to file a bug report. I know how to make the debugging dump and can look for other clues. That being said, do I open a ticket with IX-Systems or some other provider for I am using this without a support license.

 

[TAG_TEAM]

If it is a concern about the task manager messages, I can likely reproduce the errors by updating the apps and/or the docker images...

 

[stavros-k]

Since I cannot deploy any docker images I probably need to file a bug report. I know how to make the debugging dump and can look for other clues. That being said, do I open a ticket with IX-Systems or some other provider for I am using this without a support license.

Catalog create/syncing could be a github's fluke (yesterday I saw it had some timeouts at some point).

But since you can't deploy any Apps, Yes open a bugreport in iXSystems Jira.
You don't need support license for that. It's open for anyone to file a bug report.

(Of course you won't get the "paid" priority)

 

[ASap]

I'm having the same issue with qBittorrent here, and I can confirm the following:

First run (after installation) with the custom specified Downloads Volume, it works;
But then, if the docker stopped and ran again, it stuck in Deploying state;
If I untick the Downloads Volume, the docker can run without issues.