Looking for Bluefin systems where Apps don't start

[morganL]

Thanks for asking, lucky me found the thread :) I ve updated just yesterday, wanted to wait for a later version,
to NOT get stuck in such a mess :P

two apps aren't starting and hanging in deploying, the more important one Traefik (hence none of my services is reachable atm)

Application events:

2023-04-16 19:16:30
Updated LoadBalancer with new IPs: [] -> [192.168.0.112]
2023-04-16 19:16:30
Container image "ghcr.io/truecharts/alpine:v3.14.2@sha256:4095394abbae907e94b1f2fd2e2de6c4f201a5b9704573243ca8eb16db8cdb7c" already present on machine
2023-04-16 19:16:30
Error: Error response from daemon: invalid volume specification: '/usr/bin:/host/usr/bin': '/usr/bin' 'path' not allowed to be mounted
2023-04-16 19:16:29
Updated LoadBalancer with new IPs: [] -> [192.168.0.112]
2023-04-16 19:16:29
Created container autopermissions
2023-04-16 19:16:29
Started container autopermissions
2023-04-16 19:16:28
Add eth0 [172.16.1.115/16] from ix-net
2023-04-16 19:16:28
Container image "ghcr.io/truecharts/alpine:v3.14.2@sha256:4095394abbae907e94b1f2fd2e2de6c4f201a5b9704573243ca8eb16db8cdb7c" already present on machine
2023-04-16 19:16:27
Ensuring load balancer
2023-04-16 19:16:27
Applied LoadBalancer DaemonSet kube-system/svclb-traefik-3861cbce
2023-04-16 19:16:27
Ensuring load balancer
2023-04-16 19:16:27
Applied LoadBalancer DaemonSet kube-system/svclb-traefik-tcp-ed4c2336
2023-04-16 19:16:27
Scaled up replica set traefik-759745fd85 to 1
2023-04-16 19:16:27
Created pod: traefik-759745fd85-v2twt
2023-04-16 19:16:27
Successfully assigned ix-traefik/traefik-759745fd85-v2twt to ix-truenas


if some one got a hint, so far I didn't stumble upon anything :|

There are some issues with TrueCharts Apps after their independent update on April 11. Is it only TrueCharts apps you are having issues with?

Their news is here: https://truecharts.org/news/post-easter-stable-update/

 

[derWalter]

There are some issues with TrueCharts Apps after their independent update on April 11. Is it only TrueCharts apps you are having issues with?

Their news is here: https://truecharts.org/news/post-easter-stable-update/

omg .. thanks for the news :/

yes, it is :|

i knew this will happen once I hit the update button, the truecharts gang is very hard working, but so aggressive towards people who are not 100% into their world as they are, tbh. I really dont want to get in touch with them anymore.

the road to canossa I will have to take, once more

edit: well, well, well, they told me I had to reinstall, but at least now it runs again.

 

[serving_myself]

There've been many users hitting the problem with pool initialization for Apps (e.g., see: https://www.truenas.com/community/threads/kubernetes-error-when-choosing-pool.106260/#post-757635 / NAS-121509).

For me personally, no workaround has helped.
However, I've recreated the problem on a test VM (fresh install of 22.12.2) and it *seems* to me that the problem is with the name of the pool:
- when the pool is named "applications" I can consistently recreate the issue (tried multiple times with destroying and recreation of it with this name - always hit the problem).
- when the pool is named something else (I've named it "apps") - all's good! Just repeated it on my baremetal - all's good as well.

 

[speedtriple]

Have only 1 app, Syncthing from Official charts, and it does not start - stuck on 'Deploying'.
And the app does not 'upgrade' without errors.

Workaround - disable the SMB Share for Syncthing. The the app starts or does the upgrade.
When 'Active' enable the SMB Share again.

Have been like this for all my SCALE versions, including 22.12.2.

Running TrueNAS SCALE as a ESXi VM. (and have always done so)

 

[efalsken]

(repost from another discussion)

Adding a status note on TrueNAS SCALE Bluefin.... There are now about 15,000 Bluefin systems up and active. Most of iX TrueNAS test systems have been running Bluefin very reliably.

Apart from the bugs identified in the release notes, the Apps have had the most reports of problems. So far 8 bug reports. We'd guess 10s of systems are having problems.... but frankly we don't have a great way of collecting this data, apart from these forums.

Unfortunately, we have not seen a pattern in those existing reports nor have we been able to reproduce a case. We are not sure if there are one or more causes of the issues... and whether there are specific processors/hardware that cause issues. It may or may not be related to having multiple pools.

So, we have three requests:

1) Please report any case where Apps are failing to run. Make a comment here. Start a new thread with hardware details, we'll see if there's any obvious problems and then ask for a bug report. We can also see more clearly how many systems have an issue. Finding a simple system with an issue would be useful.

2) If anyone can reproduce the issue on a SCALE VM, that would be very useful. We might be able to reproduce and isolate any software causes faster.

3) If anyone has time and spare hardware available, try building an angelfish system, installing Apps and then updating to Bluefin. Please report success/failure and the hardware spec. 1st person with a failure should start a separate thread.

Obviously, we'd like to chase down any issues like this ASAP. Thanks in advance for any help.

Pretty reliably, my apps do not start after a reboot. I typically have to go to the Apps screen and Unset the pool, then Choose Pool and they'll start right up.

 

[morganL]

Pretty reliably, my apps do not start after a reboot. I typically have to go to the Apps screen and Unset the pool, then Choose Pool and they'll start right up.

We're going to make 22.12.3 available in the next month. I'd suggest waiting for that and seeing if problem persists. If it does, please start another thread and report-a-bug if you have the time to help us diagnose the issue.

There is one bugfix that is likely to be relevant in 22.12.3 - expected in mid-June,

[NAS-120955] - iXsystems TrueNAS Jira

ixsystems.atlassian.net

This bug forces Apps starting to wait for the cgroups in kernel to be available. We think this is the cause of many customer issues. The behaviour varied based on the system build.

 

[BoostedIS300]

Hello not sure if this is relevant but I'm having an issue with some of my apps not starting about a week after an update to TrueNAS-SCALE-22.12.4.2

Issue with Apps Starting/Removing/Re-installing

Hello, Yesterday I started noticing an issue with one of my apps (uptime-kuma) not starting after a system reboot. It would go through the loading screen of starting, reaching 100% but then never deploy. I attempted this multiple times to no avail. I rebooted my system in an attempt to get a...

www.truenas.com

 

[morganL]

Hello not sure if this is relevant but I'm having an issue with some of my apps not starting about a week after an update to TrueNAS-SCALE-22.12.4.2

Issue with Apps Starting/Removing/Re-installing

Hello, Yesterday I started noticing an issue with one of my apps (uptime-kuma) not starting after a system reboot. It would go through the loading screen of starting, reaching 100% but then never deploy. I attempted this multiple times to no avail. I rebooted my system in an attempt to get a...

www.truenas.com

Probably not the same issue.... we found an issue in Bluefin where all Apps on a system didn't start due to a Kubernetes race condition.
If it starts and fails, its a different issue... either in App on general systems.