Apps not working after update

[eirik.lars1]

Updated from 22.12.0 to 22.12.2 and now my apps are not working. I get this error message:

Code:

Failed to configure kubernetes cluster for Applications: Missing 'cpu, cpuset, hugetlb' cgroup controller(s) which are required for apps to function

I tried unset pool, restarting the system and setting the pool for the apps again but then I get the same error message.

Have I screwed up my apps?

 

[morganL]

Updated from 22.12.0 to 22.12.2 and now my apps are not working. I get this error message:

Code:

Failed to configure kubernetes cluster for Applications: Missing 'cpu, cpuset, hugetlb' cgroup controller(s) which are required for apps to function

I tried unset pool, restarting the system and setting the pool for the apps again but then I get the same error message.

Have I screwed up my apps?

There's been a couple of similar events here:

https://www.truenas.com/community/threads/selecting-pool-fails.109285/

 

[eirik.lars1]

There's been a couple of similar events here:

https://www.truenas.com/community/threads/selecting-pool-fails.109285/

Yes I saw that one. Unfortunately rebooting didn’t solve it for me. I tried rebooting after unsetting the pool and then setting it again after reboot but still the same error.

 

[eirik.lars1]

Anyone else having this issue who can shed some light ? I really want my apps back…

 

[silkie]

I'm getting this. Combined with the Truechart cat issues I feel like I'm banging my head against a brick wall. Would love to work with someone knowledgeable to resolve. I thought I got everything stable today until my apps died again after reboot with this message. It was after a good few hours of upgrades, reinstalls & reconfigs. Even falling back to the previous boot image wont resolve now. I fear I've nuked my K3S install somehow.

Having a little blind poke and issuing journalctl -xe I get this:

Apr 17 18:47:59 xxxx k3s[83919]: time="2023-04-17T18:47:59Z" level=fatal msg="failed to get docker runtime info: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
Apr 17 18:47:59 xxxx systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE

 

[silkie]

I also notice I have this message in the "Manage Docker Images" tab.

No Docker Images are available​

 

[eirik.lars1]

I'm getting this. Combined with the Truechart cat issues I feel like I'm banging my head against a brick wall. Would love to work with someone knowledgeable to resolve. I thought I got everything stable today until my apps died again after reboot with this message. It was after a good few hours of upgrades, reinstalls & reconfigs. Even falling back to the previous boot image wont resolve now. I fear I've nuked my K3S install somehow.

Having a little blind poke and issuing journalctl -xe I get this:

I unset/set the pool again and got my apps back. Next step is to reinstall everything to get updates working again.

 

[silkie]

@eirik.lars1 thanks for the reply. unset/set failed for me. on set it repeats the error message in your original post. I hope I haven't killed the main apps dataset somehow. The idea of a rebuild is quite unappealing. might take me a day or so to build up the effort required! :)

 

[eirik.lars1]

@eirik.lars1 thanks for the reply. unset/set failed for me. on set it repeats the error message in your original post. I hope I haven't killed the main apps dataset somehow. The idea of a rebuild is quiet unappealing. might take me a day or so to build up the effort required! :)

Sorry, I wasn't beeing precise. I actually changed my pool to my main pool, and then switched it back to my apps pool. After that I got the apps back. Only unsetting/setting the pool didn't work for me either.

 

[silkie]

Sorry, I wasn't beeing precise. I actually changed my pool to my main pool, and then switched it back to my apps pool. After that I got the apps back. Only unsetting/setting the pool didn't work for me either.

Bingo! thankyou buddy!! I'm back in the game.

 

[Kalaxis]

This took me several stages to fix:

1. Unset the pool.
2. Reboot
3. Set the pool to my other pool.
4. Unset the pool.
5. Reboot.
6. Set the pool back to my original pool.
7. TrueNAS reported that Plex was running, except it wasn't. I stopped Plex.
8. Upgraded it, though a simple start of the application would probably have sufficed

 

[omeganot]

Does it really take moving to another pool to get this to work?

 

[Busthead]

I'm a long time FreeNAS user. Spun up a test instance of TrueNAS Scale hoping there would be fewer bugs, then I received this error :(

 

[conzty01]

I had the same error message, but didn't have a second pool to switch to. I was able to resolve my problem by stopping my apps, unsetting the pool, rebooting TrueNas and then selecting my pool again. The catalog updated and I was able to turn on my apps one at a time.

 

[omeganot]

I had the same error message, but didn't have a second pool to switch to. I was able to resolve my problem by stopping my apps, unsetting the pool, rebooting TrueNas and then selecting my pool again. The catalog updated and I was able to turn on my apps one at a time.

I too have only one pool.

When I do what you did, when I go back to apps and select the pool I get:

Code:

Error: [EINVAL] kubernetes_update.force: Missing '/mnt/data/ix-applications/config.json' configuration file. Specify force to override this and let system re-initialize applications.

This is after I had tried to set the pool before, and left the ix-applications dataset in place.

I have to cancel pool selection to move on. Once I do, I try to force the update by changing the CIDR and DNS values for Kubernetes by shifting the IP address range and checking the "Force" box. The values seem to update, but the same error comes up. Restart, try to set the app pool again and same error about the config persists and continues with another force update in advanced settings.

I confirm the pool is unset, then delete the ix-applications dataset. Reboot. Go back to apps, select the pool and I get:

Code:

Error: [EFAULT] Missing 'cpuset, cpu, hugetlb' cgroup controller(s) which are required for apps to function

Clear that message, pool selection is still up and when I try again to select the pool the Kubernetes config error at the top comes back up. Close pool selection without selecting anything, and apps thinks the pool is set (I have Unset Pool as a menu option and the ix-applications folder is created). If I try to install an app, I get:

Code:

[EFAULT] Kubernetes service is not running.

I feel like there's something I need to set or configure in the shell that got screwed up somewhere, but I have no idea what it is. I'm definitely not able to resolve this through the UI.

 

[ralphte]

Had the same issue and tracked it down to a problem starting the docker service. I SSH to my host and used nano and modified the docker service file

nano /lib/systemd/system/docker.service

I changed the ExecStart line to the following.
ExecStart=/usr/bin/dockerd -H unix:// --containerd=/run/containerd/containerd.sock

I then rebooted the server and everything was back to normal. Your mileage may vary but this should not hurt your system.

 

[omeganot]

@ralphte - Appreciate the tip. I made the same change, unset the pool and cleared the dataset to start from scratch. Upon attempting to set the pool and bring kubernetes up, I eventually get an error stating "Docker services is not running" which is different than my previous error of the kubernetes service not running, but even more odd with your fix to try and start the docker service.

I checked

Code:

lib/systemd/system/kube-router.service

and it's current contents are:

Code:

[Unit]
Description=Kube-router CNI
Documentation=https://github.com/cloudnative/kube-router
Wants=network-online.target

[Install]
WantedBy=multi-user.target

[Service]
Type=exec
TimeoutStartSec=0
Restart=always
RestartSec=5s
Environment="NODE_NAME=ix-truenas"
Environment="KUBE_ROUTER_CNI_CONF_FILE=/etc/cni/net.d/10-kuberouter.conflist"
ExecStart=/usr/local/bin/kube-router \
        '--run-router=true' \
        '--run-firewall=true' \
        '--run-service-proxy=true' \
        '--bgp-graceful-restart=true' \
        '--kubeconfig=/etc/cni/net.d/kube-router.d/kubeconfig' \
        '--enable-overlay=false' \

 

[hunacfunac]

Had the same issue and tracked it down to a problem starting the docker service. I SSH to my host and used nano and modified the docker service file

nano /lib/systemd/system/docker.service

I changed the ExecStart line to the following.
ExecStart=/usr/bin/dockerd -H unix:// --containerd=/run/containerd/containerd.sock

I then rebooted the server and everything was back to normal. Your mileage may vary but this should not hurt your system.

Hey thanks so much for taking the time to write this up. I couldn't find a solution anywhere else. I had this exact problem, followed your directions, and the problem was solved as described. Not every day does troubleshooting go so smoothly.

 

[danb35]

Interesting. I'm running SCALE 22.12.2, and decided to reboot in hopes that Traefik would pick up the renewed Let's Encrypt cert. Instead, none of my apps are running, and I'm getting the same message ("Failed to configure kubernetes cluster for Applications: Missing 'cpuset, cpu, hugetlb' cgroup controller(s) which are required for apps to function") in my alerts.

Interestingly, a search of this forum on the error message didn't find this thread, but a Google search found it right away. So unset pool, reboot, choose pool to select another pool, and like @omeganot, I get the same error ("Failed to configure kubernetes cluster..."). Deleted the ix-applications dataset from that pool.

Tried @ralphte's suggestion, editing docker.service. Rebooted. Back to choose pool, choose the pool I chose above, same error again ("Failed to configure kubernetes cluster..."). Tried setting it to my "normal" apps pool--same error there too.

@morganL, there was another thread where you were asking for systems where apps didn't start, but I don't find it right now--where is that?

 

[morganL]

Interesting. I'm running SCALE 22.12.2, and decided to reboot in hopes that Traefik would pick up the renewed Let's Encrypt cert. Instead, none of my apps are running, and I'm getting the same message ("Failed to configure kubernetes cluster for Applications: Missing 'cpuset, cpu, hugetlb' cgroup controller(s) which are required for apps to function") in my alerts.

Interestingly, a search of this forum on the error message didn't find this thread, but a Google search found it right away. So unset pool, reboot, choose pool to select another pool, and like @omeganot, I get the same error ("Failed to configure kubernetes cluster..."). Deleted the ix-applications dataset from that pool.

Tried @ralphte's suggestion, editing docker.service. Rebooted. Back to choose pool, choose the pool I chose above, same error again ("Failed to configure kubernetes cluster..."). Tried setting it to my "normal" apps pool--same error there too.

@morganL, there was another thread where you were asking for systems where apps didn't start, but I don't find it right now--where is that?

The thread was here: https://www.truenas.com/community/threads/looking-for-bluefin-systems-where-apps-dont-start.106968/

We're going to make 22.12.3 available next week. There is one bugfix that is likely to be relevant.

[NAS-120955] - iXsystems Jira​

This bugfix forces Apps starting to wait for the cgroups in kernel to be available. We think this is the cause of many customer issues. The behaviour varied based on the system build.

As a "hall of Famer"... please report a bug.. if you are willing to use the pre-release nightly, we can verify and otherwise help you resolve.