Applications are not running

[Nero3000]

Hi,

I am unable to get k3s service to start. The system had an uptime of over a year beforehand, but was having trouble recently in updating one of the apps, so I rebooted the system and then got hit with the "Application are not running" screen when i look. Unable to install new ones either. Updated SCALE to the latest, and that didn't work. Have been trying all the various solutions on here:

- unset, set pool and restart - didnt work
- set to new pool and set back - didnt work (new pool did seem to work on its own though)

Below is the status output for k3s which doesn't match and other peoples issues. Hopefully someone has a thought?

Code:

root@truenas[~]# systemctl status k3s
● k3s.service - Lightweight Kubernetes
     Loaded: loaded (/lib/systemd/system/k3s.service; disabled; preset: disabled)
     Active: activating (start) since Sun 2023-07-30 21:17:18 BST; 45s ago
       Docs: https://k3s.io
    Process: 1226519 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
    Process: 1226520 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
   Main PID: 1226521 (k3s-server)
      Tasks: 104
     Memory: 625.6M
        CPU: 2min 8.814s
     CGroup: /system.slice/k3s.service
             └─1226521 "/usr/local/bin/k3s server"

Jul 30 21:17:59 truenas k3s[1226521]: E0730 21:17:59.210574 1226521 storage_rbac.go:232] unable to reconcile clusterrole.rbac.authorization.k8s.io/system:controller:statefulset-controller: exceeded maximum attempts
Jul 30 21:17:59 truenas k3s[1226521]: E0730 21:17:59.807876 1226521 storage_rbac.go:232] unable to reconcile clusterrole.rbac.authorization.k8s.io/system:controller:ttl-controller: exceeded maximum attempts
Jul 30 21:17:59 truenas k3s[1226521]: time="2023-07-30T21:17:59+01:00" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
Jul 30 21:18:00 truenas k3s[1226521]: time="2023-07-30T21:18:00+01:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:6443/v1-k3s/readyz: 500 Internal Server Error"
Jul 30 21:18:00 truenas k3s[1226521]: E0730 21:18:00.402094 1226521 storage_rbac.go:232] unable to reconcile clusterrole.rbac.authorization.k8s.io/system:controller:certificate-controller: exceeded maximum attempts
Jul 30 21:18:01 truenas k3s[1226521]: E0730 21:18:01.011501 1226521 storage_rbac.go:232] unable to reconcile clusterrole.rbac.authorization.k8s.io/system:controller:pvc-protection-controller: exceeded maximum attempts
Jul 30 21:18:01 truenas k3s[1226521]: E0730 21:18:01.611309 1226521 storage_rbac.go:232] unable to reconcile clusterrole.rbac.authorization.k8s.io/system:controller:pv-protection-controller: exceeded maximum attempts
Jul 30 21:18:02 truenas k3s[1226521]: E0730 21:18:02.212712 1226521 storage_rbac.go:232] unable to reconcile clusterrole.rbac.authorization.k8s.io/system:controller:ttl-after-finished-controller: exceeded maximum attempts
Jul 30 21:18:02 truenas k3s[1226521]: E0730 21:18:02.811431 1226521 storage_rbac.go:232] unable to reconcile clusterrole.rbac.authorization.k8s.io/system:controller:root-ca-cert-publisher: exceeded maximum attempts
Jul 30 21:18:03 truenas k3s[1226521]: E0730 21:18:03.614929 1226521 storage_rbac.go:264] unable to reconcile clusterrolebinding.rbac.authorization.k8s.io/cluster-admin: exceeded maximum attempts

 

[Elope1958]

I am having this same issue. `k3s` is pinning my CPU. I am unable to start any apps, here is the `journalctl -fu k3s` output when I am trying to restart the service. If anyone has any solutions to this, that'd be great.