Applications are not running/k3s not starting

[ilembnor]

TrueNAS-SCALE-22.12.3.2
Xeon X5675
evga x58 classified MB
18 GiB ram

Hi, for a week I have been trying to figure this out with no luck. My apps all disappeared and in their place was this message:

A search told me that the message was from k3s failing to start.

@truenas[~]# systemctl start k3s
Job for k3s.service failed because the control process exited with error code.
See "systemctl status k3s.service" and "journalctl -xeu k3s.service" for details.

k3s_daemon.log snippet:

Jul 19 17:21:19 truenas k3s[11787]: W0719 17:21:19.611695 11787 reflector.go:424] storage/cacher.go:/apiextensions.k8s.io/customresourcedefinitions: failed to list *apiextensions.CustomResourceDefinition: couldn't get version/kind; json parse error: invalid character ';' after object key
Jul 19 17:21:19 truenas k3s[11787]: E0719 17:21:19.611726 11787 cacher.go:440] cacher (*apiextensions.CustomResourceDefinition): unexpected ListAndWatch error: failed to list *apiextensions.CustomResourceDefinition: couldn't get version/kind; json parse error: invalid character ';' after object key; reinitializing...
Jul 19 17:21:20 truenas k3s[11787]: W0719 17:21:20.662290 11787 reflector.go:424] storage/cacher.go:/apiextensions.k8s.io/customresourcedefinitions: failed to list *apiextensions.CustomResourceDefinition: couldn't get version/kind; json parse error: invalid character ';' after object key
Jul 19 17:21:20 truenas k3s[11787]: E0719 17:21:20.662323 11787 cacher.go:440] cacher (*apiextensions.CustomResourceDefinition): unexpected ListAndWatch error: failed to list *apiextensions.CustomResourceDefinition: couldn't get version/kind; json parse error: invalid character ';' after object key; reinitializing...
Jul 19 17:21:21 truenas k3s[11787]: W0719 17:21:21.701995 11787 reflector.go:424] storage/cacher.go:/apiextensions.k8s.io/customresourcedefinitions: failed to list *apiextensions.CustomResourceDefinition: couldn't get version/kind; json parse error: invalid character ';' after object key
Jul 19 17:21:21 truenas k3s[11787]: E0719 17:21:21.702022 11787 cacher.go:440] cacher (*apiextensions.CustomResourceDefinition): unexpected ListAndWatch error: failed to list *apiextensions.CustomResourceDefinition: couldn't get version/kind; json parse error: invalid character ';' after object key; reinitializing...
Jul 19 17:21:22 truenas k3s[11787]: W0719 17:21:22.778184 11787 reflector.go:424] storage/cacher.go:/apiextensions.k8s.io/customresourcedefinitions: failed to list *apiextensions.CustomResourceDefinition: couldn't get version/kind; json parse error: invalid character ';' after object key
Jul 19 17:21:22 truenas k3s[11787]: E0719 17:21:22.778223 11787 cacher.go:440] cacher (*apiextensions.CustomResourceDefinition): unexpected ListAndWatch error: failed to list *apiextensions.CustomResourceDefinition: couldn't get version/kind; json parse error: invalid character ';' after object key; reinitializing...
Jul 19 17:21:23 truenas k3s[11787]: time="2023-07-19T17:21:23-07:00" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
Jul 19 17:21:23 truenas k3s[11787]: W0719 17:21:23.828970 11787 reflector.go:424] storage/cacher.go:/apiextensions.k8s.io/customresourcedefinitions: failed to list *apiextensions.CustomResourceDefinition: couldn't get version/kind; json parse error: invalid character ';' after object key
Jul 19 17:21:23 truenas k3s[11787]: E0719 17:21:23.828999 11787 cacher.go:440] cacher (*apiextensions.CustomResourceDefinition): unexpected ListAndWatch error: failed to list *apiextensions.CustomResourceDefinition: couldn't get version/kind; json parse error: invalid character ';' after object key; reinitializing...
Jul 19 17:21:23 truenas k3s[11787]: time="2023-07-19T17:21:23-07:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:6443/v1-k3s/readyz: 500 Internal Server Error"

I've tried a number of things from other posts including:
unset pool/restart/set pool
made sure my system clock is set properly
deleted the token file in ix-applications/k3s/server

Any advice would be appreciated, thank you.

 

[sos_nz]

I had a similar issue a couple of weeks ago - turned out I'd messed up networking when I created a bridge interface. Make sure your installation can access WAN.

 

[ilembnor]

I had a similar issue a couple of weeks ago - turned out I'd messed up networking when I created a bridge interface. Make sure your installation can access WAN.

Thanks for the reply, my network did have WAN access. In the end a combination of reverting to 22.12.2 and restoring a backup with heavy_script got k3s running again.

 

[daymr]

I too experience the same behavior upgrading from 22.12.3 to 22.12.3.3. I have tried everything I could find on the forums (this same thing happened last upgrade and I had to wipe the ix-applicaitons dataset). I did not feel like reinstalling and reconfiguring 19 containers. so I too chose to simply rollback versions. and VOILA! it magically works.

seriously considering using truenas only for storage at this point and migrating to something better for containers. One should not have to rebuild their entire infrastructure every minor upgrade on the stable chain. It's enough to drive one mad!

 

[tangles]

Agreed.
I got burnt with updates earlier on. Now I use VMs on TrueNAS that have their own dedicated NICs/IPs on my subnets and use NFS when the VMs need access to my storage on TrueNAS. It's a shame because there's so many nice applications that I'd like to try out and see what they're like and TrueNAS seemed (at the time) like a nice tool to quickly spin up an instance to play with and explore.
Still, I love TrueNAS for storage, not so much for SaaS.