Kube-router settings broken?

[eloydark]

I think that I have somehow broken my k3s configuration files/setup.

My setup is (hopefully) simple, I have a single docker-compose application that inside runs portainer with multiple docker containers spawned internally.

At some point, months ago, I got into an error state that said that CNI was not working, and in my effort to fix that, I installed flannel with this (Which in retrospect I of course regret...):

Code:

k3s kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

That of course didn't improve my problems, so I removed it by doing this:

Code:

k3s kubectl delete -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

After that, weirdly enough, I removed some problematic pods, started them again and eventually my system appeared to be working again.

Fast forward to yesterday, I upgraded to 22.02.2 and everything broke again.
Digging deeper now, I can see that the kube-router service is failing. The "/etc/cni" folder is completely empty and the kube-system pods don't startup properly:

Code:

k3s kubectl get pods -A 
NAMESPACE     NAME                                   READY   STATUS              RESTARTS        AGE
kube-system   nvidia-device-plugin-daemonset-fn6wb   0/1     Completed           0               15d
kube-system   openebs-zfs-controller-0               0/5     ContainerCreating   0               46m
kube-system   coredns-d76bd69b-8szdn                 0/1     ContainerCreating   0               46m
kube-system   openebs-zfs-node-tq4p8                 1/2     CrashLoopBackOff    16 (110s ago)   46m

I tried reconstructing the configuration files in "/etc/cni" manually - created a VM with TNS and stole the configuration files ("/etc/cni/*") from that, but the folder gets deleted on every reboot. Similarly, in the TNS WebUI, under apps, when I click into any of the settings menus, nothing happens - it's all broken.

Is there something that I am not seeing? What configuration files do I need to recreate to fix kube-router and get my pods back working?

I am not a linux beginner, but I do not have a lot of experience with kubernetes. Any feedback is welcome!

 

[morganL]

Anything you previously installed will get wiped by a software update. It's never recommended..
I doubt many people have done what you have done.... fresh install and reimport the pool might be easiest solution if no-one responds.

 

[eloydark]

Feedback for anyone unlucky enough to end up in my shoes...
My main issue was that the sqlite database under ix-applications/k3s/server/db/state.db was also messed up/corrupted, so no matter what I tried, the system could not get into a working state.
I ended up removing my apps dataset and creating a new one. Given that I had everything for my containers in a separate dataset, the process did not take that long, and I could restore everything without major problems.

I still cannot enable my nvidia gpu into the docker-compose app, but that is a topic for another day/discussion!
Thx for helping out @morganL