Many apps have 2 pods after restart

[dgrigo]

Hello, This is ongoing on every release of scale that after reboot apps have 2 pods on workloads, from which 1 only working .

I run manual sudo k3s kubectl get pod --all-namespaces | awk '{if ($4 != "Running") system ("sudo k3s kubectl -n " $1 " delete pods " $2 " --grace-period=0 " " --force ")}' and that gets rid of the ghost and goes back to normal 1 POD.
Is this an ongoing bug? Are there any workaround to fix it?
i run this command as post init script with a delay of 300

Thanks

Dinos

 

[zer0vector]

I've been messing around with deploying some stuff directly in k3s on Truenas Scale, using kubectl, and I'm running into this same or similar problem. After a reboot a number of pods are created, fail to actually boot, and then go into 'TaintToleration' status. The deployment creates another replica and that one usually boots fine but the original failed pods stick around. There's nothing actually running there, just a record that it failed, but it's still annoying.

Looking into the cluster events it seems like the system is attempting to boot pods before the node is ready. I'm seeing a bunch of

Code:

Predicate TaintToleration failed

and

Code:

MountVolume.SetUp failed for volume...

and then about 10 seconds later the node actually comes up

Code:

Node ix-truenas status is now: NodeReady

After that the pods start up normally. I know just enough about kubernetes to be dangerous here, but it seems odd that containers are trying to start before the node is ready. I'll keep investigating and post anything here that I find.