Node Not Ready, Network Not Ready. Everything is stuck or something

[mwaleedh]

I don't know what happens, I Just restarted my server and now the whole app system is not working

Code:

root@beta-server[~]# k3s kubectl get pods -A
NAMESPACE        NAME                                    READY   STATUS             RESTARTS         AGE
kube-system      coredns-d76bd69b-wxxxn                  0/1     Error              114              5d21h
kube-system      openebs-zfs-controller-0                0/5     Error              151              5d21h
ix-shinobi-nvr   shinobi-nvr-ix-chart-7c5b55b6c7-mplj4   0/1     Terminating        0                4d17h
kube-system      openebs-zfs-node-bfjbk                  1/2     CrashLoopBackOff   26 (2m17s ago)   5d21h

Code:

root@beta-server[~]# k3s kubectl get nodes -A
NAME         STATUS     ROLES                  AGE     VERSION
ix-truenas   NotReady   control-plane,master   5d21h   v1.23.5+k3s-fbfa51e5-dirty

Code:

root@beta-server[~]# k3s kubectl describe nodes ix-truenas
Name:               ix-truenas
Roles:              control-plane,master
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=ix-truenas
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/control-plane=true
                    node-role.kubernetes.io/master=true
                    openebs.io/nodeid=ix-truenas
                    openebs.io/nodename=ix-truenas
Annotations:        csi.volume.kubernetes.io/nodeid: {"zfs.csi.openebs.io":"ix-truenas"}
                    k3s.io/node-args:
                      ["server","--cluster-cidr","172.16.0.0/16","--cluster-dns","172.17.0.10","--data-dir","/mnt/Main pool/ix-applications/k3s","--kube-apiserv...
                    k3s.io/node-config-hash: XYYS4OMZPCCXD42QHOZBQ7S2EHLKXRGUUQZYYV562LD2Q5LW53TQ====
                    k3s.io/node-env: {"K3S_DATA_DIR":"/mnt/Main pool/ix-applications/k3s/data/ff6f39c3a41a6a1bacb2d725eceedfc101fb13e2966798b12e735eae4ffe665b"}
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Sun, 26 Jun 2022 15:51:10 +0500
Taints:             node.kubernetes.io/not-ready:NoSchedule
Unschedulable:      false
Lease:
  HolderIdentity:  ix-truenas
  AcquireTime:     <unset>
  RenewTime:       Sat, 02 Jul 2022 13:43:16 +0500
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason
    Message
  ----             ------  -----------------                 ------------------                ------
    -------
  MemoryPressure   False   Sat, 02 Jul 2022 13:39:13 +0500   Sat, 02 Jul 2022 12:05:05 +0500   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Sat, 02 Jul 2022 13:39:13 +0500   Sat, 02 Jul 2022 12:05:05 +0500   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Sat, 02 Jul 2022 13:39:13 +0500   Sat, 02 Jul 2022 12:05:05 +0500   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            False   Sat, 02 Jul 2022 13:39:13 +0500   Sat, 02 Jul 2022 12:23:18 +0500   KubeletNotReady
    container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Addresses:
  InternalIP:  192.168.1.250
  Hostname:    ix-truenas
Capacity:
  cpu:                4
  ephemeral-storage:  4110298624Ki
  hugepages-2Mi:      0
  memory:             8094644Ki
  pods:               250
Allocatable:
  cpu:                4
  ephemeral-storage:  3998498498292
  hugepages-2Mi:      0
  memory:             8094644Ki
  pods:               250
System Info:
  Machine ID:                 8939c8e5d8154934ab69f390561fe785
  System UUID:                30303734-3536-5a43-3133-303830313357
  Boot ID:                    b7ab9ed1-2336-4ff6-8950-9fea5daa5e48
  Kernel Version:             5.10.120+truenas
  OS Image:                   Debian GNU/Linux 11 (bullseye)
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  docker://Unknown
  Kubelet Version:            v1.23.5+k3s-fbfa51e5-dirty
  Kube-Proxy Version:         v1.23.5+k3s-fbfa51e5-dirty
PodCIDR:                      172.16.0.0/16
PodCIDRs:                     172.16.0.0/16
Non-terminated Pods:          (4 in total)
  Namespace                   Name                                     CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                                     ------------  ----------  ---------------  -------------  ---
  kube-system                 coredns-d76bd69b-wxxxn                   100m (2%)     0 (0%)      70Mi (0%)        170Mi (2%)     5d21h
  kube-system                 openebs-zfs-controller-0                 0 (0%)        0 (0%)      0 (0%)           0 (0%)         5d21h
  ix-shinobi-nvr              shinobi-nvr-ix-chart-7c5b55b6c7-mplj4    0 (0%)        0 (0%)      0 (0%)           0 (0%)         4d17h
  kube-system                 openebs-zfs-node-bfjbk                   0 (0%)        0 (0%)      0 (0%)           0 (0%)         5d21h
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests   Limits
  --------           --------   ------
  cpu                100m (2%)  0 (0%)
  memory             70Mi (0%)  170Mi (2%)
  ephemeral-storage  0 (0%)     0 (0%)
  hugepages-2Mi      0 (0%)     0 (0%)
Events:
  Type     Reason                   Age                   From     Message
  ----     ------                   ----                  ----     -------
  Normal   NodeHasSufficientMemory  98m (x16 over 5d21h)  kubelet  Node ix-truenas status is now: NodeHasSufficientMemory
  Normal   NodeHasNoDiskPressure    98m (x16 over 5d21h)  kubelet  Node ix-truenas status is now: NodeHasNoDiskPressure
  Normal   NodeHasSufficientPID     98m (x16 over 5d21h)  kubelet  Node ix-truenas status is now: NodeHasSufficientPID
  Normal   NodeReady                98m (x17 over 5d21h)  kubelet  Node ix-truenas status is now: NodeReady
  Normal   Starting                 79m                   kubelet  Starting kubelet.
  Normal   NodeHasSufficientMemory  79m                   kubelet  Node ix-truenas status is now: NodeHasSufficientMemory
  Normal   NodeHasNoDiskPressure    79m                   kubelet  Node ix-truenas status is now: NodeHasNoDiskPressure
  Normal   NodeHasSufficientPID     79m                   kubelet  Node ix-truenas status is now: NodeHasSufficientPID
  Warning  Rebooted                 79m                   kubelet  Node ix-truenas has been rebooted, boot id: d3cd51dc-95a5-400e-b922-808175797f94
  Normal   NodeNotReady             79m                   kubelet  Node ix-truenas status is now: NodeNotReady
  Normal   NodeAllocatableEnforced  79m                   kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 64m                   kubelet  Starting kubelet.
  Normal   NodeHasSufficientMemory  64m                   kubelet  Node ix-truenas status is now: NodeHasSufficientMemory
  Normal   NodeHasNoDiskPressure    64m                   kubelet  Node ix-truenas status is now: NodeHasNoDiskPressure
  Normal   NodeHasSufficientPID     64m                   kubelet  Node ix-truenas status is now: NodeHasSufficientPID
  Warning  Rebooted                 64m                   kubelet  Node ix-truenas has been rebooted, boot id: b7ab9ed1-2336-4ff6-8950-9fea5daa5e48
  Normal   NodeAllocatableEnforced  64m                   kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 34m                   kubelet  Starting kubelet.
  Normal   NodeAllocatableEnforced  34m                   kubelet  Updated Node Allocatable limit across pods
  Normal   NodeHasSufficientMemory  34m                   kubelet  Node ix-truenas status is now: NodeHasSufficientMemory
  Normal   NodeHasNoDiskPressure    34m                   kubelet  Node ix-truenas status is now: NodeHasNoDiskPressure
  Normal   NodeHasSufficientPID     34m                   kubelet  Node ix-truenas status is now: NodeHasSufficientPID

 

[brando56894]

Yeah I have no idea either, IMO K3S was a bad choice for the Apps system, pure Docker would have been a lot easier to manage.

My suggestion is to hit up the TrueCharts Discord, since it's most likely their apps that you're using anyway. Either way they'll get you straightened out more quickly than posting here will. They've gotten me out a jam a few times.

 

[mwaleedh]

Agreed, k3s was a bad idea, instead, they should give a choice like if you are running just one standalone server then allow people to use docker and docker-compose but if you have multiple servers then Kubernetes should be activated or something. Anyways, I'll try discord. Thanks for your help.

Yeah I have no idea either, IMO K3S was a bad choice for the Apps system, pure Docker would have been a lot easier to manage.

My suggestion is to hit up the TrueCharts Discord, since it's most likely their apps that you're using anyway. Either way they'll get you straightened out more quickly than posting here will. They've gotten me out a jam a few times.

 

[FabrizioR8]

Just ran into this same issue in my bluefin dev system... doesn't seem like a user malfunction.

Any luck from the TrueCharts Discord?

Anyone know how to completely reset the apps subsystems without re-installing and reconfiguring everything again?

root@tndev[/var/log]# k3s kubectl describe node -A Name: ix-truenas Roles: control-plane,master Labels: beta.kubernetes.io/arch=amd64 beta.kubernetes.io/os=linux egress.k3s.io/cluster=true kubernetes.io/arch=amd64 kubernetes.io/hostname=ix-truenas kubernetes.io/os=linux node-role.kubernetes.io/control-plane=true node-role.kubernetes.io/master=true openebs.io/nodeid=ix-truenas openebs.io/nodename=ix-truenas Annotations: csi.volume.kubernetes.io/nodeid: {"zfs.csi.openebs.io":"ix-truenas"} k3s.io/node-args: ["server","--cluster-cidr","172.16.0.0/16","--cluster-dns","172.17.0.10","--data-dir","/mnt/data/ix-applications/k3s","--kube-apiserver-ar... k3s.io/node-config-hash: EA7ZADJ5GDHT3SIHE5YAR43FX2I2VPWBQQQQT7ZRUDLN42QMRLFQ==== k3s.io/node-env: {"K3S_DATA_DIR":"/mnt/data/ix-applications/k3s/data/3622b68605bf9a49243ca44f328acbb9e681a95150de9d9f94baaba1d5511603"} node.alpha.kubernetes.io/ttl: 0 volumes.kubernetes.io/controller-managed-attach-detach: true CreationTimestamp: Mon, 18 Jul 2022 17:05:28 -0700 Taints: node.kubernetes.io/not-ready:NoSchedule Unschedulable: false Lease: HolderIdentity: ix-truenas AcquireTime: <unset> RenewTime: Fri, 06 Jan 2023 10:43:32 -0800 Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- MemoryPressure False Fri, 06 Jan 2023 10:42:06 -0800 Tue, 13 Dec 2022 23:11:04 -0800 KubeletHasSufficientMemory kubelet has sufficient memory available DiskPressure False Fri, 06 Jan 2023 10:42:06 -0800 Tue, 13 Dec 2022 23:11:04 -0800 KubeletHasNoDiskPressure kubelet has no disk pressure PIDPressure False Fri, 06 Jan 2023 10:42:06 -0800 Tue, 13 Dec 2022 23:11:04 -0800 KubeletHasSufficientPID kubelet has sufficient PID available Ready False Fri, 06 Jan 2023 10:42:06 -0800 Thu, 05 Jan 2023 10:31:07 -0800 KubeletNotReady container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized Addresses: InternalIP: 192.168.4.88 Hostname: ix-truenas Capacity: cpu: 8 ephemeral-storage: 14163712Ki hugepages-2Mi: 0 memory: 32881392Ki pods: 250 Allocatable: cpu: 8 ephemeral-storage: 13778459023 hugepages-2Mi: 0 memory: 32881392Ki pods: 250 System Info: Machine ID: 72bddb19b77a471191af0014b9ac647f System UUID: ed1f82af-9dcd-7f47-9eee-d4d5905b8422 Boot ID: 5268f03d-d5ae-41dd-9848-a3aa76ff55c2 Kernel Version: 5.15.79+truenas OS Image: Debian GNU/Linux 11 (bullseye) Operating System: linux Architecture: amd64 Container Runtime Version: docker://Unknown Kubelet Version: v1.25.3+k3s-9afcd6b9-dirty Kube-Proxy Version: v1.25.3+k3s-9afcd6b9-dirty PodCIDR: 172.16.0.0/16 PodCIDRs: 172.16.0.0/16 Non-terminated Pods: (0 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age --------- ---- ------------ ---------- --------------- ------------- --- Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits -------- -------- ------ cpu 0 (0%) 0 (0%) memory 0 (0%) 0 (0%) ephemeral-storage 0 (0%) 0 (0%) hugepages-2Mi 0 (0%) 0 (0%) Events: <none> root@tndev[/var/log]# k3s kubectl get pods -A -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kube-system openebs-zfs-controller-0 0/5 Pending 0 22h <none> <none> <none> <none> kube-system coredns-57d9466bf6-rg2tg 0/1 Pending 0 22h <none> <none> <none> <none> kube-system helm-delete-traefik-crd-z4m8f 0/1 Pending 0 22h <none> <none> <none> <none> kube-system helm-delete-traefik-sm66j 0/1 Pending 0 22h <none> <none> <none> <none>

 

[NikoliHanson]

I had this exact same problem happen to me as I was trying to upgrade pihole today, just came across you post. I was doing my own research into kubernetes, as I have zero expereince with it and could not get the pod to terminate.

I came across this website https://lepczynski.it/en/k8s_en/k8s-pods-stuck-on-terminating/ and specifically this command

"kubectl delete pod POD_NAME --grace-period=0 --force -n NAMESPACE_NAME"

that worked for me (I added k3s to the beginning of the command "k3s kubectl delete pod POD_NAME --grace-period=0 --force -n NAMESPACE_NAME")

It does give a warning that "Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely."

However it has removed the "Terminating" status which allowed me to reinstall the app. (My Problem was that the Terminating status didn't allow for the writing of new helm secrets, but I have no idea what any of that means)

Use at your own risk and make sure it's what is needed to solve your problem, just letting you know what worked for me in a similar situation and try and point you in the right direction.

 

[FabrizioR8]

I tried that... new pods fired up and never became active. something in /mnt/vpool1/ix-applications (or whatever the folder name is - memory unclear) was hosed. I re-installed bluefin and reformatted the pool drives cleanly. Still setting up baseline vpool/system config from scratch and haven't enabled apps yet. Too slow on my dev box. 1/5/15 minute load-averages with apps enabled (just cluster, no actual apps) is up in the 15-20 active processes range full-time.