I don't know what happens, I Just restarted my server and now the whole app system is not working
Code:
root@beta-server[~]# k3s kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system coredns-d76bd69b-wxxxn 0/1 Error 114 5d21h kube-system openebs-zfs-controller-0 0/5 Error 151 5d21h ix-shinobi-nvr shinobi-nvr-ix-chart-7c5b55b6c7-mplj4 0/1 Terminating 0 4d17h kube-system openebs-zfs-node-bfjbk 1/2 CrashLoopBackOff 26 (2m17s ago) 5d21h
Code:
root@beta-server[~]# k3s kubectl get nodes -A NAME STATUS ROLES AGE VERSION ix-truenas NotReady control-plane,master 5d21h v1.23.5+k3s-fbfa51e5-dirty
Code:
root@beta-server[~]# k3s kubectl describe nodes ix-truenas Name: ix-truenas Roles: control-plane,master Labels: beta.kubernetes.io/arch=amd64 beta.kubernetes.io/os=linux kubernetes.io/arch=amd64 kubernetes.io/hostname=ix-truenas kubernetes.io/os=linux node-role.kubernetes.io/control-plane=true node-role.kubernetes.io/master=true openebs.io/nodeid=ix-truenas openebs.io/nodename=ix-truenas Annotations: csi.volume.kubernetes.io/nodeid: {"zfs.csi.openebs.io":"ix-truenas"} k3s.io/node-args: ["server","--cluster-cidr","172.16.0.0/16","--cluster-dns","172.17.0.10","--data-dir","/mnt/Main pool/ix-applications/k3s","--kube-apiserv... k3s.io/node-config-hash: XYYS4OMZPCCXD42QHOZBQ7S2EHLKXRGUUQZYYV562LD2Q5LW53TQ==== k3s.io/node-env: {"K3S_DATA_DIR":"/mnt/Main pool/ix-applications/k3s/data/ff6f39c3a41a6a1bacb2d725eceedfc101fb13e2966798b12e735eae4ffe665b"} node.alpha.kubernetes.io/ttl: 0 volumes.kubernetes.io/controller-managed-attach-detach: true CreationTimestamp: Sun, 26 Jun 2022 15:51:10 +0500 Taints: node.kubernetes.io/not-ready:NoSchedule Unschedulable: false Lease: HolderIdentity: ix-truenas AcquireTime: <unset> RenewTime: Sat, 02 Jul 2022 13:43:16 +0500 Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- MemoryPressure False Sat, 02 Jul 2022 13:39:13 +0500 Sat, 02 Jul 2022 12:05:05 +0500 KubeletHasSufficientMemory kubelet has sufficient memory available DiskPressure False Sat, 02 Jul 2022 13:39:13 +0500 Sat, 02 Jul 2022 12:05:05 +0500 KubeletHasNoDiskPressure kubelet has no disk pressure PIDPressure False Sat, 02 Jul 2022 13:39:13 +0500 Sat, 02 Jul 2022 12:05:05 +0500 KubeletHasSufficientPID kubelet has sufficient PID available Ready False Sat, 02 Jul 2022 13:39:13 +0500 Sat, 02 Jul 2022 12:23:18 +0500 KubeletNotReady container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized Addresses: InternalIP: 192.168.1.250 Hostname: ix-truenas Capacity: cpu: 4 ephemeral-storage: 4110298624Ki hugepages-2Mi: 0 memory: 8094644Ki pods: 250 Allocatable: cpu: 4 ephemeral-storage: 3998498498292 hugepages-2Mi: 0 memory: 8094644Ki pods: 250 System Info: Machine ID: 8939c8e5d8154934ab69f390561fe785 System UUID: 30303734-3536-5a43-3133-303830313357 Boot ID: b7ab9ed1-2336-4ff6-8950-9fea5daa5e48 Kernel Version: 5.10.120+truenas OS Image: Debian GNU/Linux 11 (bullseye) Operating System: linux Architecture: amd64 Container Runtime Version: docker://Unknown Kubelet Version: v1.23.5+k3s-fbfa51e5-dirty Kube-Proxy Version: v1.23.5+k3s-fbfa51e5-dirty PodCIDR: 172.16.0.0/16 PodCIDRs: 172.16.0.0/16 Non-terminated Pods: (4 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age --------- ---- ------------ ---------- --------------- ------------- --- kube-system coredns-d76bd69b-wxxxn 100m (2%) 0 (0%) 70Mi (0%) 170Mi (2%) 5d21h kube-system openebs-zfs-controller-0 0 (0%) 0 (0%) 0 (0%) 0 (0%) 5d21h ix-shinobi-nvr shinobi-nvr-ix-chart-7c5b55b6c7-mplj4 0 (0%) 0 (0%) 0 (0%) 0 (0%) 4d17h kube-system openebs-zfs-node-bfjbk 0 (0%) 0 (0%) 0 (0%) 0 (0%) 5d21h Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits -------- -------- ------ cpu 100m (2%) 0 (0%) memory 70Mi (0%) 170Mi (2%) ephemeral-storage 0 (0%) 0 (0%) hugepages-2Mi 0 (0%) 0 (0%) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal NodeHasSufficientMemory 98m (x16 over 5d21h) kubelet Node ix-truenas status is now: NodeHasSufficientMemory Normal NodeHasNoDiskPressure 98m (x16 over 5d21h) kubelet Node ix-truenas status is now: NodeHasNoDiskPressure Normal NodeHasSufficientPID 98m (x16 over 5d21h) kubelet Node ix-truenas status is now: NodeHasSufficientPID Normal NodeReady 98m (x17 over 5d21h) kubelet Node ix-truenas status is now: NodeReady Normal Starting 79m kubelet Starting kubelet. Normal NodeHasSufficientMemory 79m kubelet Node ix-truenas status is now: NodeHasSufficientMemory Normal NodeHasNoDiskPressure 79m kubelet Node ix-truenas status is now: NodeHasNoDiskPressure Normal NodeHasSufficientPID 79m kubelet Node ix-truenas status is now: NodeHasSufficientPID Warning Rebooted 79m kubelet Node ix-truenas has been rebooted, boot id: d3cd51dc-95a5-400e-b922-808175797f94 Normal NodeNotReady 79m kubelet Node ix-truenas status is now: NodeNotReady Normal NodeAllocatableEnforced 79m kubelet Updated Node Allocatable limit across pods Normal Starting 64m kubelet Starting kubelet. Normal NodeHasSufficientMemory 64m kubelet Node ix-truenas status is now: NodeHasSufficientMemory Normal NodeHasNoDiskPressure 64m kubelet Node ix-truenas status is now: NodeHasNoDiskPressure Normal NodeHasSufficientPID 64m kubelet Node ix-truenas status is now: NodeHasSufficientPID Warning Rebooted 64m kubelet Node ix-truenas has been rebooted, boot id: b7ab9ed1-2336-4ff6-8950-9fea5daa5e48 Normal NodeAllocatableEnforced 64m kubelet Updated Node Allocatable limit across pods Normal Starting 34m kubelet Starting kubelet. Normal NodeAllocatableEnforced 34m kubelet Updated Node Allocatable limit across pods Normal NodeHasSufficientMemory 34m kubelet Node ix-truenas status is now: NodeHasSufficientMemory Normal NodeHasNoDiskPressure 34m kubelet Node ix-truenas status is now: NodeHasNoDiskPressure Normal NodeHasSufficientPID 34m kubelet Node ix-truenas status is now: NodeHasSufficientPID