+--------------------------------------------------------------------------------+
+                      Kubernetes Configuration @1656907166                      +
+--------------------------------------------------------------------------------+
{
  "id": 1,
  "pool": "pa",
  "cluster_cidr": "172.16.0.0/16",
  "service_cidr": "172.17.0.0/16",
  "cluster_dns_ip": "172.17.0.10",
  "route_v4_interface": null,
  "route_v4_gateway": null,
  "route_v6_interface": null,
  "route_v6_gateway": null,
  "node_ip": "0.0.0.0",
  "configure_gpus": true,
  "servicelb": true,
  "dataset": "pa/ix-applications"
}
debug finished in 0 seconds for Kubernetes Configuration


+--------------------------------------------------------------------------------+
+                     k3s kubectl describe nodes @1656907166                     +
+--------------------------------------------------------------------------------+
Name:               ix-truenas
Roles:              control-plane,master
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=ix-truenas
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/control-plane=true
                    node-role.kubernetes.io/master=true
                    openebs.io/nodeid=ix-truenas
                    openebs.io/nodename=ix-truenas
Annotations:        csi.volume.kubernetes.io/nodeid: {"zfs.csi.openebs.io":"ix-truenas"}
                    k3s.io/node-args:
                      ["server","--cluster-cidr","172.16.0.0/16","--cluster-dns","172.17.0.10","--data-dir","/mnt/pa/ix-applications/k3s","--kube-apiserver-arg"...
                    k3s.io/node-config-hash: O5GMF6GSK6PNBKPAHKUZPG2VDAT53ZZUD6LTPHRGHKF6JOIAKPDA====
                    k3s.io/node-env: {"K3S_DATA_DIR":"/mnt/pa/ix-applications/k3s/data/ff6f39c3a41a6a1bacb2d725eceedfc101fb13e2966798b12e735eae4ffe665b"}
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Sat, 13 Nov 2021 16:57:55 +0800
Taints:             <none>
Unschedulable:      false
Lease:
  HolderIdentity:  ix-truenas
  AcquireTime:     <unset>
  RenewTime:       Mon, 04 Jul 2022 11:59:23 +0800
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Mon, 04 Jul 2022 11:55:19 +0800   Sat, 13 Nov 2021 16:57:53 +0800   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Mon, 04 Jul 2022 11:55:19 +0800   Sat, 13 Nov 2021 16:57:53 +0800   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Mon, 04 Jul 2022 11:55:19 +0800   Sat, 13 Nov 2021 16:57:53 +0800   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Mon, 04 Jul 2022 11:55:19 +0800   Mon, 04 Jul 2022 02:04:21 +0800   KubeletReady                 kubelet is posting ready status. AppArmor enabled
Addresses:
  InternalIP:  192.168.1.5
  Hostname:    ix-truenas
Capacity:
  cpu:                64
  ephemeral-storage:  1314970656Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             131887540Ki
  pods:               250
Allocatable:
  cpu:                64
  ephemeral-storage:  1279203453154
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             131887540Ki
  pods:               250
System Info:
  Machine ID:                 dab61aa5364d4ef795fed602a0311a96
  System UUID:                00000000-0000-0000-0000-ac1f6be82a88
  Boot ID:                    4282e2b6-d398-4c22-9423-4d8c441b8577
  Kernel Version:             5.10.120+truenas
  OS Image:                   Debian GNU/Linux 11 (bullseye)
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  docker://Unknown
  Kubelet Version:            v1.23.5+k3s-fbfa51e5-dirty
  Kube-Proxy Version:         v1.23.5+k3s-fbfa51e5-dirty
PodCIDR:                      172.16.0.0/16
PodCIDRs:                     172.16.0.0/16
Non-terminated Pods:          (9 in total)
  Namespace                   Name                             CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                             ------------  ----------  ---------------  -------------  ---
  kube-system                 coredns-d76bd69b-ks26g           100m (0%)     0 (0%)      70Mi (0%)        170Mi (0%)     9h
  ix-openldap                 svclb-openldap-ldaps-vzm7c       0 (0%)        0 (0%)      0 (0%)           0 (0%)         9h
  ix-aria2                    svclb-aria2-listen-5zljr         0 (0%)        0 (0%)      0 (0%)           0 (0%)         9h
  ix-aria2                    svclb-aria2-fsw89                0 (0%)        0 (0%)      0 (0%)           0 (0%)         9h
  ix-ariang                   svclb-ariang-custom-app-svbxp    0 (0%)        0 (0%)      0 (0%)           0 (0%)         9h
  ix-openldap                 svclb-openldap-4rjmb             0 (0%)        0 (0%)      0 (0%)           0 (0%)         9h
  ix-jellyfin                 svclb-jellyfin-g8sj9             0 (0%)        0 (0%)      0 (0%)           0 (0%)         9h
  kube-system                 openebs-zfs-node-5zm8n           0 (0%)        0 (0%)      0 (0%)           0 (0%)         9h
  kube-system                 openebs-zfs-controller-0         0 (0%)        0 (0%)      0 (0%)           0 (0%)         9h
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests   Limits
  --------           --------   ------
  cpu                100m (0%)  0 (0%)
  memory             70Mi (0%)  170Mi (0%)
  ephemeral-storage  0 (0%)     0 (0%)
  hugepages-1Gi      0 (0%)     0 (0%)
  hugepages-2Mi      0 (0%)     0 (0%)
Events:              <none>
debug finished in 3 seconds for k3s kubectl describe nodes


+--------------------------------------------------------------------------------+
+k3s kubectl get pods,svc,daemonsets,deployments,statefulset,sc,pvc,ns,job --all-namespaces -o wide @1656907169+
+--------------------------------------------------------------------------------+
NAMESPACE     NAME                                READY   STATUS             RESTARTS   AGE   IP            NODE         NOMINATED NODE   READINESS GATES
kube-system   pod/coredns-d76bd69b-ks26g          1/1     Running            0          9h    172.16.0.18   ix-truenas   <none>           <none>
ix-openldap   pod/svclb-openldap-ldaps-vzm7c      1/1     Running            0          9h    172.16.0.19   ix-truenas   <none>           <none>
ix-aria2      pod/svclb-aria2-listen-5zljr        1/1     Running            0          9h    172.16.0.20   ix-truenas   <none>           <none>
ix-aria2      pod/svclb-aria2-fsw89               1/1     Running            0          9h    172.16.0.21   ix-truenas   <none>           <none>
ix-ariang     pod/svclb-ariang-custom-app-svbxp   1/1     Running            0          9h    172.16.0.22   ix-truenas   <none>           <none>
ix-openldap   pod/svclb-openldap-4rjmb            1/1     Running            0          9h    172.16.0.23   ix-truenas   <none>           <none>
ix-jellyfin   pod/svclb-jellyfin-g8sj9            1/1     Running            0          9h    172.16.0.24   ix-truenas   <none>           <none>
kube-system   pod/openebs-zfs-node-5zm8n          1/2     ImagePullBackOff   0          9h    192.168.1.5   ix-truenas   <none>           <none>
kube-system   pod/openebs-zfs-controller-0        3/5     ImagePullBackOff   0          9h    172.16.0.25   ix-truenas   <none>           <none>

NAMESPACE     NAME                        TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                  AGE    SELECTOR
default       service/kubernetes          ClusterIP      172.17.0.1       <none>        443/TCP                  232d   <none>
kube-system   service/kube-dns            ClusterIP      172.17.0.10      <none>        53/UDP,53/TCP,9153/TCP   232d   k8s-app=kube-dns
ix-openldap   service/openldap            LoadBalancer   172.17.145.141   192.168.1.5   389:62332/TCP            169d   app.kubernetes.io/instance=openldap,app.kubernetes.io/name=openldap
ix-openldap   service/openldap-ldaps      LoadBalancer   172.17.6.182     192.168.1.5   636:63942/TCP            169d   app.kubernetes.io/instance=openldap,app.kubernetes.io/name=openldap
ix-aria2      service/aria2-listen        LoadBalancer   172.17.185.94    192.168.1.5   6888:42288/TCP           112d   app.kubernetes.io/instance=aria2,app.kubernetes.io/name=aria2
ix-aria2      service/aria2               LoadBalancer   172.17.134.236   192.168.1.5   6800:32665/TCP           112d   app.kubernetes.io/instance=aria2,app.kubernetes.io/name=aria2
ix-ariang     service/ariang-custom-app   LoadBalancer   172.17.132.154   192.168.1.5   6880:44900/TCP           112d   app.kubernetes.io/instance=ariang,app.kubernetes.io/name=custom-app
ix-jellyfin   service/jellyfin            LoadBalancer   172.17.130.42    192.168.1.5   8096:13950/TCP           112d   app.kubernetes.io/instance=jellyfin,app.kubernetes.io/name=jellyfin

NAMESPACE     NAME                                     DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE    CONTAINERS                                     IMAGES                                                                             SELECTOR
kube-system   daemonset.apps/openebs-zfs-node          1         1         0       1            0           <none>          232d   csi-node-driver-registrar,openebs-zfs-plugin   k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0,openebs/zfs-driver:2.0.0   app=openebs-zfs-node
ix-openldap   daemonset.apps/svclb-openldap-ldaps      1         1         1       1            1           <none>          169d   lb-tcp-636                                     rancher/klipper-lb:v0.3.4                                                          app=svclb-openldap-ldaps
ix-aria2      daemonset.apps/svclb-aria2               1         1         1       1            1           <none>          112d   lb-tcp-6800                                    rancher/klipper-lb:v0.3.4                                                          app=svclb-aria2
ix-aria2      daemonset.apps/svclb-aria2-listen        1         1         1       1            1           <none>          112d   lb-tcp-6888                                    rancher/klipper-lb:v0.3.4                                                          app=svclb-aria2-listen
ix-ariang     daemonset.apps/svclb-ariang-custom-app   1         1         1       1            1           <none>          112d   lb-tcp-6880                                    rancher/klipper-lb:v0.3.4                                                          app=svclb-ariang-custom-app
ix-openldap   daemonset.apps/svclb-openldap            1         1         1       1            1           <none>          169d   lb-tcp-389                                     rancher/klipper-lb:v0.3.4                                                          app=svclb-openldap
ix-jellyfin   daemonset.apps/svclb-jellyfin            1         1         1       1            1           <none>          112d   lb-tcp-8096                                    rancher/klipper-lb:v0.3.4                                                          app=svclb-jellyfin

NAMESPACE     NAME                                READY   UP-TO-DATE   AVAILABLE   AGE    CONTAINERS          IMAGES                                                                                                        SELECTOR
ix-aria2      deployment.apps/aria2               0/0     0            0           112d   aria2               tccr.io/truecharts/aria2-pro:latest@sha256:6c0ddcc7be4da69ac146ff3153df727a5818f733636a1c4d9b78ccffd6106a23   app.kubernetes.io/instance=aria2,app.kubernetes.io/name=aria2
ix-ariang     deployment.apps/ariang-custom-app   0/0     0            0           112d   ariang-custom-app   p3terx/ariang:latest                                                                                          app.kubernetes.io/instance=ariang,app.kubernetes.io/name=custom-app
ix-jellyfin   deployment.apps/jellyfin            0/0     0            0           112d   jellyfin            tccr.io/truecharts/jellyfin:v10.7.7@sha256:0136db4677a2ee2ee8a6962d813d6e3b49aa86784a7cfdc3af76427db32c3470   app.kubernetes.io/instance=jellyfin,app.kubernetes.io/name=jellyfin
kube-system   deployment.apps/coredns             1/1     1            1           232d   coredns             rancher/mirrored-coredns-coredns:1.9.1                                                                        k8s-app=kube-dns

NAMESPACE     NAME                                      READY   AGE    CONTAINERS                                                                           IMAGES
ix-openldap   statefulset.apps/openldap                 0/0     169d   openldap                                                                             tccr.io/truecharts/openldap:v1.5.0@sha256:0260d37c41f0c1207aaa642d7c786851385471a5ddf02bc6efc178241ddd8706
kube-system   statefulset.apps/openebs-zfs-controller   0/1     232d   csi-resizer,csi-snapshotter,snapshot-controller,csi-provisioner,openebs-zfs-plugin   k8s.gcr.io/sig-storage/csi-resizer:v1.2.0,k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0,k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0,k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0,openebs/zfs-driver:2.0.0

NAMESPACE   NAME                                                          PROVISIONER          RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
            storageclass.storage.k8s.io/ix-storage-class-openldap         zfs.csi.openebs.io   Retain          Immediate           true                   169d
            storageclass.storage.k8s.io/ix-storage-class-jellyfin         zfs.csi.openebs.io   Retain          Immediate           true                   112d
            storageclass.storage.k8s.io/ix-storage-class-aria2            zfs.csi.openebs.io   Retain          Immediate           true                   112d
            storageclass.storage.k8s.io/ix-storage-class-ariang           zfs.csi.openebs.io   Retain          Immediate           true                   112d
            storageclass.storage.k8s.io/openebs-zfspv-default (default)   zfs.csi.openebs.io   Delete          Immediate           true                   9h

NAMESPACE     NAME                                     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                AGE    VOLUMEMODE
ix-openldap   persistentvolumeclaim/slapd-openldap-0   Bound    pvc-8bcdf333-f203-48ca-b8d1-a17d8ebb1bff   10Gi       RWO            ix-storage-class-openldap   169d   Filesystem
ix-openldap   persistentvolumeclaim/data-openldap-0    Bound    pvc-bd1302a5-ff1b-4e73-a218-e7bd904e3c17   10Gi       RWO            ix-storage-class-openldap   169d   Filesystem
ix-jellyfin   persistentvolumeclaim/jellyfin-config    Bound    pvc-d500389b-44bb-4baa-9dd6-ba313e8b7529   999Gi      RWO            ix-storage-class-jellyfin   112d   Filesystem
ix-aria2      persistentvolumeclaim/aria2-config       Bound    pvc-d45ec44c-55c6-426c-befb-f2e822de4398   999Gi      RWO            ix-storage-class-aria2      112d   Filesystem

NAMESPACE   NAME                        STATUS   AGE
            namespace/default           Active   232d
            namespace/kube-system       Active   232d
            namespace/kube-public       Active   232d
            namespace/kube-node-lease   Active   232d
            namespace/ix-openldap       Active   169d
            namespace/ix-jellyfin       Active   112d
            namespace/ix-aria2          Active   112d
            namespace/ix-ariang         Active   112d
            namespace/openebs           Active   232d
debug finished in 1 seconds for k3s kubectl get pods,svc,daemonsets,deployments,statefulset,sc,pvc,ns,job --all-namespaces -o wide


+--------------------------------------------------------------------------------+
+         k3s kubectl describe deployments --all-namespaces @1656907170          +
+--------------------------------------------------------------------------------+
Name:               aria2
Namespace:          ix-aria2
CreationTimestamp:  Sun, 13 Mar 2022 21:56:33 +0800
Labels:             app.kubernetes.io/instance=aria2
                    app.kubernetes.io/managed-by=Helm
                    app.kubernetes.io/name=aria2
                    app.kubernetes.io/version=latest
                    helm-revision=2
                    helm.sh/chart=aria2-5.0.7
Annotations:        deployment.kubernetes.io/revision: 2
                    meta.helm.sh/release-name: aria2
                    meta.helm.sh/release-namespace: ix-aria2
Selector:           app.kubernetes.io/instance=aria2,app.kubernetes.io/name=aria2
Replicas:           0 desired | 0 updated | 0 total | 0 available | 0 unavailable
StrategyType:       Recreate
MinReadySeconds:    0
Pod Template:
  Labels:           app.kubernetes.io/instance=aria2
                    app.kubernetes.io/name=aria2
  Service Account:  default
  Init Containers:
   autopermissions:
    Image:      ghcr.io/truecharts/alpine:v3.16.0@sha256:16dc15f3d61a1e30b1df9f839e53636847b6097286b2b74c637b25fd8264f730
    Port:       <none>
    Host Port:  <none>
    Command:
      /bin/sh
      -c
      /bin/bash <<'EOF'
      echo "Automatically correcting permissions..."
      echo "Automatically correcting permissions for /mnt/pa..."
      if nfs4xdr_getfacl && nfs4xdr_getfacl | grep -qv "Failed to get NFSv4 ACL"; then
        echo "NFSv4 ACLs detected, using nfs4_setfacl to set permissions..."
        nfs4_setfacl -R -a A:g:568:RWX '/mnt/pa'
      else
        echo "No NFSv4 ACLs detected, trying chown/chmod..."
        chown -R :568 '/mnt/pa'
        chmod -R g+rwx '/mnt/pa'
      fi
      echo "increasing inotify limits..."
      ( sysctl -w fs.inotify.max_user_watches=524288 || echo "error setting inotify") && ( sysctl -w fs.inotify.max_user_instances=512 || echo "error setting inotify")
      EOF
      
    Limits:
      cpu:     4
      memory:  8Gi
    Requests:
      cpu:        10m
      memory:     50Mi
    Environment:  <none>
    Mounts:
      /mnt/pa from list-0 (rw)
  Containers:
   aria2:
    Image:       tccr.io/truecharts/aria2-pro:latest@sha256:6c0ddcc7be4da69ac146ff3153df727a5818f733636a1c4d9b78ccffd6106a23
    Ports:       6888/TCP, 6800/TCP
    Host Ports:  0/TCP, 0/TCP
    Limits:
      cpu:     4
      memory:  8Gi
    Requests:
      cpu:      10m
      memory:   50Mi
    Liveness:   tcp-socket :6800 delay=10s timeout=5s period=10s #success=1 #failure=5
    Readiness:  tcp-socket :6800 delay=10s timeout=5s period=10s #success=1 #failure=5
    Startup:    tcp-socket :6800 delay=10s timeout=2s period=5s #success=1 #failure=60
    Environment Variables from:
      aria2  Secret  Optional: false
    Environment:
      PUID:                    568
      USER_ID:                 568
      UID:                     568
      UMASK:                   "002"
      UMASK_SET:               "002"
      PGID:                    568
      GROUP_ID:                568
      GID:                     568
      NVIDIA_VISIBLE_DEVICES:  void
      TZ:                      Asia/Shanghai
      RPC_PORT:                6800
    Mounts:
      /config from config (rw)
      /mnt/pa from list-0 (rw)
      /shared from shared (rw)
      /tmp from temp (rw)
      /var/logs from varlogs (rw)
      /var/run from varrun (rw)
  Volumes:
   config:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  aria2-config
    ReadOnly:   false
   list-0:
    Type:          HostPath (bare host directory volume)
    Path:          /mnt/pa
    HostPathType:  
   shared:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
   temp:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
   varlogs:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
   varrun:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   aria2-9dc9f7bb8 (0/0 replicas created)
Events:          <none>


Name:               ariang-custom-app
Namespace:          ix-ariang
CreationTimestamp:  Sun, 13 Mar 2022 22:10:53 +0800
Labels:             app.kubernetes.io/instance=ariang
                    app.kubernetes.io/managed-by=Helm
                    app.kubernetes.io/name=custom-app
                    app.kubernetes.io/version=0.20.660
                    helm.sh/chart=custom-app-4.0.2
Annotations:        deployment.kubernetes.io/revision: 1
                    meta.helm.sh/release-name: ariang
                    meta.helm.sh/release-namespace: ix-ariang
                    rollme: E3QSP
Selector:           app.kubernetes.io/instance=ariang,app.kubernetes.io/name=custom-app
Replicas:           0 desired | 0 updated | 0 total | 0 available | 0 unavailable
StrategyType:       Recreate
MinReadySeconds:    0
Pod Template:
  Labels:           app.kubernetes.io/instance=ariang
                    app.kubernetes.io/name=custom-app
  Annotations:      rollme: 9djMs
  Service Account:  default
  Init Containers:
   autopermissions:
    Image:      ghcr.io/truecharts/alpine:v3.14.2@sha256:4095394abbae907e94b1f2fd2e2de6c4f201a5b9704573243ca8eb16db8cdb7c
    Port:       <none>
    Host Port:  <none>
    Command:
      /bin/sh
      -c
      echo 'Automatically correcting permissions...';
    Limits:
      cpu:     4
      memory:  8Gi
    Requests:
      cpu:        10m
      memory:     50Mi
    Environment:  <none>
    Mounts:       <none>
   hostpatch:
    Image:      ghcr.io/truecharts/alpine:v3.14.2@sha256:4095394abbae907e94b1f2fd2e2de6c4f201a5b9704573243ca8eb16db8cdb7c
    Port:       <none>
    Host Port:  <none>
    Command:
      /bin/sh
      -c
      ( sysctl -w fs.inotify.max_user_watches=524288 || echo "error setting inotify") && ( sysctl -w fs.inotify.max_user_instances=512 || echo "error setting inotify") && ( chmod -x /host/usr/bin/docker-compose || echo "error locking docker-compose") && ( chmod -x /host/bin/docker-compose || echo "error locking docker-compose")
    Environment:  <none>
    Mounts:
      /host/bin from host-bin (rw)
      /host/usr/bin from host-usr-bin (rw)
  Containers:
   ariang-custom-app:
    Image:      p3terx/ariang:latest
    Port:       6880/TCP
    Host Port:  0/TCP
    Limits:
      cpu:     4
      memory:  8Gi
    Requests:
      cpu:      10m
      memory:   50Mi
    Liveness:   tcp-socket :6880 delay=10s timeout=5s period=10s #success=1 #failure=5
    Readiness:  tcp-socket :6880 delay=10s timeout=5s period=10s #success=1 #failure=5
    Startup:    tcp-socket :6880 delay=10s timeout=2s period=5s #success=1 #failure=60
    Environment:
      PUID:                    568
      USER_ID:                 568
      UID:                     568
      UMASK:                   002
      UMASK_SET:               002
      PGID:                    568
      GROUP_ID:                568
      GID:                     568
      S6_READ_ONLY_ROOT:       1
      NVIDIA_VISIBLE_DEVICES:  void
      TZ:                      Asia/Shanghai
    Mounts:
      /shared from shared (rw)
      /tmp from temp (rw)
      /var/logs from varlogs (rw)
  Volumes:
   host-bin:
    Type:          HostPath (bare host directory volume)
    Path:          /bin
    HostPathType:  
   host-usr-bin:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/bin
    HostPathType:  
   shared:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
   temp:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
   varlogs:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Progressing    True    NewReplicaSetAvailable
  Available      True    MinimumReplicasAvailable
OldReplicaSets:  <none>
NewReplicaSet:   ariang-custom-app-78b84f865d (0/0 replicas created)
Events:          <none>


Name:               jellyfin
Namespace:          ix-jellyfin
CreationTimestamp:  Sun, 13 Mar 2022 21:11:40 +0800
Labels:             app.kubernetes.io/instance=jellyfin
                    app.kubernetes.io/managed-by=Helm
                    app.kubernetes.io/name=jellyfin
                    app.kubernetes.io/version=10.7.7
                    helm.sh/chart=jellyfin-10.0.1
Annotations:        deployment.kubernetes.io/revision: 1
                    meta.helm.sh/release-name: jellyfin
                    meta.helm.sh/release-namespace: ix-jellyfin
                    rollme: lcxRU
Selector:           app.kubernetes.io/instance=jellyfin,app.kubernetes.io/name=jellyfin
Replicas:           0 desired | 0 updated | 0 total | 0 available | 0 unavailable
StrategyType:       Recreate
MinReadySeconds:    0
Pod Template:
  Labels:           app.kubernetes.io/instance=jellyfin
                    app.kubernetes.io/name=jellyfin
  Annotations:      rollme: 7XHSs
  Service Account:  default
  Init Containers:
   autopermissions:
    Image:      ghcr.io/truecharts/alpine:v3.14.2@sha256:4095394abbae907e94b1f2fd2e2de6c4f201a5b9704573243ca8eb16db8cdb7c
    Port:       <none>
    Host Port:  <none>
    Command:
      /bin/sh
      -c
      echo 'Automatically correcting permissions...';
    Limits:
      cpu:     4
      memory:  8Gi
    Requests:
      cpu:        10m
      memory:     50Mi
    Environment:  <none>
    Mounts:       <none>
   hostpatch:
    Image:      ghcr.io/truecharts/alpine:v3.14.2@sha256:4095394abbae907e94b1f2fd2e2de6c4f201a5b9704573243ca8eb16db8cdb7c
    Port:       <none>
    Host Port:  <none>
    Command:
      /bin/sh
      -c
      ( sysctl -w fs.inotify.max_user_watches=524288 || echo "error setting inotify") && ( sysctl -w fs.inotify.max_user_instances=512 || echo "error setting inotify") && ( chmod -x /host/usr/bin/docker-compose || echo "error locking docker-compose") && ( chmod -x /host/bin/docker-compose || echo "error locking docker-compose")
    Environment:  <none>
    Mounts:
      /host/bin from host-bin (rw)
      /host/usr/bin from host-usr-bin (rw)
  Containers:
   jellyfin:
    Image:      tccr.io/truecharts/jellyfin:v10.7.7@sha256:0136db4677a2ee2ee8a6962d813d6e3b49aa86784a7cfdc3af76427db32c3470
    Port:       8096/TCP
    Host Port:  0/TCP
    Limits:
      cpu:     4
      memory:  8Gi
    Requests:
      cpu:      10m
      memory:   50Mi
    Liveness:   tcp-socket :8096 delay=10s timeout=5s period=10s #success=1 #failure=5
    Readiness:  tcp-socket :8096 delay=10s timeout=5s period=10s #success=1 #failure=5
    Startup:    tcp-socket :8096 delay=10s timeout=2s period=5s #success=1 #failure=60
    Environment:
      PUID:                         568
      USER_ID:                      568
      UID:                          568
      UMASK:                        002
      UMASK_SET:                    002
      PGID:                         568
      GROUP_ID:                     568
      GID:                          568
      S6_READ_ONLY_ROOT:            1
      NVIDIA_VISIBLE_DEVICES:       void
      TZ:                           Asia/Shanghai
      JELLYFIN_PublishedServerUrl:  
    Mounts:
      /cache from cache (rw)
      /config from config (rw)
      /media from list-0 (rw)
      /shared from shared (rw)
      /tmp from temp (rw)
      /var/logs from varlogs (rw)
  Volumes:
   cache:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
   config:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  jellyfin-config
    ReadOnly:   false
   host-bin:
    Type:          HostPath (bare host directory volume)
    Path:          /bin
    HostPathType:  
   host-usr-bin:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/bin
    HostPathType:  
   list-0:
    Type:          HostPath (bare host directory volume)
    Path:          /mnt/pa/media
    HostPathType:  
   shared:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
   temp:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
   varlogs:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Progressing    True    NewReplicaSetAvailable
  Available      True    MinimumReplicasAvailable
OldReplicaSets:  <none>
NewReplicaSet:   jellyfin-fb89c84d4 (0/0 replicas created)
Events:          <none>


Name:                   coredns
Namespace:              kube-system
CreationTimestamp:      Sat, 13 Nov 2021 16:57:59 +0800
Labels:                 k8s-app=kube-dns
                        kubernetes.io/name=CoreDNS
                        objectset.rio.cattle.io/hash=bce283298811743a0386ab510f2f67ef74240c57
Annotations:            deployment.kubernetes.io/revision: 2
                        objectset.rio.cattle.io/applied:
                          H4sIAAAAAAAA/6xVUW/bNhD+K8M9S7GVtI0rYA+dna1FV8+ok74URkFTJ4szxePIkxMj0H8fTrIdu02TdtiTZfK743cfvzveg/LmE4ZoyEEOyvs42GSQwNq4AnKYoLe0rdExJFAjq0...
                        objectset.rio.cattle.io/id: 
                        objectset.rio.cattle.io/owner-gvk: k3s.cattle.io/v1, Kind=Addon
                        objectset.rio.cattle.io/owner-name: coredns
                        objectset.rio.cattle.io/owner-namespace: kube-system
Selector:               k8s-app=kube-dns
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  1 max unavailable, 25% max surge
Pod Template:
  Labels:           k8s-app=kube-dns
  Service Account:  coredns
  Containers:
   coredns:
    Image:       rancher/mirrored-coredns-coredns:1.9.1
    Ports:       53/UDP, 53/TCP, 9153/TCP
    Host Ports:  0/UDP, 0/TCP, 0/TCP
    Args:
      -conf
      /etc/coredns/Corefile
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=1s period=10s #success=1 #failure=3
    Readiness:    http-get http://:8181/ready delay=0s timeout=1s period=2s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (ro)
      /etc/coredns/custom from custom-config-volume (ro)
  Volumes:
   config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      coredns
    Optional:  false
   custom-config-volume:
    Type:               ConfigMap (a volume populated by a ConfigMap)
    Name:               coredns-custom
    Optional:           true
  Priority Class Name:  system-cluster-critical
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   coredns-d76bd69b (1/1 replicas created)
Events:          <none>
debug finished in 2 seconds for k3s kubectl describe deployments --all-namespaces


+--------------------------------------------------------------------------------+
+             k3s kubectl describe pods --all-namespaces @1656907172             +
+--------------------------------------------------------------------------------+
Name:                 coredns-d76bd69b-ks26g
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 ix-truenas/192.168.1.5
Start Time:           Mon, 04 Jul 2022 02:21:53 +0800
Labels:               k8s-app=kube-dns
                      pod-template-hash=d76bd69b
Annotations:          k8s.v1.cni.cncf.io/network-status:
                        [{
                            "name": "ix-net",
                            "interface": "eth0",
                            "ips": [
                                "172.16.0.18"
                            ],
                            "mac": "7e:79:99:6d:9a:7e",
                            "default": true,
                            "dns": {}
                        }]
                      k8s.v1.cni.cncf.io/networks-status:
                        [{
                            "name": "ix-net",
                            "interface": "eth0",
                            "ips": [
                                "172.16.0.18"
                            ],
                            "mac": "7e:79:99:6d:9a:7e",
                            "default": true,
                            "dns": {}
                        }]
Status:               Running
IP:                   172.16.0.18
IPs:
  IP:           172.16.0.18
Controlled By:  ReplicaSet/coredns-d76bd69b
Containers:
  coredns:
    Container ID:  docker://47f4c930a8a204c48284890e136a28f8219bdf35c6b23e2b975fb5e86a6c14ad
    Image:         rancher/mirrored-coredns-coredns:1.9.1
    Image ID:      docker-pullable://rancher/mirrored-coredns-coredns@sha256:35e38f3165a19cb18c65d83334c13d61db6b24905f45640aa8c2d2a6f55ebcb0
    Ports:         53/UDP, 53/TCP, 9153/TCP
    Host Ports:    0/UDP, 0/TCP, 0/TCP
    Args:
      -conf
      /etc/coredns/Corefile
    State:          Running
      Started:      Mon, 04 Jul 2022 02:22:55 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=1s period=10s #success=1 #failure=3
    Readiness:    http-get http://:8181/ready delay=0s timeout=1s period=2s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (ro)
      /etc/coredns/custom from custom-config-volume (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-hzvkj (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      coredns
    Optional:  false
  custom-config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      coredns-custom
    Optional:  true
  kube-api-access-hzvkj:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              beta.kubernetes.io/os=linux
Tolerations:                 CriticalAddonsOnly op=Exists
                             node-role.kubernetes.io/control-plane:NoSchedule op=Exists
                             node-role.kubernetes.io/master:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>


Name:         svclb-openldap-ldaps-vzm7c
Namespace:    ix-openldap
Priority:     0
Node:         ix-truenas/192.168.1.5
Start Time:   Mon, 04 Jul 2022 02:23:09 +0800
Labels:       app=svclb-openldap-ldaps
              controller-revision-hash=c67d8db6
              pod-template-generation=2
              svccontroller.k3s.cattle.io/svcname=openldap-ldaps
Annotations:  k8s.v1.cni.cncf.io/network-status:
                [{
                    "name": "ix-net",
                    "interface": "eth0",
                    "ips": [
                        "172.16.0.19"
                    ],
                    "mac": "32:4d:c2:86:e3:a8",
                    "default": true,
                    "dns": {}
                }]
              k8s.v1.cni.cncf.io/networks-status:
                [{
                    "name": "ix-net",
                    "interface": "eth0",
                    "ips": [
                        "172.16.0.19"
                    ],
                    "mac": "32:4d:c2:86:e3:a8",
                    "default": true,
                    "dns": {}
                }]
Status:       Running
IP:           172.16.0.19
IPs:
  IP:           172.16.0.19
Controlled By:  DaemonSet/svclb-openldap-ldaps
Containers:
  lb-tcp-636:
    Container ID:   docker://c288dbe013811e7c059bbf717906ee294c234000792c37e47bbd5cef2592bcc1
    Image:          rancher/klipper-lb:v0.3.4
    Image ID:       docker-pullable://rancher/klipper-lb@sha256:d5ec8350ee1a80ec783effe95a2fefae371d1498ac04865cb6024499882f291c
    Port:           636/TCP
    Host Port:      636/TCP
    State:          Running
      Started:      Mon, 04 Jul 2022 02:24:04 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      SRC_PORT:    636
      DEST_PROTO:  TCP
      DEST_PORT:   636
      DEST_IPS:    172.17.6.182
    Mounts:        <none>
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:            <none>
QoS Class:          BestEffort
Node-Selectors:     <none>
Tolerations:        CriticalAddonsOnly op=Exists
                    node-role.kubernetes.io/control-plane:NoSchedule op=Exists
                    node-role.kubernetes.io/master:NoSchedule op=Exists
                    node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                    node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                    node.kubernetes.io/not-ready:NoExecute op=Exists
                    node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                    node.kubernetes.io/unreachable:NoExecute op=Exists
                    node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:             <none>


Name:         svclb-aria2-listen-5zljr
Namespace:    ix-aria2
Priority:     0
Node:         ix-truenas/192.168.1.5
Start Time:   Mon, 04 Jul 2022 02:23:09 +0800
Labels:       app=svclb-aria2-listen
              controller-revision-hash=5dd7669b4b
              pod-template-generation=2
              svccontroller.k3s.cattle.io/svcname=aria2-listen
Annotations:  k8s.v1.cni.cncf.io/network-status:
                [{
                    "name": "ix-net",
                    "interface": "eth0",
                    "ips": [
                        "172.16.0.20"
                    ],
                    "mac": "aa:44:f2:24:1d:dc",
                    "default": true,
                    "dns": {}
                }]
              k8s.v1.cni.cncf.io/networks-status:
                [{
                    "name": "ix-net",
                    "interface": "eth0",
                    "ips": [
                        "172.16.0.20"
                    ],
                    "mac": "aa:44:f2:24:1d:dc",
                    "default": true,
                    "dns": {}
                }]
Status:       Running
IP:           172.16.0.20
IPs:
  IP:           172.16.0.20
Controlled By:  DaemonSet/svclb-aria2-listen
Containers:
  lb-tcp-6888:
    Container ID:   docker://9cec67af1af718a3d7fcafe3038c4f7a6ead6aeaddab261c025ac1cd0eedbc39
    Image:          rancher/klipper-lb:v0.3.4
    Image ID:       docker-pullable://rancher/klipper-lb@sha256:d5ec8350ee1a80ec783effe95a2fefae371d1498ac04865cb6024499882f291c
    Port:           6888/TCP
    Host Port:      6888/TCP
    State:          Running
      Started:      Mon, 04 Jul 2022 02:24:09 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      SRC_PORT:    6888
      DEST_PROTO:  TCP
      DEST_PORT:   6888
      DEST_IPS:    172.17.185.94
    Mounts:        <none>
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:            <none>
QoS Class:          BestEffort
Node-Selectors:     <none>
Tolerations:        CriticalAddonsOnly op=Exists
                    node-role.kubernetes.io/control-plane:NoSchedule op=Exists
                    node-role.kubernetes.io/master:NoSchedule op=Exists
                    node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                    node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                    node.kubernetes.io/not-ready:NoExecute op=Exists
                    node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                    node.kubernetes.io/unreachable:NoExecute op=Exists
                    node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:             <none>


Name:         svclb-aria2-fsw89
Namespace:    ix-aria2
Priority:     0
Node:         ix-truenas/192.168.1.5
Start Time:   Mon, 04 Jul 2022 02:23:11 +0800
Labels:       app=svclb-aria2
              controller-revision-hash=867587f4c4
              pod-template-generation=2
              svccontroller.k3s.cattle.io/svcname=aria2
Annotations:  k8s.v1.cni.cncf.io/network-status:
                [{
                    "name": "ix-net",
                    "interface": "eth0",
                    "ips": [
                        "172.16.0.21"
                    ],
                    "mac": "22:f5:a5:5a:44:f6",
                    "default": true,
                    "dns": {}
                }]
              k8s.v1.cni.cncf.io/networks-status:
                [{
                    "name": "ix-net",
                    "interface": "eth0",
                    "ips": [
                        "172.16.0.21"
                    ],
                    "mac": "22:f5:a5:5a:44:f6",
                    "default": true,
                    "dns": {}
                }]
Status:       Running
IP:           172.16.0.21
IPs:
  IP:           172.16.0.21
Controlled By:  DaemonSet/svclb-aria2
Containers:
  lb-tcp-6800:
    Container ID:   docker://ed624ae5bad39dfa2a7107e5f3944d663136e74a91abec14f1a25f74b5db94c4
    Image:          rancher/klipper-lb:v0.3.4
    Image ID:       docker-pullable://rancher/klipper-lb@sha256:d5ec8350ee1a80ec783effe95a2fefae371d1498ac04865cb6024499882f291c
    Port:           6800/TCP
    Host Port:      6800/TCP
    State:          Running
      Started:      Mon, 04 Jul 2022 02:24:07 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      SRC_PORT:    6800
      DEST_PROTO:  TCP
      DEST_PORT:   6800
      DEST_IPS:    172.17.134.236
    Mounts:        <none>
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:            <none>
QoS Class:          BestEffort
Node-Selectors:     <none>
Tolerations:        CriticalAddonsOnly op=Exists
                    node-role.kubernetes.io/control-plane:NoSchedule op=Exists
                    node-role.kubernetes.io/master:NoSchedule op=Exists
                    node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                    node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                    node.kubernetes.io/not-ready:NoExecute op=Exists
                    node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                    node.kubernetes.io/unreachable:NoExecute op=Exists
                    node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:             <none>


Name:         svclb-ariang-custom-app-svbxp
Namespace:    ix-ariang
Priority:     0
Node:         ix-truenas/192.168.1.5
Start Time:   Mon, 04 Jul 2022 02:23:12 +0800
Labels:       app=svclb-ariang-custom-app
              controller-revision-hash=76748dd6d9
              pod-template-generation=2
              svccontroller.k3s.cattle.io/svcname=ariang-custom-app
Annotations:  k8s.v1.cni.cncf.io/network-status:
                [{
                    "name": "ix-net",
                    "interface": "eth0",
                    "ips": [
                        "172.16.0.22"
                    ],
                    "mac": "d2:e9:1c:4f:ee:f8",
                    "default": true,
                    "dns": {}
                }]
              k8s.v1.cni.cncf.io/networks-status:
                [{
                    "name": "ix-net",
                    "interface": "eth0",
                    "ips": [
                        "172.16.0.22"
                    ],
                    "mac": "d2:e9:1c:4f:ee:f8",
                    "default": true,
                    "dns": {}
                }]
Status:       Running
IP:           172.16.0.22
IPs:
  IP:           172.16.0.22
Controlled By:  DaemonSet/svclb-ariang-custom-app
Containers:
  lb-tcp-6880:
    Container ID:   docker://3da8a838c657e2fcd7180b9d730ac2fbb6a0e52b27e08c90fe61aa4c66758ce8
    Image:          rancher/klipper-lb:v0.3.4
    Image ID:       docker-pullable://rancher/klipper-lb@sha256:d5ec8350ee1a80ec783effe95a2fefae371d1498ac04865cb6024499882f291c
    Port:           6880/TCP
    Host Port:      6880/TCP
    State:          Running
      Started:      Mon, 04 Jul 2022 02:24:10 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      SRC_PORT:    6880
      DEST_PROTO:  TCP
      DEST_PORT:   6880
      DEST_IPS:    172.17.132.154
    Mounts:        <none>
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:            <none>
QoS Class:          BestEffort
Node-Selectors:     <none>
Tolerations:        CriticalAddonsOnly op=Exists
                    node-role.kubernetes.io/control-plane:NoSchedule op=Exists
                    node-role.kubernetes.io/master:NoSchedule op=Exists
                    node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                    node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                    node.kubernetes.io/not-ready:NoExecute op=Exists
                    node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                    node.kubernetes.io/unreachable:NoExecute op=Exists
                    node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:             <none>


Name:         svclb-openldap-4rjmb
Namespace:    ix-openldap
Priority:     0
Node:         ix-truenas/192.168.1.5
Start Time:   Mon, 04 Jul 2022 02:23:19 +0800
Labels:       app=svclb-openldap
              controller-revision-hash=75fcc8f9c
              pod-template-generation=2
              svccontroller.k3s.cattle.io/svcname=openldap
Annotations:  k8s.v1.cni.cncf.io/network-status:
                [{
                    "name": "ix-net",
                    "interface": "eth0",
                    "ips": [
                        "172.16.0.23"
                    ],
                    "mac": "72:44:75:ba:ba:e3",
                    "default": true,
                    "dns": {}
                }]
              k8s.v1.cni.cncf.io/networks-status:
                [{
                    "name": "ix-net",
                    "interface": "eth0",
                    "ips": [
                        "172.16.0.23"
                    ],
                    "mac": "72:44:75:ba:ba:e3",
                    "default": true,
                    "dns": {}
                }]
Status:       Running
IP:           172.16.0.23
IPs:
  IP:           172.16.0.23
Controlled By:  DaemonSet/svclb-openldap
Containers:
  lb-tcp-389:
    Container ID:   docker://5fca6b5e0b1661365388306082c195d03e79229fd0ffbd5f323a3562fba687d8
    Image:          rancher/klipper-lb:v0.3.4
    Image ID:       docker-pullable://rancher/klipper-lb@sha256:d5ec8350ee1a80ec783effe95a2fefae371d1498ac04865cb6024499882f291c
    Port:           389/TCP
    Host Port:      389/TCP
    State:          Running
      Started:      Mon, 04 Jul 2022 02:24:13 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      SRC_PORT:    389
      DEST_PROTO:  TCP
      DEST_PORT:   389
      DEST_IPS:    172.17.145.141
    Mounts:        <none>
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:            <none>
QoS Class:          BestEffort
Node-Selectors:     <none>
Tolerations:        CriticalAddonsOnly op=Exists
                    node-role.kubernetes.io/control-plane:NoSchedule op=Exists
                    node-role.kubernetes.io/master:NoSchedule op=Exists
                    node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                    node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                    node.kubernetes.io/not-ready:NoExecute op=Exists
                    node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                    node.kubernetes.io/unreachable:NoExecute op=Exists
                    node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:             <none>


Name:         svclb-jellyfin-g8sj9
Namespace:    ix-jellyfin
Priority:     0
Node:         ix-truenas/192.168.1.5
Start Time:   Mon, 04 Jul 2022 02:23:09 +0800
Labels:       app=svclb-jellyfin
              controller-revision-hash=6d576c749c
              pod-template-generation=2
              svccontroller.k3s.cattle.io/svcname=jellyfin
Annotations:  k8s.v1.cni.cncf.io/network-status:
                [{
                    "name": "ix-net",
                    "interface": "eth0",
                    "ips": [
                        "172.16.0.24"
                    ],
                    "mac": "ca:a6:1c:db:9c:d1",
                    "default": true,
                    "dns": {}
                }]
              k8s.v1.cni.cncf.io/networks-status:
                [{
                    "name": "ix-net",
                    "interface": "eth0",
                    "ips": [
                        "172.16.0.24"
                    ],
                    "mac": "ca:a6:1c:db:9c:d1",
                    "default": true,
                    "dns": {}
                }]
Status:       Running
IP:           172.16.0.24
IPs:
  IP:           172.16.0.24
Controlled By:  DaemonSet/svclb-jellyfin
Containers:
  lb-tcp-8096:
    Container ID:   docker://36ad7daca9093348a80ee9eaebcc664a02322928fe0ff7a5133c5d247f66e6c7
    Image:          rancher/klipper-lb:v0.3.4
    Image ID:       docker-pullable://rancher/klipper-lb@sha256:d5ec8350ee1a80ec783effe95a2fefae371d1498ac04865cb6024499882f291c
    Port:           8096/TCP
    Host Port:      8096/TCP
    State:          Running
      Started:      Mon, 04 Jul 2022 02:24:16 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      SRC_PORT:    8096
      DEST_PROTO:  TCP
      DEST_PORT:   8096
      DEST_IPS:    172.17.130.42
    Mounts:        <none>
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:            <none>
QoS Class:          BestEffort
Node-Selectors:     <none>
Tolerations:        CriticalAddonsOnly op=Exists
                    node-role.kubernetes.io/control-plane:NoSchedule op=Exists
                    node-role.kubernetes.io/master:NoSchedule op=Exists
                    node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                    node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                    node.kubernetes.io/not-ready:NoExecute op=Exists
                    node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                    node.kubernetes.io/unreachable:NoExecute op=Exists
                    node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:             <none>


Name:                 openebs-zfs-node-5zm8n
Namespace:            kube-system
Priority:             900001000
Priority Class Name:  openebs-zfs-csi-node-critical
Node:                 ix-truenas/192.168.1.5
Start Time:           Mon, 04 Jul 2022 02:23:29 +0800
Labels:               app=openebs-zfs-node
                      controller-revision-hash=57f5455f6b
                      openebs.io/component-name=openebs-zfs-node
                      openebs.io/version=ci
                      pod-template-generation=2
                      role=openebs-zfs
Annotations:          <none>
Status:               Pending
IP:                   192.168.1.5
IPs:
  IP:           192.168.1.5
Controlled By:  DaemonSet/openebs-zfs-node
Containers:
  csi-node-driver-registrar:
    Container ID:  
    Image:         k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Args:
      --v=5
      --csi-address=$(ADDRESS)
      --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Environment:
      ADDRESS:               /plugin/csi.sock
      DRIVER_REG_SOCK_PATH:  /var/lib/kubelet/plugins/zfs-localpv/csi.sock
      KUBE_NODE_NAME:         (v1:spec.nodeName)
      NODE_DRIVER:           openebs-zfs
    Mounts:
      /plugin from plugin-dir (rw)
      /registration from registration-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-89mwk (ro)
  openebs-zfs-plugin:
    Container ID:  docker://14f362d0cc9ca88fee8e6ab31c809419fb3c15357741103eb905fc8229d5eff8
    Image:         openebs/zfs-driver:2.0.0
    Image ID:      docker-pullable://openebs/zfs-driver@sha256:472dc7ca43f9780246cc6d2d478ff4b08ead904d757dbb263f796fb613dbe04d
    Port:          <none>
    Host Port:     <none>
    Args:
      --nodename=$(OPENEBS_NODE_NAME)
      --endpoint=$(OPENEBS_CSI_ENDPOINT)
      --plugin=$(OPENEBS_NODE_DRIVER)
    State:          Running
      Started:      Mon, 04 Jul 2022 02:25:20 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      OPENEBS_NODE_NAME:      (v1:spec.nodeName)
      OPENEBS_CSI_ENDPOINT:  unix:///plugin/csi.sock
      OPENEBS_NODE_DRIVER:   agent
      OPENEBS_NAMESPACE:     openebs
      ALLOWED_TOPOLOGIES:    All
    Mounts:
      /dev from device-dir (rw)
      /home/keys from encr-keys (rw)
      /host from host-root (ro)
      /plugin from plugin-dir (rw)
      /sbin/zfs from chroot-zfs (rw,path="zfs")
      /var/lib/kubelet/ from pods-mount-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-89mwk (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  device-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /dev
    HostPathType:  Directory
  encr-keys:
    Type:          HostPath (bare host directory volume)
    Path:          /home/keys
    HostPathType:  DirectoryOrCreate
  chroot-zfs:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      openebs-zfspv-bin
    Optional:  false
  host-root:
    Type:          HostPath (bare host directory volume)
    Path:          /
    HostPathType:  Directory
  registration-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/plugins_registry/
    HostPathType:  DirectoryOrCreate
  plugin-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/plugins/zfs-localpv/
    HostPathType:  DirectoryOrCreate
  pods-mount-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/
    HostPathType:  Directory
  kube-api-access-89mwk:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/network-unavailable:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason   Age                  From     Message
  ----     ------   ----                 ----     -------
  Warning  Failed   35m (x78 over 9h)    kubelet  Failed to pull image "k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0": rpc error: code = Unknown desc = Error response from daemon: Get "https://k8s.gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  Normal   BackOff  26s (x2393 over 9h)  kubelet  Back-off pulling image "k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0"


Name:                 openebs-zfs-controller-0
Namespace:            kube-system
Priority:             900000000
Priority Class Name:  openebs-zfs-csi-controller-critical
Node:                 ix-truenas/192.168.1.5
Start Time:           Mon, 04 Jul 2022 02:24:58 +0800
Labels:               app=openebs-zfs-controller
                      controller-revision-hash=openebs-zfs-controller-698698d48b
                      openebs.io/component-name=openebs-zfs-controller
                      openebs.io/version=ci
                      role=openebs-zfs
                      statefulset.kubernetes.io/pod-name=openebs-zfs-controller-0
Annotations:          k8s.v1.cni.cncf.io/network-status:
                        [{
                            "name": "ix-net",
                            "interface": "eth0",
                            "ips": [
                                "172.16.0.25"
                            ],
                            "mac": "ca:a8:c1:98:81:1f",
                            "default": true,
                            "dns": {}
                        }]
                      k8s.v1.cni.cncf.io/networks-status:
                        [{
                            "name": "ix-net",
                            "interface": "eth0",
                            "ips": [
                                "172.16.0.25"
                            ],
                            "mac": "ca:a8:c1:98:81:1f",
                            "default": true,
                            "dns": {}
                        }]
Status:               Pending
IP:                   172.16.0.25
IPs:
  IP:           172.16.0.25
Controlled By:  StatefulSet/openebs-zfs-controller
Containers:
  csi-resizer:
    Container ID:  
    Image:         k8s.gcr.io/sig-storage/csi-resizer:v1.2.0
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Args:
      --v=5
      --csi-address=$(ADDRESS)
      --leader-election
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Environment:
      ADDRESS:  /var/lib/csi/sockets/pluginproxy/csi.sock
    Mounts:
      /var/lib/csi/sockets/pluginproxy/ from socket-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-8tngl (ro)
  csi-snapshotter:
    Container ID:  docker://1d86c89041582b13dbc542f912b030770c10648bb6dcfac6b72b8f3608345641
    Image:         k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0
    Image ID:      docker-pullable://k8s.gcr.io/sig-storage/csi-snapshotter@sha256:51f2dfde5bccac7854b3704689506aeecfb793328427b91115ba253a93e60782
    Port:          <none>
    Host Port:     <none>
    Args:
      --csi-address=$(ADDRESS)
      --leader-election
    State:          Running
      Started:      Mon, 04 Jul 2022 02:25:44 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      ADDRESS:  /var/lib/csi/sockets/pluginproxy/csi.sock
    Mounts:
      /var/lib/csi/sockets/pluginproxy/ from socket-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-8tngl (ro)
  snapshot-controller:
    Container ID:  docker://23a4580cd74c7d6c67272010d7679b2ceb6fc72eafa47dcad14e0669e8b99d50
    Image:         k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0
    Image ID:      docker-pullable://k8s.gcr.io/sig-storage/snapshot-controller@sha256:00fcc441ea9f72899c25eed61d602272a2a58c5f0014332bdcb5ac24acef08e4
    Port:          <none>
    Host Port:     <none>
    Args:
      --v=5
      --leader-election=true
    State:          Running
      Started:      Mon, 04 Jul 2022 02:25:58 +0800
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-8tngl (ro)
  csi-provisioner:
    Container ID:  
    Image:         k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Args:
      --csi-address=$(ADDRESS)
      --v=5
      --feature-gates=Topology=true
      --strict-topology
      --leader-election
      --extra-create-metadata=true
      --enable-capacity=true
      --default-fstype=ext4
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Environment:
      ADDRESS:    /var/lib/csi/sockets/pluginproxy/csi.sock
      NAMESPACE:  kube-system (v1:metadata.namespace)
      POD_NAME:   openebs-zfs-controller-0 (v1:metadata.name)
    Mounts:
      /var/lib/csi/sockets/pluginproxy/ from socket-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-8tngl (ro)
  openebs-zfs-plugin:
    Container ID:  docker://f838a18b3148965fbb33697b2453fe10a9b300c7a5c8ea0fa1f6605c24c385b7
    Image:         openebs/zfs-driver:2.0.0
    Image ID:      docker-pullable://openebs/zfs-driver@sha256:472dc7ca43f9780246cc6d2d478ff4b08ead904d757dbb263f796fb613dbe04d
    Port:          <none>
    Host Port:     <none>
    Args:
      --endpoint=$(OPENEBS_CSI_ENDPOINT)
      --plugin=$(OPENEBS_CONTROLLER_DRIVER)
    State:          Running
      Started:      Mon, 04 Jul 2022 02:26:29 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      OPENEBS_CONTROLLER_DRIVER:    controller
      OPENEBS_CSI_ENDPOINT:         unix:///var/lib/csi/sockets/pluginproxy/csi.sock
      OPENEBS_NAMESPACE:            openebs
      OPENEBS_IO_INSTALLER_TYPE:    zfs-operator
      OPENEBS_IO_ENABLE_ANALYTICS:  true
    Mounts:
      /var/lib/csi/sockets/pluginproxy/ from socket-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-8tngl (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  socket-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-8tngl:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason   Age                    From     Message
  ----     ------   ----                   ----     -------
  Warning  Failed   24m (x86 over 9h)      kubelet  Failed to pull image "k8s.gcr.io/sig-storage/csi-resizer:v1.2.0": rpc error: code = Unknown desc = Error response from daemon: Get "https://k8s.gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  Normal   BackOff  4m13s (x2258 over 9h)  kubelet  Back-off pulling image "k8s.gcr.io/sig-storage/csi-resizer:v1.2.0"
debug finished in 0 seconds for k3s kubectl describe pods --all-namespaces


+--------------------------------------------------------------------------------+
+           k3s kubectl describe services --all-namespaces @1656907172           +
+--------------------------------------------------------------------------------+
Name:              kubernetes
Namespace:         default
Labels:            component=apiserver
                   provider=kubernetes
Annotations:       <none>
Selector:          <none>
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                172.17.0.1
IPs:               172.17.0.1
Port:              https  443/TCP
TargetPort:        6443/TCP
Endpoints:         192.168.1.5:6443
Session Affinity:  None
Events:            <none>


Name:              kube-dns
Namespace:         kube-system
Labels:            k8s-app=kube-dns
                   kubernetes.io/cluster-service=true
                   kubernetes.io/name=CoreDNS
                   objectset.rio.cattle.io/hash=bce283298811743a0386ab510f2f67ef74240c57
Annotations:       objectset.rio.cattle.io/applied:
                     H4sIAAAAAAAA/4yRwW7UMBCGXwXN2Qlxstukljig9oKQUKUCF8TBcWZZk8S2PJMgtMq7I++mYqEK9JZoPn/6558T6GA/YyTrHSiYJQjoretAwSPG2RoEASOy7jRrUCfQznnWbL2j9O...
                   objectset.rio.cattle.io/id: 
                   objectset.rio.cattle.io/owner-gvk: k3s.cattle.io/v1, Kind=Addon
                   objectset.rio.cattle.io/owner-name: coredns
                   objectset.rio.cattle.io/owner-namespace: kube-system
                   prometheus.io/port: 9153
                   prometheus.io/scrape: true
Selector:          k8s-app=kube-dns
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                172.17.0.10
IPs:               172.17.0.10
Port:              dns  53/UDP
TargetPort:        53/UDP
Endpoints:         172.16.0.18:53
Port:              dns-tcp  53/TCP
TargetPort:        53/TCP
Endpoints:         172.16.0.18:53
Port:              metrics  9153/TCP
TargetPort:        9153/TCP
Endpoints:         172.16.0.18:9153
Session Affinity:  None
Events:            <none>


Name:                     openldap
Namespace:                ix-openldap
Labels:                   app.kubernetes.io/instance=openldap
                          app.kubernetes.io/managed-by=Helm
                          app.kubernetes.io/name=openldap
                          app.kubernetes.io/version=1.5.0
                          helm.sh/chart=openldap-5.0.24
Annotations:              meta.helm.sh/release-name: openldap
                          meta.helm.sh/release-namespace: ix-openldap
Selector:                 app.kubernetes.io/instance=openldap,app.kubernetes.io/name=openldap
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       172.17.145.141
IPs:                      172.17.145.141
LoadBalancer Ingress:     192.168.1.5
Port:                     main  389/TCP
TargetPort:               389/TCP
NodePort:                 main  62332/TCP
Endpoints:                <none>
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>


Name:                     openldap-ldaps
Namespace:                ix-openldap
Labels:                   app.kubernetes.io/instance=openldap
                          app.kubernetes.io/managed-by=Helm
                          app.kubernetes.io/name=openldap
                          app.kubernetes.io/version=1.5.0
                          helm.sh/chart=openldap-5.0.24
Annotations:              meta.helm.sh/release-name: openldap
                          meta.helm.sh/release-namespace: ix-openldap
Selector:                 app.kubernetes.io/instance=openldap,app.kubernetes.io/name=openldap
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       172.17.6.182
IPs:                      172.17.6.182
LoadBalancer Ingress:     192.168.1.5
Port:                     ldaps  636/TCP
TargetPort:               636/TCP
NodePort:                 ldaps  63942/TCP
Endpoints:                <none>
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>


Name:                     aria2-listen
Namespace:                ix-aria2
Labels:                   app.kubernetes.io/instance=aria2
                          app.kubernetes.io/managed-by=Helm
                          app.kubernetes.io/name=aria2
                          app.kubernetes.io/version=latest
                          helm-revision=2
                          helm.sh/chart=aria2-5.0.7
Annotations:              meta.helm.sh/release-name: aria2
                          meta.helm.sh/release-namespace: ix-aria2
                          metallb.universe.tf/allow-shared-ip: aria2
Selector:                 app.kubernetes.io/instance=aria2,app.kubernetes.io/name=aria2
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       172.17.185.94
IPs:                      172.17.185.94
LoadBalancer Ingress:     192.168.1.5
Port:                     listen  6888/TCP
TargetPort:               6888/TCP
NodePort:                 listen  42288/TCP
Endpoints:                <none>
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>


Name:                     aria2
Namespace:                ix-aria2
Labels:                   app.kubernetes.io/instance=aria2
                          app.kubernetes.io/managed-by=Helm
                          app.kubernetes.io/name=aria2
                          app.kubernetes.io/version=latest
                          helm-revision=2
                          helm.sh/chart=aria2-5.0.7
Annotations:              meta.helm.sh/release-name: aria2
                          meta.helm.sh/release-namespace: ix-aria2
                          metallb.universe.tf/allow-shared-ip: aria2
Selector:                 app.kubernetes.io/instance=aria2,app.kubernetes.io/name=aria2
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       172.17.134.236
IPs:                      172.17.134.236
LoadBalancer Ingress:     192.168.1.5
Port:                     main  6800/TCP
TargetPort:               6800/TCP
NodePort:                 main  32665/TCP
Endpoints:                <none>
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>


Name:                     ariang-custom-app
Namespace:                ix-ariang
Labels:                   app.kubernetes.io/instance=ariang
                          app.kubernetes.io/managed-by=Helm
                          app.kubernetes.io/name=custom-app
                          app.kubernetes.io/version=0.20.660
                          helm.sh/chart=custom-app-4.0.2
Annotations:              meta.helm.sh/release-name: ariang
                          meta.helm.sh/release-namespace: ix-ariang
                          metallb.universe.tf/allow-shared-ip: ariang-custom-app
Selector:                 app.kubernetes.io/instance=ariang,app.kubernetes.io/name=custom-app
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       172.17.132.154
IPs:                      172.17.132.154
LoadBalancer Ingress:     192.168.1.5
Port:                     main  6880/TCP
TargetPort:               6880/TCP
NodePort:                 main  44900/TCP
Endpoints:                <none>
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>


Name:                     jellyfin
Namespace:                ix-jellyfin
Labels:                   app.kubernetes.io/instance=jellyfin
                          app.kubernetes.io/managed-by=Helm
                          app.kubernetes.io/name=jellyfin
                          app.kubernetes.io/version=10.7.7
                          helm.sh/chart=jellyfin-10.0.1
Annotations:              meta.helm.sh/release-name: jellyfin
                          meta.helm.sh/release-namespace: ix-jellyfin
                          metallb.universe.tf/allow-shared-ip: jellyfin
Selector:                 app.kubernetes.io/instance=jellyfin,app.kubernetes.io/name=jellyfin
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       172.17.130.42
IPs:                      172.17.130.42
LoadBalancer Ingress:     192.168.1.5
Port:                     main  8096/TCP
TargetPort:               8096/TCP
NodePort:                 main  13950/TCP
Endpoints:                <none>
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>
debug finished in 1 seconds for k3s kubectl describe services --all-namespaces


+--------------------------------------------------------------------------------+
+         k3s kubectl describe statefulset --all-namespaces @1656907173          +
+--------------------------------------------------------------------------------+
Name:               openldap
Namespace:          ix-openldap
CreationTimestamp:  Sat, 15 Jan 2022 22:52:55 +0800
Selector:           app.kubernetes.io/instance=openldap,app.kubernetes.io/name=openldap
Labels:             app.kubernetes.io/instance=openldap
                    app.kubernetes.io/managed-by=Helm
                    app.kubernetes.io/name=openldap
                    app.kubernetes.io/version=1.5.0
                    helm.sh/chart=openldap-5.0.24
Annotations:        meta.helm.sh/release-name: openldap
                    meta.helm.sh/release-namespace: ix-openldap
Replicas:           0 desired | 0 total
Update Strategy:    RollingUpdate
Pods Status:        0 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           app.kubernetes.io/instance=openldap
                    app.kubernetes.io/name=openldap
  Service Account:  default
  Init Containers:
   autopermissions:
    Image:      ghcr.io/truecharts/alpine:v3.14.2@sha256:4095394abbae907e94b1f2fd2e2de6c4f201a5b9704573243ca8eb16db8cdb7c
    Port:       <none>
    Host Port:  <none>
    Command:
      /bin/sh
      -c
      echo 'Automatically correcting permissions...';
    Limits:
      cpu:     4
      memory:  8Gi
    Requests:
      cpu:        10m
      memory:     50Mi
    Environment:  <none>
    Mounts:       <none>
   inotify:
    Image:      ghcr.io/truecharts/alpine:v3.14.2@sha256:4095394abbae907e94b1f2fd2e2de6c4f201a5b9704573243ca8eb16db8cdb7c
    Port:       <none>
    Host Port:  <none>
    Command:
      /bin/sh
      -c
      sysctl -w fs.inotify.max_user_watches=524288 && sysctl -w fs.inotify.max_user_instances=512 
    Environment:  <none>
    Mounts:       <none>
  Containers:
   openldap:
    Image:       tccr.io/truecharts/openldap:v1.5.0@sha256:0260d37c41f0c1207aaa642d7c786851385471a5ddf02bc6efc178241ddd8706
    Ports:       636/TCP, 389/TCP
    Host Ports:  0/TCP, 0/TCP
    Limits:
      cpu:     4
      memory:  8Gi
    Requests:
      cpu:      10m
      memory:   50Mi
    Liveness:   tcp-socket :389 delay=10s timeout=5s period=10s #success=1 #failure=5
    Readiness:  tcp-socket :389 delay=10s timeout=5s period=10s #success=1 #failure=5
    Startup:    tcp-socket :389 delay=10s timeout=2s period=5s #success=1 #failure=60
    Environment Variables from:
      openldapconfig  ConfigMap  Optional: false
      openldap        Secret     Optional: false
    Environment:
      PGID:                            568
      GROUP_ID:                        568
      GID:                             568
      NVIDIA_VISIBLE_DEVICES:          void
      CONTAINER_LOG_LEVEL:             4
      KEEP_EXISTING_CONFIG:            false
      LDAP_BACKEND:                    mdb
      LDAP_BASE_DN:                    dc=leerenbo,dc=cn
      LDAP_DOMAIN:                     leerenbo.cn
      LDAP_LOG_LEVEL:                  256
      LDAP_ORGANISATION:               Household
      LDAP_READONLY_USER:              false
      LDAP_REMOVE_CONFIG_AFTER_SETUP:  true
      LDAP_RFC2307BIS_SCHEMA:          false
      LDAP_SSL_HELPER_PREFIX:          ldap
      LDAP_TLS:                        true
      LDAP_TLS_CIPHER_SUITE:           NORMAL
      LDAP_TLS_ENFORCE:                false
      LDAP_TLS_PROTOCOL_MIN:           3.0
      LDAP_TLS_REQCERT:                never
      LDAP_TLS_VERIFY_CLIENT:          never
      TZ:                              Asia/Shanghai
      UMASK:                           002
    Mounts:
      /etc/ldap/slapd.d/ from slapd (rw)
      /shared from shared (rw)
      /tmp from temp (rw)
      /var/lib/ldap/ from data (rw)
      /var/logs from varlogs (rw)
  Volumes:
   shared:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
   temp:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
   varlogs:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
Volume Claims:
  Name:          data
  StorageClass:  ix-storage-class-openldap
  Labels:        <none>
  Annotations:   <none>
  Capacity:      10Gi
  Access Modes:  [ReadWriteOnce]
  Name:          slapd
  StorageClass:  ix-storage-class-openldap
  Labels:        <none>
  Annotations:   <none>
  Capacity:      10Gi
  Access Modes:  [ReadWriteOnce]
Events:          <none>


Name:               openebs-zfs-controller
Namespace:          kube-system
CreationTimestamp:  Sat, 13 Nov 2021 16:58:00 +0800
Selector:           app=openebs-zfs-controller,role=openebs-zfs
Labels:             objectset.rio.cattle.io/hash=4b1efd47d7f37713706abe9f8b0de3f0adf02e2c
                    openebs.io/component-name=openebs-zfs-controller
                    openebs.io/version=ci
Annotations:        objectset.rio.cattle.io/applied:
                      H4sIAAAAAAAA/8xWXY/aOBT9KyurD60U8zEzu9ONxAMdshLqFBCgSlWFkHFughfH9to3Wego/33lBJh0Zpgvdau+ge17fM7xuY5vCDPiM1gntCIhYca4dtElAdkIFZOQzJAhJLmcAZ...
                    objectset.rio.cattle.io/id: 
                    objectset.rio.cattle.io/owner-gvk: k3s.cattle.io/v1, Kind=Addon
                    objectset.rio.cattle.io/owner-name: zfs-operator
                    objectset.rio.cattle.io/owner-namespace: kube-system
Replicas:           1 desired | 1 total
Update Strategy:    RollingUpdate
  Partition:        0
Pods Status:        0 Running / 1 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           app=openebs-zfs-controller
                    openebs.io/component-name=openebs-zfs-controller
                    openebs.io/version=ci
                    role=openebs-zfs
  Service Account:  openebs-zfs-controller-sa
  Containers:
   csi-resizer:
    Image:      k8s.gcr.io/sig-storage/csi-resizer:v1.2.0
    Port:       <none>
    Host Port:  <none>
    Args:
      --v=5
      --csi-address=$(ADDRESS)
      --leader-election
    Environment:
      ADDRESS:  /var/lib/csi/sockets/pluginproxy/csi.sock
    Mounts:
      /var/lib/csi/sockets/pluginproxy/ from socket-dir (rw)
   csi-snapshotter:
    Image:      k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0
    Port:       <none>
    Host Port:  <none>
    Args:
      --csi-address=$(ADDRESS)
      --leader-election
    Environment:
      ADDRESS:  /var/lib/csi/sockets/pluginproxy/csi.sock
    Mounts:
      /var/lib/csi/sockets/pluginproxy/ from socket-dir (rw)
   snapshot-controller:
    Image:      k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0
    Port:       <none>
    Host Port:  <none>
    Args:
      --v=5
      --leader-election=true
    Environment:  <none>
    Mounts:       <none>
   csi-provisioner:
    Image:      k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0
    Port:       <none>
    Host Port:  <none>
    Args:
      --csi-address=$(ADDRESS)
      --v=5
      --feature-gates=Topology=true
      --strict-topology
      --leader-election
      --extra-create-metadata=true
      --enable-capacity=true
      --default-fstype=ext4
    Environment:
      ADDRESS:    /var/lib/csi/sockets/pluginproxy/csi.sock
      NAMESPACE:   (v1:metadata.namespace)
      POD_NAME:    (v1:metadata.name)
    Mounts:
      /var/lib/csi/sockets/pluginproxy/ from socket-dir (rw)
   openebs-zfs-plugin:
    Image:      openebs/zfs-driver:2.0.0
    Port:       <none>
    Host Port:  <none>
    Args:
      --endpoint=$(OPENEBS_CSI_ENDPOINT)
      --plugin=$(OPENEBS_CONTROLLER_DRIVER)
    Environment:
      OPENEBS_CONTROLLER_DRIVER:    controller
      OPENEBS_CSI_ENDPOINT:         unix:///var/lib/csi/sockets/pluginproxy/csi.sock
      OPENEBS_NAMESPACE:            openebs
      OPENEBS_IO_INSTALLER_TYPE:    zfs-operator
      OPENEBS_IO_ENABLE_ANALYTICS:  true
    Mounts:
      /var/lib/csi/sockets/pluginproxy/ from socket-dir (rw)
  Volumes:
   socket-dir:
    Type:               EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:             
    SizeLimit:          <unset>
  Priority Class Name:  openebs-zfs-csi-controller-critical
Volume Claims:          <none>
Events:                 <none>
debug finished in 0 seconds for k3s kubectl describe statefulset --all-namespaces


+--------------------------------------------------------------------------------+
+             k3s kubectl describe job --all-namespaces @1656907173              +
+--------------------------------------------------------------------------------+
No resources found
debug finished in 1 seconds for k3s kubectl describe job --all-namespaces


+--------------------------------------------------------------------------------+
+           k3s kubectl describe cronjob --all-namespaces @1656907174            +
+--------------------------------------------------------------------------------+
No resources found
debug finished in 0 seconds for k3s kubectl describe cronjob --all-namespaces


+--------------------------------------------------------------------------------+
+          k3s kubectl describe daemonsets --all-namespaces @1656907174          +
+--------------------------------------------------------------------------------+
Name:           openebs-zfs-node
Selector:       app=openebs-zfs-node
Node-Selector:  <none>
Labels:         objectset.rio.cattle.io/hash=4b1efd47d7f37713706abe9f8b0de3f0adf02e2c
                openebs.io/component-name=openebs-zfs-node
                openebs.io/version=ci
Annotations:    deprecated.daemonset.template.generation: 2
                objectset.rio.cattle.io/applied:
                  H4sIAAAAAAAA/6xWUY/aOBD+K5XVk1opIezSdnuR9oFCrl11FxBsew/VCjn2JPhw7Mh20uVW+e8nOwEChO2e1Lcknvlm5ptvJn5COGffQWkmBQoRznMdlBfIQ2smKArRGEMmxQIM8l...
                objectset.rio.cattle.io/id: 
                objectset.rio.cattle.io/owner-gvk: k3s.cattle.io/v1, Kind=Addon
                objectset.rio.cattle.io/owner-name: zfs-operator
                objectset.rio.cattle.io/owner-namespace: kube-system
Desired Number of Nodes Scheduled: 1
Current Number of Nodes Scheduled: 1
Number of Nodes Scheduled with Up-to-date Pods: 1
Number of Nodes Scheduled with Available Pods: 0
Number of Nodes Misscheduled: 0
Pods Status:  0 Running / 1 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           app=openebs-zfs-node
                    openebs.io/component-name=openebs-zfs-node
                    openebs.io/version=ci
                    role=openebs-zfs
  Service Account:  openebs-zfs-node-sa
  Containers:
   csi-node-driver-registrar:
    Image:      k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0
    Port:       <none>
    Host Port:  <none>
    Args:
      --v=5
      --csi-address=$(ADDRESS)
      --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)
    Environment:
      ADDRESS:               /plugin/csi.sock
      DRIVER_REG_SOCK_PATH:  /var/lib/kubelet/plugins/zfs-localpv/csi.sock
      KUBE_NODE_NAME:         (v1:spec.nodeName)
      NODE_DRIVER:           openebs-zfs
    Mounts:
      /plugin from plugin-dir (rw)
      /registration from registration-dir (rw)
   openebs-zfs-plugin:
    Image:      openebs/zfs-driver:2.0.0
    Port:       <none>
    Host Port:  <none>
    Args:
      --nodename=$(OPENEBS_NODE_NAME)
      --endpoint=$(OPENEBS_CSI_ENDPOINT)
      --plugin=$(OPENEBS_NODE_DRIVER)
    Environment:
      OPENEBS_NODE_NAME:      (v1:spec.nodeName)
      OPENEBS_CSI_ENDPOINT:  unix:///plugin/csi.sock
      OPENEBS_NODE_DRIVER:   agent
      OPENEBS_NAMESPACE:     openebs
      ALLOWED_TOPOLOGIES:    All
    Mounts:
      /dev from device-dir (rw)
      /home/keys from encr-keys (rw)
      /host from host-root (ro)
      /plugin from plugin-dir (rw)
      /sbin/zfs from chroot-zfs (rw,path="zfs")
      /var/lib/kubelet/ from pods-mount-dir (rw)
  Volumes:
   device-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /dev
    HostPathType:  Directory
   encr-keys:
    Type:          HostPath (bare host directory volume)
    Path:          /home/keys
    HostPathType:  DirectoryOrCreate
   chroot-zfs:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      openebs-zfspv-bin
    Optional:  false
   host-root:
    Type:          HostPath (bare host directory volume)
    Path:          /
    HostPathType:  Directory
   registration-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/plugins_registry/
    HostPathType:  DirectoryOrCreate
   plugin-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/plugins/zfs-localpv/
    HostPathType:  DirectoryOrCreate
   pods-mount-dir:
    Type:               HostPath (bare host directory volume)
    Path:               /var/lib/kubelet/
    HostPathType:       Directory
  Priority Class Name:  openebs-zfs-csi-node-critical
Events:                 <none>


Name:           svclb-openldap-ldaps
Selector:       app=svclb-openldap-ldaps
Node-Selector:  <none>
Labels:         objectset.rio.cattle.io/hash=dc0ead4565c62b378475f4d0b27aa3e2f6c5cc60
                svccontroller.k3s.cattle.io/nodeselector=false
Annotations:    deprecated.daemonset.template.generation: 2
                objectset.rio.cattle.io/applied:
                  H4sIAAAAAAAA/5xUX2/iOBD/Kqd5TtIAbaCR7qFq+1DdLUXA3ssKVRN7KD4c27InuaIq3/3kQFv2Frqre0lke+Yn//6MXwGd+ot8UNZACehcuGgHkMBWGQkl3CHV1iyIIYGaGCUyQv...
                objectset.rio.cattle.io/id: svccontroller
                objectset.rio.cattle.io/owner-gvk: /v1, Kind=Service
                objectset.rio.cattle.io/owner-name: openldap-ldaps
                objectset.rio.cattle.io/owner-namespace: ix-openldap
Desired Number of Nodes Scheduled: 1
Current Number of Nodes Scheduled: 1
Number of Nodes Scheduled with Up-to-date Pods: 1
Number of Nodes Scheduled with Available Pods: 1
Number of Nodes Misscheduled: 0
Pods Status:  1 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:  app=svclb-openldap-ldaps
           svccontroller.k3s.cattle.io/svcname=openldap-ldaps
  Containers:
   lb-tcp-636:
    Image:      rancher/klipper-lb:v0.3.4
    Port:       636/TCP
    Host Port:  636/TCP
    Environment:
      SRC_PORT:    636
      DEST_PROTO:  TCP
      DEST_PORT:   636
      DEST_IPS:    172.17.6.182
    Mounts:        <none>
  Volumes:         <none>
Events:            <none>


Name:           svclb-aria2
Selector:       app=svclb-aria2
Node-Selector:  <none>
Labels:         objectset.rio.cattle.io/hash=c9211a23c5478bba6a06832b54cde0d5510472e0
                svccontroller.k3s.cattle.io/nodeselector=false
Annotations:    deprecated.daemonset.template.generation: 2
                objectset.rio.cattle.io/applied:
                  H4sIAAAAAAAA/5xUzW7jNhB+lWLOkiLJvxHQQ5DkELTrGLa3l4URjKhxzJoiCXKkxgj07gVlJ3Gy3jToSSBn5gO/n9EzoJV/kfPSaCgArfUXbQYR7KSuoIAbpNroJTFEUBNjhYxQPA...
                objectset.rio.cattle.io/id: svccontroller
                objectset.rio.cattle.io/owner-gvk: /v1, Kind=Service
                objectset.rio.cattle.io/owner-name: aria2
                objectset.rio.cattle.io/owner-namespace: ix-aria2
Desired Number of Nodes Scheduled: 1
Current Number of Nodes Scheduled: 1
Number of Nodes Scheduled with Up-to-date Pods: 1
Number of Nodes Scheduled with Available Pods: 1
Number of Nodes Misscheduled: 0
Pods Status:  1 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:  app=svclb-aria2
           svccontroller.k3s.cattle.io/svcname=aria2
  Containers:
   lb-tcp-6800:
    Image:      rancher/klipper-lb:v0.3.4
    Port:       6800/TCP
    Host Port:  6800/TCP
    Environment:
      SRC_PORT:    6800
      DEST_PROTO:  TCP
      DEST_PORT:   6800
      DEST_IPS:    172.17.134.236
    Mounts:        <none>
  Volumes:         <none>
Events:            <none>


Name:           svclb-aria2-listen
Selector:       app=svclb-aria2-listen
Node-Selector:  <none>
Labels:         objectset.rio.cattle.io/hash=9043459edc1e01677d62f51caea6523c10346d7a
                svccontroller.k3s.cattle.io/nodeselector=false
Annotations:    deprecated.daemonset.template.generation: 2
                objectset.rio.cattle.io/applied:
                  H4sIAAAAAAAA/5xUzW7jNhB+lWLOkiL5J3YE9BAkOQTtOobt7WVhBCNyHLOmSIIcqTECvXtB2Um8W2cb7EkgZ+aDvp/hC6BTf5EPyhooAZ0LF20BCeyUkVDCLVJtzZIYEqiJUSIjlC...
                objectset.rio.cattle.io/id: svccontroller
                objectset.rio.cattle.io/owner-gvk: /v1, Kind=Service
                objectset.rio.cattle.io/owner-name: aria2-listen
                objectset.rio.cattle.io/owner-namespace: ix-aria2
Desired Number of Nodes Scheduled: 1
Current Number of Nodes Scheduled: 1
Number of Nodes Scheduled with Up-to-date Pods: 1
Number of Nodes Scheduled with Available Pods: 1
Number of Nodes Misscheduled: 0
Pods Status:  1 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:  app=svclb-aria2-listen
           svccontroller.k3s.cattle.io/svcname=aria2-listen
  Containers:
   lb-tcp-6888:
    Image:      rancher/klipper-lb:v0.3.4
    Port:       6888/TCP
    Host Port:  6888/TCP
    Environment:
      SRC_PORT:    6888
      DEST_PROTO:  TCP
      DEST_PORT:   6888
      DEST_IPS:    172.17.185.94
    Mounts:        <none>
  Volumes:         <none>
Events:            <none>


Name:           svclb-ariang-custom-app
Selector:       app=svclb-ariang-custom-app
Node-Selector:  <none>
Labels:         objectset.rio.cattle.io/hash=d658f3b37c73a50aba9f2e9f775e053e47a3e690
                svccontroller.k3s.cattle.io/nodeselector=false
Annotations:    deprecated.daemonset.template.generation: 2
                objectset.rio.cattle.io/applied:
                  H4sIAAAAAAAA/5xUT0/rOBD/Kqs5J6EhhFJLe0DAAe0+qNq+vTxVaOJMW28d27InWSqU775yWqBvt7BoT5U9M7/m92f8AujUH+SDsgYEoHPhrMshga0yNQi4RWqsmRNDAg0x1sgI4g...
                objectset.rio.cattle.io/id: svccontroller
                objectset.rio.cattle.io/owner-gvk: /v1, Kind=Service
                objectset.rio.cattle.io/owner-name: ariang-custom-app
                objectset.rio.cattle.io/owner-namespace: ix-ariang
Desired Number of Nodes Scheduled: 1
Current Number of Nodes Scheduled: 1
Number of Nodes Scheduled with Up-to-date Pods: 1
Number of Nodes Scheduled with Available Pods: 1
Number of Nodes Misscheduled: 0
Pods Status:  1 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:  app=svclb-ariang-custom-app
           svccontroller.k3s.cattle.io/svcname=ariang-custom-app
  Containers:
   lb-tcp-6880:
    Image:      rancher/klipper-lb:v0.3.4
    Port:       6880/TCP
    Host Port:  6880/TCP
    Environment:
      SRC_PORT:    6880
      DEST_PROTO:  TCP
      DEST_PORT:   6880
      DEST_IPS:    172.17.132.154
    Mounts:        <none>
  Volumes:         <none>
Events:            <none>


Name:           svclb-openldap
Selector:       app=svclb-openldap
Node-Selector:  <none>
Labels:         objectset.rio.cattle.io/hash=a03bad55cd6bd1f07ea76113367b1da4e8a1dbd3
                svccontroller.k3s.cattle.io/nodeselector=false
Annotations:    deprecated.daemonset.template.generation: 2
                objectset.rio.cattle.io/applied:
                  H4sIAAAAAAAA/5xUX2/iuBf9Kj/d5yQlJYUS6fdQtX2odociYPZlhKob+1K8OLZl32SLqnz3lQOlzJTOjPYJ2T73iPPn5hXQqb/IB2UNlIDOhYs2hwS2ykgo4Q6ptmZBDAnUxCiREc...
                objectset.rio.cattle.io/id: svccontroller
                objectset.rio.cattle.io/owner-gvk: /v1, Kind=Service
                objectset.rio.cattle.io/owner-name: openldap
                objectset.rio.cattle.io/owner-namespace: ix-openldap
Desired Number of Nodes Scheduled: 1
Current Number of Nodes Scheduled: 1
Number of Nodes Scheduled with Up-to-date Pods: 1
Number of Nodes Scheduled with Available Pods: 1
Number of Nodes Misscheduled: 0
Pods Status:  1 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:  app=svclb-openldap
           svccontroller.k3s.cattle.io/svcname=openldap
  Containers:
   lb-tcp-389:
    Image:      rancher/klipper-lb:v0.3.4
    Port:       389/TCP
    Host Port:  389/TCP
    Environment:
      SRC_PORT:    389
      DEST_PROTO:  TCP
      DEST_PORT:   389
      DEST_IPS:    172.17.145.141
    Mounts:        <none>
  Volumes:         <none>
Events:            <none>


Name:           svclb-jellyfin
Selector:       app=svclb-jellyfin
Node-Selector:  <none>
Labels:         objectset.rio.cattle.io/hash=03de1f4eb2112eaa99fa6112a80bde82fbc4810a
                svccontroller.k3s.cattle.io/nodeselector=false
Annotations:    deprecated.daemonset.template.generation: 2
                objectset.rio.cattle.io/applied:
                  H4sIAAAAAAAA/5xUX0/jOBD/Kqd5TkJSOGgt3QMCHtDdlqrt3suqQhN7Qr117Mie5KhQvvvJaSndpeyhe4rsmfkpvz/jF8BG/00+aGdBADZNOOsKSGCjrQIBt0i1swtiSKAmRoWMIF...
                objectset.rio.cattle.io/id: svccontroller
                objectset.rio.cattle.io/owner-gvk: /v1, Kind=Service
                objectset.rio.cattle.io/owner-name: jellyfin
                objectset.rio.cattle.io/owner-namespace: ix-jellyfin
Desired Number of Nodes Scheduled: 1
Current Number of Nodes Scheduled: 1
Number of Nodes Scheduled with Up-to-date Pods: 1
Number of Nodes Scheduled with Available Pods: 1
Number of Nodes Misscheduled: 0
Pods Status:  1 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:  app=svclb-jellyfin
           svccontroller.k3s.cattle.io/svcname=jellyfin
  Containers:
   lb-tcp-8096:
    Image:      rancher/klipper-lb:v0.3.4
    Port:       8096/TCP
    Host Port:  8096/TCP
    Environment:
      SRC_PORT:    8096
      DEST_PROTO:  TCP
      DEST_PORT:   8096
      DEST_IPS:    172.17.130.42
    Mounts:        <none>
  Volumes:         <none>
Events:            <none>
debug finished in 0 seconds for k3s kubectl describe daemonsets --all-namespaces


+--------------------------------------------------------------------------------+
+                            docker ps -a @1656907174                            +
+--------------------------------------------------------------------------------+
CONTAINER ID   IMAGE                        COMMAND                  CREATED        STATUS        PORTS     NAMES
f838a18b3148   2e17748d7c44                 "/usr/local/bin/zfs-…"   10 hours ago   Up 10 hours             k8s_openebs-zfs-plugin_openebs-zfs-controller-0_kube-system_a277d60e-69ba-4e90-9eb9-197ba5ebd72d_0
23a4580cd74c   f1d8a00ae690                 "/snapshot-controlle…"   10 hours ago   Up 10 hours             k8s_snapshot-controller_openebs-zfs-controller-0_kube-system_a277d60e-69ba-4e90-9eb9-197ba5ebd72d_0
1d86c8904158   da32a49a903a                 "/csi-snapshotter --…"   10 hours ago   Up 10 hours             k8s_csi-snapshotter_openebs-zfs-controller-0_kube-system_a277d60e-69ba-4e90-9eb9-197ba5ebd72d_0
14f362d0cc9c   2e17748d7c44                 "/usr/local/bin/zfs-…"   10 hours ago   Up 10 hours             k8s_openebs-zfs-plugin_openebs-zfs-node-5zm8n_kube-system_6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a_0
8dfc7cc5eeeb   rancher/mirrored-pause:3.6   "/pause"                 10 hours ago   Up 10 hours             k8s_POD_openebs-zfs-controller-0_kube-system_a277d60e-69ba-4e90-9eb9-197ba5ebd72d_0
36ad7daca909   746788bcc27e                 "entry"                  10 hours ago   Up 10 hours             k8s_lb-tcp-8096_svclb-jellyfin-g8sj9_ix-jellyfin_4d46719e-8c66-4550-9379-6969ad0a654b_0
5fca6b5e0b16   746788bcc27e                 "entry"                  10 hours ago   Up 10 hours             k8s_lb-tcp-389_svclb-openldap-4rjmb_ix-openldap_137d2385-cdba-4630-b874-a359c113a009_0
3da8a838c657   746788bcc27e                 "entry"                  10 hours ago   Up 10 hours             k8s_lb-tcp-6880_svclb-ariang-custom-app-svbxp_ix-ariang_cb9f9ee4-1aec-4dd9-85dc-70984a38a07e_0
ed624ae5bad3   746788bcc27e                 "entry"                  10 hours ago   Up 10 hours             k8s_lb-tcp-6800_svclb-aria2-fsw89_ix-aria2_7dbeeeeb-e536-42e8-88e5-8ad9ee9515bd_0
9cec67af1af7   746788bcc27e                 "entry"                  10 hours ago   Up 10 hours             k8s_lb-tcp-6888_svclb-aria2-listen-5zljr_ix-aria2_b6b217e0-696b-4efc-a6a7-03ee76fd13c0_0
c288dbe01381   746788bcc27e                 "entry"                  10 hours ago   Up 10 hours             k8s_lb-tcp-636_svclb-openldap-ldaps-vzm7c_ix-openldap_d360fd61-16b2-4b58-a3d7-259ae6a7ac9b_0
d08c481e17bb   rancher/mirrored-pause:3.6   "/pause"                 10 hours ago   Up 10 hours             k8s_POD_openebs-zfs-node-5zm8n_kube-system_6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a_0
e374a4accc4c   rancher/mirrored-pause:3.6   "/pause"                 10 hours ago   Up 10 hours             k8s_POD_svclb-openldap-4rjmb_ix-openldap_137d2385-cdba-4630-b874-a359c113a009_0
eb7b90427c30   rancher/mirrored-pause:3.6   "/pause"                 10 hours ago   Up 10 hours             k8s_POD_svclb-ariang-custom-app-svbxp_ix-ariang_cb9f9ee4-1aec-4dd9-85dc-70984a38a07e_0
0ce5c606aeac   rancher/mirrored-pause:3.6   "/pause"                 10 hours ago   Up 10 hours             k8s_POD_svclb-aria2-fsw89_ix-aria2_7dbeeeeb-e536-42e8-88e5-8ad9ee9515bd_0
f6ccb249b045   rancher/mirrored-pause:3.6   "/pause"                 10 hours ago   Up 10 hours             k8s_POD_svclb-aria2-listen-5zljr_ix-aria2_b6b217e0-696b-4efc-a6a7-03ee76fd13c0_0
098214b4b9ad   rancher/mirrored-pause:3.6   "/pause"                 10 hours ago   Up 10 hours             k8s_POD_svclb-jellyfin-g8sj9_ix-jellyfin_4d46719e-8c66-4550-9379-6969ad0a654b_0
ee6ad3368c37   rancher/mirrored-pause:3.6   "/pause"                 10 hours ago   Up 10 hours             k8s_POD_svclb-openldap-ldaps-vzm7c_ix-openldap_d360fd61-16b2-4b58-a3d7-259ae6a7ac9b_0
47f4c930a8a2   99376d8f35e0                 "/coredns -conf /etc…"   10 hours ago   Up 10 hours             k8s_coredns_coredns-d76bd69b-ks26g_kube-system_806c0aab-116c-4dc3-ab0c-7ac1e5673491_0
bc0c0f8eed00   rancher/mirrored-pause:3.6   "/pause"                 10 hours ago   Up 10 hours             k8s_POD_coredns-d76bd69b-ks26g_kube-system_806c0aab-116c-4dc3-ab0c-7ac1e5673491_0
debug finished in 1 seconds for docker ps -a


+--------------------------------------------------------------------------------+
+                          docker images -a @1656907175                          +
+--------------------------------------------------------------------------------+
REPOSITORY                                         TAG       IMAGE ID       CREATED         SIZE
rancher/mirrored-coredns-coredns                   1.9.1     99376d8f35e0   3 months ago    49.5MB
p3terx/ariang                                      latest    fb32bf220507   3 months ago    2.69MB
openebs/zfs-driver                                 2.0.0     2e17748d7c44   6 months ago    176MB
ghcr.io/truecharts/alpine                          <none>    f937e4e4c343   7 months ago    23.1MB
rancher/klipper-lb                                 v0.3.4    746788bcc27e   8 months ago    8.08MB
openebs/zfs-driver                                 1.9.3     10dbacde5cc2   9 months ago    175MB
tccr.io/truecharts/jellyfin                        <none>    61f1173b3d21   10 months ago   717MB
rancher/mirrored-pause                             3.6       6270bb605e12   10 months ago   683kB
tccr.io/truecharts/aria2-pro                       <none>    c8918cdf83bc   10 months ago   29.1MB
rancher/coredns-coredns                            1.8.3     3885a5b7f138   16 months ago   43.5MB
tccr.io/truecharts/openldap                        <none>    99f637700948   16 months ago   257MB
k8s.gcr.io/sig-storage/csi-snapshotter             v4.0.0    da32a49a903a   18 months ago   49.5MB
k8s.gcr.io/sig-storage/snapshot-controller         v4.0.0    f1d8a00ae690   18 months ago   46.6MB
k8s.gcr.io/sig-storage/csi-resizer                 v1.1.0    a8fe79377034   18 months ago   49.2MB
k8s.gcr.io/sig-storage/csi-provisioner             v2.1.0    e0d187f105d6   18 months ago   51.6MB
k8s.gcr.io/sig-storage/csi-node-driver-registrar   v2.1.0    ef2b13b2a066   18 months ago   19.6MB
rancher/klipper-lb                                 v0.1.2    897ce3c5fc8f   3 years ago     6.1MB
rancher/pause                                      3.1       da86e6ba6ca1   4 years ago     742kB
debug finished in 0 seconds for docker images -a


+--------------------------------------------------------------------------------+
+                     K3s Logs (last 1000 lines) @1656907175                     +
+--------------------------------------------------------------------------------+
Jul 04 11:36:43 truenas k3s[12736]: E0704 11:36:43.369033   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:36:45 truenas k3s[12736]: E0704 11:36:45.378085   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:36:46 truenas k3s[12736]: E0704 11:36:46.350917   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:36:47 truenas k3s[12736]: E0704 11:36:47.379951   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:36:49 truenas k3s[12736]: E0704 11:36:49.370294   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:36:51 truenas k3s[12736]: E0704 11:36:51.378095   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:36:52 truenas k3s[12736]: E0704 11:36:52.353664   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:36:53 truenas k3s[12736]: E0704 11:36:53.368897   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:36:55 truenas k3s[12736]: E0704 11:36:55.399668   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:36:57 truenas k3s[12736]: E0704 11:36:57.377691   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:36:58 truenas k3s[12736]: E0704 11:36:58.351325   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:36:59 truenas k3s[12736]: E0704 11:36:59.376886   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:37:01 truenas k3s[12736]: E0704 11:37:01.379019   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:37:03 truenas k3s[12736]: {"level":"warn","ts":"2022-07-04T11:37:03.046+0800","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ed6c40/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Jul 04 11:37:03 truenas k3s[12736]: E0704 11:37:03.377935   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:37:05 truenas k3s[12736]: E0704 11:37:05.370094   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:37:07 truenas k3s[12736]: E0704 11:37:07.352748   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:37:07 truenas k3s[12736]: E0704 11:37:07.372481   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:37:09 truenas k3s[12736]: E0704 11:37:09.369874   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:37:11 truenas k3s[12736]: E0704 11:37:11.351577   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:37:11 truenas k3s[12736]: E0704 11:37:11.384341   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:37:13 truenas k3s[12736]: E0704 11:37:13.369350   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:37:15 truenas k3s[12736]: E0704 11:37:15.379120   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:37:17 truenas k3s[12736]: E0704 11:37:17.369483   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:37:19 truenas k3s[12736]: E0704 11:37:19.378748   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:37:20 truenas k3s[12736]: E0704 11:37:20.352575   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:37:21 truenas k3s[12736]: E0704 11:37:21.370144   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:37:23 truenas k3s[12736]: E0704 11:37:23.380615   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:37:24 truenas k3s[12736]: E0704 11:37:24.351354   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:37:25 truenas k3s[12736]: E0704 11:37:25.378397   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:37:27 truenas k3s[12736]: E0704 11:37:27.369748   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:37:29 truenas k3s[12736]: E0704 11:37:29.378419   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:37:31 truenas k3s[12736]: E0704 11:37:31.370196   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:37:33 truenas k3s[12736]: E0704 11:37:33.380834   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:37:35 truenas k3s[12736]: E0704 11:37:35.351089   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:37:35 truenas k3s[12736]: E0704 11:37:35.352679   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:37:35 truenas k3s[12736]: E0704 11:37:35.369717   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:37:37 truenas k3s[12736]: E0704 11:37:37.377985   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:37:39 truenas k3s[12736]: E0704 11:37:39.368901   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:37:41 truenas k3s[12736]: E0704 11:37:41.379562   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:37:42 truenas k3s[12736]: E0704 11:37:42.451111   12736 goroutinemap.go:150] Operation for "/var/lib/kubelet/plugins_registry/zfs.csi.openebs.io-reg.sock" failed. No retries permitted until 2022-07-04 11:39:44.451069517 +0800 CST m=+33503.850217836 (durationBeforeRetry 2m2s). Error: RegisterPlugin error -- dial failed at socket /var/lib/kubelet/plugins_registry/zfs.csi.openebs.io-reg.sock, err: failed to dial socket /var/lib/kubelet/plugins_registry/zfs.csi.openebs.io-reg.sock, err: context deadline exceeded
Jul 04 11:37:43 truenas k3s[12736]: E0704 11:37:43.373066   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:37:45 truenas k3s[12736]: E0704 11:37:45.378585   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:37:46 truenas k3s[12736]: {"level":"warn","ts":"2022-07-04T11:37:46.714+0800","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ed6c40/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Jul 04 11:37:47 truenas k3s[12736]: E0704 11:37:47.369664   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:37:49 truenas k3s[12736]: E0704 11:37:49.350834   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:37:49 truenas k3s[12736]: E0704 11:37:49.352516   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:37:49 truenas k3s[12736]: E0704 11:37:49.377745   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:37:51 truenas k3s[12736]: E0704 11:37:51.369037   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:37:53 truenas k3s[12736]: E0704 11:37:53.378874   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:37:55 truenas k3s[12736]: E0704 11:37:55.377589   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:37:57 truenas k3s[12736]: E0704 11:37:57.371923   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:37:59 truenas k3s[12736]: E0704 11:37:59.379006   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:38:01 truenas k3s[12736]: E0704 11:38:01.351249   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:38:01 truenas k3s[12736]: E0704 11:38:01.352649   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:38:01 truenas k3s[12736]: E0704 11:38:01.379776   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:38:03 truenas k3s[12736]: E0704 11:38:03.364871   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:38:05 truenas k3s[12736]: E0704 11:38:05.377022   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:38:07 truenas k3s[12736]: E0704 11:38:07.368397   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:38:09 truenas k3s[12736]: E0704 11:38:09.377429   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:38:11 truenas k3s[12736]: E0704 11:38:11.370802   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:38:12 truenas k3s[12736]: E0704 11:38:12.352257   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:38:13 truenas k3s[12736]: E0704 11:38:13.384322   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:38:15 truenas k3s[12736]: E0704 11:38:15.351529   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:38:15 truenas k3s[12736]: E0704 11:38:15.378230   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:38:17 truenas k3s[12736]: E0704 11:38:17.391688   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:38:19 truenas k3s[12736]: E0704 11:38:19.377900   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:38:21 truenas k3s[12736]: E0704 11:38:21.379692   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:38:23 truenas k3s[12736]: E0704 11:38:23.353890   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:38:23 truenas k3s[12736]: E0704 11:38:23.380804   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:38:25 truenas k3s[12736]: E0704 11:38:25.369357   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:38:26 truenas k3s[12736]: {"level":"warn","ts":"2022-07-04T11:38:26.995+0800","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ed6c40/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Jul 04 11:38:27 truenas k3s[12736]: E0704 11:38:27.376990   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:38:29 truenas k3s[12736]: E0704 11:38:29.379507   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:38:30 truenas k3s[12736]: E0704 11:38:30.350970   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:38:31 truenas k3s[12736]: E0704 11:38:31.378515   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:38:33 truenas k3s[12736]: E0704 11:38:33.373310   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:38:34 truenas k3s[12736]: E0704 11:38:34.352129   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:38:35 truenas k3s[12736]: E0704 11:38:35.379121   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:38:37 truenas k3s[12736]: E0704 11:38:37.369195   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:38:39 truenas k3s[12736]: E0704 11:38:39.377005   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:38:41 truenas k3s[12736]: E0704 11:38:41.351075   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:38:41 truenas k3s[12736]: E0704 11:38:41.377200   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:38:43 truenas k3s[12736]: E0704 11:38:43.370737   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:38:45 truenas k3s[12736]: E0704 11:38:45.352659   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:38:45 truenas k3s[12736]: E0704 11:38:45.376966   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:38:47 truenas k3s[12736]: E0704 11:38:47.370038   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:38:49 truenas k3s[12736]: E0704 11:38:49.377965   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:38:51 truenas k3s[12736]: E0704 11:38:51.369256   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:38:53 truenas k3s[12736]: E0704 11:38:53.382114   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:38:54 truenas k3s[12736]: E0704 11:38:54.350912   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:38:55 truenas k3s[12736]: E0704 11:38:55.370874   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:38:57 truenas k3s[12736]: E0704 11:38:57.352989   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:38:57 truenas k3s[12736]: E0704 11:38:57.378937   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:38:58 truenas k3s[12736]: {"level":"warn","ts":"2022-07-04T11:38:58.402+0800","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ed6c40/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Jul 04 11:38:59 truenas k3s[12736]: E0704 11:38:59.371209   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:39:01 truenas k3s[12736]: E0704 11:39:01.380303   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:39:03 truenas k3s[12736]: E0704 11:39:03.370616   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:39:05 truenas k3s[12736]: E0704 11:39:05.350834   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:39:05 truenas k3s[12736]: E0704 11:39:05.377569   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:39:07 truenas k3s[12736]: E0704 11:39:07.369936   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:39:09 truenas k3s[12736]: E0704 11:39:09.376630   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:39:11 truenas k3s[12736]: E0704 11:39:11.352810   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:39:11 truenas k3s[12736]: E0704 11:39:11.380617   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:39:13 truenas k3s[12736]: E0704 11:39:13.373271   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:39:15 truenas k3s[12736]: E0704 11:39:15.377507   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:39:17 truenas k3s[12736]: E0704 11:39:17.351309   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:39:17 truenas k3s[12736]: E0704 11:39:17.378358   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:39:19 truenas k3s[12736]: E0704 11:39:19.370629   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:39:21 truenas k3s[12736]: E0704 11:39:21.377874   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:39:23 truenas k3s[12736]: E0704 11:39:23.378995   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:39:24 truenas k3s[12736]: E0704 11:39:24.353214   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:39:25 truenas k3s[12736]: E0704 11:39:25.369115   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:39:27 truenas k3s[12736]: E0704 11:39:27.376522   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:39:29 truenas k3s[12736]: E0704 11:39:29.367254   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:39:30 truenas k3s[12736]: {"level":"warn","ts":"2022-07-04T11:39:30.011+0800","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ed6c40/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Jul 04 11:39:30 truenas k3s[12736]: E0704 11:39:30.350687   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:39:31 truenas k3s[12736]: E0704 11:39:31.377990   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:39:33 truenas k3s[12736]: E0704 11:39:33.370018   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:39:35 truenas k3s[12736]: E0704 11:39:35.378463   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:39:37 truenas k3s[12736]: E0704 11:39:37.353160   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:39:37 truenas k3s[12736]: E0704 11:39:37.378396   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:39:39 truenas k3s[12736]: E0704 11:39:39.369614   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:39:41 truenas k3s[12736]: E0704 11:39:41.378864   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:39:42 truenas k3s[12736]: E0704 11:39:42.351136   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:39:43 truenas k3s[12736]: E0704 11:39:43.378786   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:39:45 truenas k3s[12736]: E0704 11:39:45.378421   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:39:47 truenas k3s[12736]: E0704 11:39:47.369912   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:39:48 truenas k3s[12736]: E0704 11:39:48.352759   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:39:49 truenas k3s[12736]: E0704 11:39:49.379391   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:39:51 truenas k3s[12736]: E0704 11:39:51.369340   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:39:53 truenas k3s[12736]: E0704 11:39:53.377959   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:39:54 truenas k3s[12736]: E0704 11:39:54.522399   12736 goroutinemap.go:150] Operation for "/var/lib/kubelet/plugins_registry/zfs.csi.openebs.io-reg.sock" failed. No retries permitted until 2022-07-04 11:41:56.522362972 +0800 CST m=+33635.921511281 (durationBeforeRetry 2m2s). Error: RegisterPlugin error -- dial failed at socket /var/lib/kubelet/plugins_registry/zfs.csi.openebs.io-reg.sock, err: failed to dial socket /var/lib/kubelet/plugins_registry/zfs.csi.openebs.io-reg.sock, err: context deadline exceeded
Jul 04 11:39:55 truenas k3s[12736]: E0704 11:39:55.370494   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:39:57 truenas k3s[12736]: E0704 11:39:57.379041   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:39:59 truenas k3s[12736]: E0704 11:39:59.353423   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:39:59 truenas k3s[12736]: E0704 11:39:59.379541   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:40:01 truenas k3s[12736]: E0704 11:40:01.369589   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:40:03 truenas k3s[12736]: E0704 11:40:03.378845   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:40:05 truenas k3s[12736]: {"level":"warn","ts":"2022-07-04T11:40:05.141+0800","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ed6c40/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Jul 04 11:40:05 truenas k3s[12736]: E0704 11:40:05.370561   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:40:07 truenas k3s[12736]: E0704 11:40:07.379171   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:40:09 truenas k3s[12736]: E0704 11:40:09.378595   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:40:10 truenas k3s[12736]: E0704 11:40:10.450947   12736 remote_image.go:216] "PullImage from image service failed" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0"
Jul 04 11:40:10 truenas k3s[12736]: E0704 11:40:10.451037   12736 kuberuntime_image.go:51] "Failed to pull image" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0"
Jul 04 11:40:10 truenas k3s[12736]: E0704 11:40:10.451277   12736 kuberuntime_manager.go:919] container &Container{Name:csi-node-driver-registrar,Image:k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0,Command:[],Args:[--v=5 --csi-address=$(ADDRESS) --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:ADDRESS,Value:/plugin/csi.sock,ValueFrom:nil,},EnvVar{Name:DRIVER_REG_SOCK_PATH,Value:/var/lib/kubelet/plugins/zfs-localpv/csi.sock,ValueFrom:nil,},EnvVar{Name:KUBE_NODE_NAME,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:spec.nodeName,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},EnvVar{Name:NODE_DRIVER,Value:openebs-zfs,ValueFrom:nil,},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:plugin-dir,ReadOnly:false,MountPath:/plugin,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:registration-dir,ReadOnly:false,MountPath:/registration,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:kube-api-access-89mwk,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:&Lifecycle{PostStart:nil,PreStop:&LifecycleHandler{Exec:&ExecAction{Command:[/bin/sh -c rm -rf /registration/zfs-localpv /registration/zfs-localpv-reg.sock],},HTTPGet:nil,TCPSocket:nil,},},TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod openebs-zfs-node-5zm8n_kube-system(6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a): ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get "https://k8s.gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Jul 04 11:40:10 truenas k3s[12736]: E0704 11:40:10.451386   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ErrImagePull: \"rpc error: code = Unknown desc = Error response from daemon: Get \\\"https://k8s.gcr.io/v2/\\\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:40:11 truenas k3s[12736]: E0704 11:40:11.368782   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:40:13 truenas k3s[12736]: E0704 11:40:13.378701   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:40:14 truenas k3s[12736]: E0704 11:40:14.352509   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:40:15 truenas k3s[12736]: E0704 11:40:15.370190   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:40:17 truenas k3s[12736]: E0704 11:40:17.378777   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:40:19 truenas k3s[12736]: E0704 11:40:19.369515   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:40:21 truenas k3s[12736]: E0704 11:40:21.377561   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:40:23 truenas k3s[12736]: E0704 11:40:23.377540   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:40:25 truenas k3s[12736]: E0704 11:40:25.351412   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:40:25 truenas k3s[12736]: E0704 11:40:25.378070   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:40:27 truenas k3s[12736]: E0704 11:40:27.353330   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:40:27 truenas k3s[12736]: E0704 11:40:27.369098   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:40:29 truenas k3s[12736]: E0704 11:40:29.380286   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:40:31 truenas k3s[12736]: E0704 11:40:31.369645   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:40:33 truenas k3s[12736]: E0704 11:40:33.377823   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:40:35 truenas k3s[12736]: E0704 11:40:35.369004   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:40:37 truenas k3s[12736]: E0704 11:40:37.350980   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:40:37 truenas k3s[12736]: E0704 11:40:37.376403   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:40:39 truenas k3s[12736]: E0704 11:40:39.373210   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:40:41 truenas k3s[12736]: E0704 11:40:41.379209   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:40:43 truenas k3s[12736]: E0704 11:40:43.368633   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:40:45 truenas k3s[12736]: E0704 11:40:45.379160   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:40:47 truenas k3s[12736]: E0704 11:40:47.365215   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:40:47 truenas k3s[12736]: {"level":"warn","ts":"2022-07-04T11:40:47.414+0800","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ed6c40/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Jul 04 11:40:49 truenas k3s[12736]: E0704 11:40:49.378342   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:40:50 truenas k3s[12736]: E0704 11:40:50.350312   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:40:51 truenas k3s[12736]: E0704 11:40:51.369726   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:40:53 truenas k3s[12736]: E0704 11:40:53.377397   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:40:55 truenas k3s[12736]: E0704 11:40:55.389191   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:40:56 truenas k3s[12736]: E0704 11:40:56.428511   12736 remote_image.go:216] "PullImage from image service failed" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-resizer:v1.2.0"
Jul 04 11:40:56 truenas k3s[12736]: E0704 11:40:56.428559   12736 kuberuntime_image.go:51] "Failed to pull image" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-resizer:v1.2.0"
Jul 04 11:40:56 truenas k3s[12736]: E0704 11:40:56.428683   12736 kuberuntime_manager.go:919] container &Container{Name:csi-resizer,Image:k8s.gcr.io/sig-storage/csi-resizer:v1.2.0,Command:[],Args:[--v=5 --csi-address=$(ADDRESS) --leader-election],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:ADDRESS,Value:/var/lib/csi/sockets/pluginproxy/csi.sock,ValueFrom:nil,},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:socket-dir,ReadOnly:false,MountPath:/var/lib/csi/sockets/pluginproxy/,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:kube-api-access-8tngl,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod openebs-zfs-controller-0_kube-system(a277d60e-69ba-4e90-9eb9-197ba5ebd72d): ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get "https://k8s.gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Jul 04 11:40:57 truenas k3s[12736]: E0704 11:40:57.370612   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:40:59 truenas k3s[12736]: E0704 11:40:59.378855   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:41:01 truenas k3s[12736]: E0704 11:41:01.351472   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:41:01 truenas k3s[12736]: E0704 11:41:01.377982   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:41:03 truenas k3s[12736]: E0704 11:41:03.370369   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:41:05 truenas k3s[12736]: E0704 11:41:05.376291   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:41:07 truenas k3s[12736]: E0704 11:41:07.378592   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:41:09 truenas k3s[12736]: E0704 11:41:09.369238   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:41:11 truenas k3s[12736]: E0704 11:41:11.377423   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:41:11 truenas k3s[12736]: E0704 11:41:11.716846   12736 remote_image.go:216] "PullImage from image service failed" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": context deadline exceeded" image="k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0"
Jul 04 11:41:11 truenas k3s[12736]: E0704 11:41:11.716919   12736 kuberuntime_image.go:51] "Failed to pull image" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": context deadline exceeded" image="k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0"
Jul 04 11:41:11 truenas k3s[12736]: E0704 11:41:11.717154   12736 kuberuntime_manager.go:919] container &Container{Name:csi-provisioner,Image:k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0,Command:[],Args:[--csi-address=$(ADDRESS) --v=5 --feature-gates=Topology=true --strict-topology --leader-election --extra-create-metadata=true --enable-capacity=true --default-fstype=ext4],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:ADDRESS,Value:/var/lib/csi/sockets/pluginproxy/csi.sock,ValueFrom:nil,},EnvVar{Name:NAMESPACE,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},EnvVar{Name:POD_NAME,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.name,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:socket-dir,ReadOnly:false,MountPath:/var/lib/csi/sockets/pluginproxy/,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:kube-api-access-8tngl,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod openebs-zfs-controller-0_kube-system(a277d60e-69ba-4e90-9eb9-197ba5ebd72d): ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get "https://k8s.gcr.io/v2/": context deadline exceeded
Jul 04 11:41:11 truenas k3s[12736]: E0704 11:41:11.717291   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ErrImagePull: \"rpc error: code = Unknown desc = Error response from daemon: Get \\\"https://k8s.gcr.io/v2/\\\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\", failed to \"StartContainer\" for \"csi-provisioner\" with ErrImagePull: \"rpc error: code = Unknown desc = Error response from daemon: Get \\\"https://k8s.gcr.io/v2/\\\": context deadline exceeded\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:41:13 truenas k3s[12736]: E0704 11:41:13.369439   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:41:14 truenas k3s[12736]: E0704 11:41:14.350884   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:41:15 truenas k3s[12736]: E0704 11:41:15.376765   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:41:17 truenas k3s[12736]: E0704 11:41:17.378944   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:41:18 truenas k3s[12736]: {"level":"warn","ts":"2022-07-04T11:41:18.366+0800","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ed6c40/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Jul 04 11:41:19 truenas k3s[12736]: E0704 11:41:19.369852   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:41:21 truenas k3s[12736]: E0704 11:41:21.378781   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:41:22 truenas k3s[12736]: E0704 11:41:22.351936   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:41:23 truenas k3s[12736]: E0704 11:41:23.377825   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:41:25 truenas k3s[12736]: E0704 11:41:25.368902   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:41:26 truenas k3s[12736]: E0704 11:41:26.350798   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:41:27 truenas k3s[12736]: E0704 11:41:27.377116   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:41:29 truenas k3s[12736]: E0704 11:41:29.369082   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:41:31 truenas k3s[12736]: E0704 11:41:31.381265   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:41:33 truenas k3s[12736]: E0704 11:41:33.379312   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:41:35 truenas k3s[12736]: E0704 11:41:35.352517   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:41:35 truenas k3s[12736]: E0704 11:41:35.371973   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:41:37 truenas k3s[12736]: E0704 11:41:37.377546   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:41:38 truenas k3s[12736]: E0704 11:41:38.350432   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:41:39 truenas k3s[12736]: E0704 11:41:39.372206   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:41:41 truenas k3s[12736]: E0704 11:41:41.378736   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:41:43 truenas k3s[12736]: E0704 11:41:43.369648   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:41:45 truenas k3s[12736]: E0704 11:41:45.379070   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:41:47 truenas k3s[12736]: E0704 11:41:47.353760   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:41:47 truenas k3s[12736]: E0704 11:41:47.378773   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:41:49 truenas k3s[12736]: E0704 11:41:49.369805   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:41:51 truenas k3s[12736]: E0704 11:41:51.379154   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:41:51 truenas k3s[12736]: {"level":"warn","ts":"2022-07-04T11:41:51.579+0800","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ed6c40/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Jul 04 11:41:52 truenas k3s[12736]: E0704 11:41:52.350636   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:41:53 truenas k3s[12736]: E0704 11:41:53.369722   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:41:55 truenas k3s[12736]: E0704 11:41:55.377265   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:41:57 truenas k3s[12736]: E0704 11:41:57.368178   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:41:59 truenas k3s[12736]: E0704 11:41:59.353095   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:41:59 truenas k3s[12736]: E0704 11:41:59.378721   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:42:01 truenas k3s[12736]: E0704 11:42:01.377367   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:42:03 truenas k3s[12736]: E0704 11:42:03.369540   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:42:05 truenas k3s[12736]: E0704 11:42:05.351359   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:42:05 truenas k3s[12736]: E0704 11:42:05.377777   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:42:06 truenas k3s[12736]: E0704 11:42:06.601096   12736 goroutinemap.go:150] Operation for "/var/lib/kubelet/plugins_registry/zfs.csi.openebs.io-reg.sock" failed. No retries permitted until 2022-07-04 11:44:08.601058552 +0800 CST m=+33768.000206871 (durationBeforeRetry 2m2s). Error: RegisterPlugin error -- dial failed at socket /var/lib/kubelet/plugins_registry/zfs.csi.openebs.io-reg.sock, err: failed to dial socket /var/lib/kubelet/plugins_registry/zfs.csi.openebs.io-reg.sock, err: context deadline exceeded
Jul 04 11:42:07 truenas k3s[12736]: E0704 11:42:07.370284   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:42:09 truenas k3s[12736]: E0704 11:42:09.377802   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:42:11 truenas k3s[12736]: E0704 11:42:11.354085   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:42:11 truenas k3s[12736]: E0704 11:42:11.379028   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:42:13 truenas k3s[12736]: E0704 11:42:13.370119   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:42:15 truenas k3s[12736]: E0704 11:42:15.382934   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:42:17 truenas k3s[12736]: E0704 11:42:17.350919   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:42:17 truenas k3s[12736]: E0704 11:42:17.367840   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:42:19 truenas k3s[12736]: E0704 11:42:19.377514   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:42:21 truenas k3s[12736]: E0704 11:42:21.369591   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:42:22 truenas k3s[12736]: E0704 11:42:22.352075   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:42:23 truenas k3s[12736]: E0704 11:42:23.378410   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:42:25 truenas k3s[12736]: E0704 11:42:25.368804   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:42:27 truenas k3s[12736]: E0704 11:42:27.380079   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:42:29 truenas k3s[12736]: E0704 11:42:29.350738   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:42:29 truenas k3s[12736]: E0704 11:42:29.368346   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:42:31 truenas k3s[12736]: E0704 11:42:31.378486   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:42:33 truenas k3s[12736]: E0704 11:42:33.369698   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:42:34 truenas k3s[12736]: E0704 11:42:34.352941   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:42:35 truenas k3s[12736]: E0704 11:42:35.384067   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:42:36 truenas k3s[12736]: {"level":"warn","ts":"2022-07-04T11:42:36.468+0800","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ed6c40/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Jul 04 11:42:37 truenas k3s[12736]: E0704 11:42:37.377900   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:42:39 truenas k3s[12736]: E0704 11:42:39.371130   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:42:41 truenas k3s[12736]: E0704 11:42:41.378614   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:42:43 truenas k3s[12736]: E0704 11:42:43.351247   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:42:43 truenas k3s[12736]: E0704 11:42:43.369447   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:42:45 truenas k3s[12736]: E0704 11:42:45.353128   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:42:45 truenas k3s[12736]: E0704 11:42:45.377164   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:42:47 truenas k3s[12736]: E0704 11:42:47.377683   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:42:49 truenas k3s[12736]: E0704 11:42:49.377792   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:42:51 truenas k3s[12736]: E0704 11:42:51.369473   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:42:53 truenas k3s[12736]: E0704 11:42:53.378908   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:42:55 truenas k3s[12736]: E0704 11:42:55.383569   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:42:56 truenas k3s[12736]: E0704 11:42:56.354078   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:42:57 truenas k3s[12736]: E0704 11:42:57.350865   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:42:57 truenas k3s[12736]: E0704 11:42:57.377758   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:42:59 truenas k3s[12736]: E0704 11:42:59.379674   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:43:01 truenas k3s[12736]: E0704 11:43:01.369266   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:43:03 truenas k3s[12736]: E0704 11:43:03.379635   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:43:05 truenas k3s[12736]: E0704 11:43:05.371291   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:43:07 truenas k3s[12736]: E0704 11:43:07.377702   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:43:09 truenas k3s[12736]: E0704 11:43:09.352657   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:43:09 truenas k3s[12736]: E0704 11:43:09.369189   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:43:11 truenas k3s[12736]: E0704 11:43:11.351555   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:43:11 truenas k3s[12736]: E0704 11:43:11.378000   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:43:13 truenas k3s[12736]: E0704 11:43:13.369473   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:43:15 truenas k3s[12736]: E0704 11:43:15.383763   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:43:17 truenas k3s[12736]: E0704 11:43:17.369232   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:43:19 truenas k3s[12736]: E0704 11:43:19.379478   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:43:21 truenas k3s[12736]: {"level":"warn","ts":"2022-07-04T11:43:21.317+0800","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ed6c40/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Jul 04 11:43:21 truenas k3s[12736]: E0704 11:43:21.368983   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:43:23 truenas k3s[12736]: E0704 11:43:23.351773   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:43:23 truenas k3s[12736]: E0704 11:43:23.353172   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:43:23 truenas k3s[12736]: E0704 11:43:23.376062   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:43:25 truenas k3s[12736]: E0704 11:43:25.377106   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:43:27 truenas k3s[12736]: E0704 11:43:27.380008   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:43:29 truenas k3s[12736]: E0704 11:43:29.370903   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:43:31 truenas k3s[12736]: E0704 11:43:31.378109   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:43:33 truenas k3s[12736]: E0704 11:43:33.369310   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:43:34 truenas k3s[12736]: E0704 11:43:34.352599   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:43:35 truenas k3s[12736]: E0704 11:43:35.351298   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:43:35 truenas k3s[12736]: E0704 11:43:35.379535   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:43:37 truenas k3s[12736]: E0704 11:43:37.370517   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:43:39 truenas k3s[12736]: E0704 11:43:39.379741   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:43:41 truenas k3s[12736]: E0704 11:43:41.379134   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:43:43 truenas k3s[12736]: E0704 11:43:43.370273   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:43:45 truenas k3s[12736]: E0704 11:43:45.353216   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:43:45 truenas k3s[12736]: E0704 11:43:45.379487   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:43:46 truenas k3s[12736]: E0704 11:43:46.350995   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:43:47 truenas k3s[12736]: E0704 11:43:47.370611   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:43:49 truenas k3s[12736]: E0704 11:43:49.377528   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:43:51 truenas k3s[12736]: E0704 11:43:51.378042   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:43:53 truenas k3s[12736]: E0704 11:43:53.378355   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:43:55 truenas k3s[12736]: E0704 11:43:55.368929   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:43:57 truenas k3s[12736]: E0704 11:43:57.353754   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:43:57 truenas k3s[12736]: E0704 11:43:57.378667   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:43:58 truenas k3s[12736]: {"level":"warn","ts":"2022-07-04T11:43:58.531+0800","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ed6c40/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Jul 04 11:43:59 truenas k3s[12736]: E0704 11:43:59.351053   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:43:59 truenas k3s[12736]: E0704 11:43:59.369565   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:44:01 truenas k3s[12736]: E0704 11:44:01.378377   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:44:03 truenas k3s[12736]: E0704 11:44:03.377956   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:44:05 truenas k3s[12736]: E0704 11:44:05.370493   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:44:07 truenas k3s[12736]: E0704 11:44:07.378174   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:44:09 truenas k3s[12736]: E0704 11:44:09.379006   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:44:10 truenas k3s[12736]: E0704 11:44:10.352501   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:44:11 truenas k3s[12736]: E0704 11:44:11.350818   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:44:11 truenas k3s[12736]: E0704 11:44:11.369588   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:44:13 truenas k3s[12736]: E0704 11:44:13.377358   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:44:15 truenas k3s[12736]: E0704 11:44:15.377381   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:44:17 truenas k3s[12736]: E0704 11:44:17.378221   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:44:18 truenas k3s[12736]: E0704 11:44:18.679518   12736 goroutinemap.go:150] Operation for "/var/lib/kubelet/plugins_registry/zfs.csi.openebs.io-reg.sock" failed. No retries permitted until 2022-07-04 11:46:20.679481381 +0800 CST m=+33900.078629700 (durationBeforeRetry 2m2s). Error: RegisterPlugin error -- dial failed at socket /var/lib/kubelet/plugins_registry/zfs.csi.openebs.io-reg.sock, err: failed to dial socket /var/lib/kubelet/plugins_registry/zfs.csi.openebs.io-reg.sock, err: context deadline exceeded
Jul 04 11:44:19 truenas k3s[12736]: E0704 11:44:19.367322   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:44:21 truenas k3s[12736]: E0704 11:44:21.377232   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:44:23 truenas k3s[12736]: E0704 11:44:23.353272   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:44:23 truenas k3s[12736]: E0704 11:44:23.376111   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:44:25 truenas k3s[12736]: E0704 11:44:25.369025   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:44:26 truenas k3s[12736]: E0704 11:44:26.350838   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:44:27 truenas k3s[12736]: E0704 11:44:27.377870   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:44:29 truenas k3s[12736]: E0704 11:44:29.378130   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:44:31 truenas k3s[12736]: E0704 11:44:31.371996   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:44:33 truenas k3s[12736]: E0704 11:44:33.378322   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:44:34 truenas k3s[12736]: E0704 11:44:34.352885   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:44:35 truenas k3s[12736]: E0704 11:44:35.369658   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:44:37 truenas k3s[12736]: E0704 11:44:37.377356   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:44:38 truenas k3s[12736]: E0704 11:44:38.350867   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:44:39 truenas k3s[12736]: {"level":"warn","ts":"2022-07-04T11:44:39.374+0800","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ed6c40/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Jul 04 11:44:39 truenas k3s[12736]: E0704 11:44:39.378684   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:44:41 truenas k3s[12736]: E0704 11:44:41.378759   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:44:43 truenas k3s[12736]: E0704 11:44:43.369632   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:44:45 truenas k3s[12736]: E0704 11:44:45.376630   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:44:47 truenas k3s[12736]: E0704 11:44:47.377216   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:44:48 truenas k3s[12736]: E0704 11:44:48.354499   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:44:49 truenas k3s[12736]: E0704 11:44:49.368970   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:44:50 truenas k3s[12736]: E0704 11:44:50.351392   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:44:51 truenas k3s[12736]: E0704 11:44:51.377235   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:44:53 truenas k3s[12736]: E0704 11:44:53.378500   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:44:55 truenas k3s[12736]: E0704 11:44:55.369472   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:44:57 truenas k3s[12736]: E0704 11:44:57.379291   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:44:59 truenas k3s[12736]: E0704 11:44:59.377744   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:45:01 truenas k3s[12736]: E0704 11:45:01.375143   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:45:02 truenas k3s[12736]: E0704 11:45:02.352495   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:45:03 truenas k3s[12736]: E0704 11:45:03.368789   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:45:05 truenas k3s[12736]: E0704 11:45:05.351770   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:45:05 truenas k3s[12736]: E0704 11:45:05.379900   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:45:07 truenas k3s[12736]: E0704 11:45:07.369956   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:45:09 truenas k3s[12736]: E0704 11:45:09.379799   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:45:11 truenas k3s[12736]: E0704 11:45:11.370924   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:45:13 truenas k3s[12736]: E0704 11:45:13.379023   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:45:15 truenas k3s[12736]: E0704 11:45:15.369818   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:45:16 truenas k3s[12736]: E0704 11:45:16.353699   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:45:17 truenas k3s[12736]: E0704 11:45:17.382779   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:45:19 truenas k3s[12736]: E0704 11:45:19.377413   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:45:21 truenas k3s[12736]: E0704 11:45:21.374415   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:45:23 truenas k3s[12736]: E0704 11:45:23.379752   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:45:24 truenas k3s[12736]: {"level":"warn","ts":"2022-07-04T11:45:24.350+0800","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ed6c40/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Jul 04 11:45:25 truenas k3s[12736]: E0704 11:45:25.369324   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:45:27 truenas k3s[12736]: E0704 11:45:27.352645   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:45:27 truenas k3s[12736]: E0704 11:45:27.378916   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:45:29 truenas k3s[12736]: E0704 11:45:29.369562   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:45:31 truenas k3s[12736]: E0704 11:45:31.377295   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:45:33 truenas k3s[12736]: E0704 11:45:33.370366   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:45:35 truenas k3s[12736]: E0704 11:45:35.378191   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:45:35 truenas k3s[12736]: E0704 11:45:35.434895   12736 remote_image.go:216] "PullImage from image service failed" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0"
Jul 04 11:45:35 truenas k3s[12736]: E0704 11:45:35.434966   12736 kuberuntime_image.go:51] "Failed to pull image" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0"
Jul 04 11:45:35 truenas k3s[12736]: E0704 11:45:35.435207   12736 kuberuntime_manager.go:919] container &Container{Name:csi-node-driver-registrar,Image:k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0,Command:[],Args:[--v=5 --csi-address=$(ADDRESS) --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:ADDRESS,Value:/plugin/csi.sock,ValueFrom:nil,},EnvVar{Name:DRIVER_REG_SOCK_PATH,Value:/var/lib/kubelet/plugins/zfs-localpv/csi.sock,ValueFrom:nil,},EnvVar{Name:KUBE_NODE_NAME,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:spec.nodeName,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},EnvVar{Name:NODE_DRIVER,Value:openebs-zfs,ValueFrom:nil,},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:plugin-dir,ReadOnly:false,MountPath:/plugin,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:registration-dir,ReadOnly:false,MountPath:/registration,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:kube-api-access-89mwk,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:&Lifecycle{PostStart:nil,PreStop:&LifecycleHandler{Exec:&ExecAction{Command:[/bin/sh -c rm -rf /registration/zfs-localpv /registration/zfs-localpv-reg.sock],},HTTPGet:nil,TCPSocket:nil,},},TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod openebs-zfs-node-5zm8n_kube-system(6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a): ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get "https://k8s.gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Jul 04 11:45:35 truenas k3s[12736]: E0704 11:45:35.435315   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ErrImagePull: \"rpc error: code = Unknown desc = Error response from daemon: Get \\\"https://k8s.gcr.io/v2/\\\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:45:37 truenas k3s[12736]: E0704 11:45:37.384091   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:45:39 truenas k3s[12736]: E0704 11:45:39.370416   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:45:40 truenas k3s[12736]: E0704 11:45:40.353036   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:45:41 truenas k3s[12736]: E0704 11:45:41.377353   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:45:43 truenas k3s[12736]: E0704 11:45:43.378141   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:45:45 truenas k3s[12736]: E0704 11:45:45.379002   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:45:46 truenas k3s[12736]: E0704 11:45:46.351069   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:45:47 truenas k3s[12736]: E0704 11:45:47.369504   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:45:49 truenas k3s[12736]: E0704 11:45:49.378193   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:45:51 truenas k3s[12736]: E0704 11:45:51.379788   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:45:53 truenas k3s[12736]: E0704 11:45:53.370119   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:45:54 truenas k3s[12736]: E0704 11:45:54.353188   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:45:55 truenas k3s[12736]: E0704 11:45:55.375562   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:45:57 truenas k3s[12736]: E0704 11:45:57.370693   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:45:58 truenas k3s[12736]: {"level":"warn","ts":"2022-07-04T11:45:58.383+0800","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ed6c40/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Jul 04 11:45:59 truenas k3s[12736]: E0704 11:45:59.379394   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:46:01 truenas k3s[12736]: E0704 11:46:01.350894   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:46:01 truenas k3s[12736]: E0704 11:46:01.369409   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:46:03 truenas k3s[12736]: E0704 11:46:03.379739   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:46:05 truenas k3s[12736]: E0704 11:46:05.368687   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:46:07 truenas k3s[12736]: E0704 11:46:07.381463   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:46:09 truenas k3s[12736]: E0704 11:46:09.370934   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:46:11 truenas k3s[12736]: E0704 11:46:11.377775   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:46:12 truenas k3s[12736]: E0704 11:46:12.350671   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:46:13 truenas k3s[12736]: E0704 11:46:13.377853   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:46:15 truenas k3s[12736]: E0704 11:46:15.369574   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:46:17 truenas k3s[12736]: E0704 11:46:17.380349   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:46:19 truenas k3s[12736]: E0704 11:46:19.369437   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:46:20 truenas k3s[12736]: E0704 11:46:20.580732   12736 remote_image.go:216] "PullImage from image service failed" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-resizer:v1.2.0"
Jul 04 11:46:20 truenas k3s[12736]: E0704 11:46:20.580815   12736 kuberuntime_image.go:51] "Failed to pull image" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-resizer:v1.2.0"
Jul 04 11:46:20 truenas k3s[12736]: E0704 11:46:20.580971   12736 kuberuntime_manager.go:919] container &Container{Name:csi-resizer,Image:k8s.gcr.io/sig-storage/csi-resizer:v1.2.0,Command:[],Args:[--v=5 --csi-address=$(ADDRESS) --leader-election],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:ADDRESS,Value:/var/lib/csi/sockets/pluginproxy/csi.sock,ValueFrom:nil,},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:socket-dir,ReadOnly:false,MountPath:/var/lib/csi/sockets/pluginproxy/,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:kube-api-access-8tngl,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod openebs-zfs-controller-0_kube-system(a277d60e-69ba-4e90-9eb9-197ba5ebd72d): ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get "https://k8s.gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Jul 04 11:46:21 truenas k3s[12736]: E0704 11:46:21.380326   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:46:23 truenas k3s[12736]: E0704 11:46:23.370267   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:46:25 truenas k3s[12736]: E0704 11:46:25.378041   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:46:27 truenas k3s[12736]: E0704 11:46:27.378508   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:46:28 truenas k3s[12736]: E0704 11:46:28.351749   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:46:29 truenas k3s[12736]: E0704 11:46:29.370257   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:46:30 truenas k3s[12736]: E0704 11:46:30.756671   12736 goroutinemap.go:150] Operation for "/var/lib/kubelet/plugins_registry/zfs.csi.openebs.io-reg.sock" failed. No retries permitted until 2022-07-04 11:48:32.756634008 +0800 CST m=+34032.155782317 (durationBeforeRetry 2m2s). Error: RegisterPlugin error -- dial failed at socket /var/lib/kubelet/plugins_registry/zfs.csi.openebs.io-reg.sock, err: failed to dial socket /var/lib/kubelet/plugins_registry/zfs.csi.openebs.io-reg.sock, err: context deadline exceeded
Jul 04 11:46:31 truenas k3s[12736]: E0704 11:46:31.378346   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:46:31 truenas k3s[12736]: {"level":"warn","ts":"2022-07-04T11:46:31.769+0800","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ed6c40/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Jul 04 11:46:33 truenas k3s[12736]: E0704 11:46:33.377544   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:46:35 truenas k3s[12736]: E0704 11:46:35.373824   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:46:35 truenas k3s[12736]: E0704 11:46:35.754800   12736 remote_image.go:216] "PullImage from image service failed" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0"
Jul 04 11:46:35 truenas k3s[12736]: E0704 11:46:35.754872   12736 kuberuntime_image.go:51] "Failed to pull image" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0"
Jul 04 11:46:35 truenas k3s[12736]: E0704 11:46:35.755052   12736 kuberuntime_manager.go:919] container &Container{Name:csi-provisioner,Image:k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0,Command:[],Args:[--csi-address=$(ADDRESS) --v=5 --feature-gates=Topology=true --strict-topology --leader-election --extra-create-metadata=true --enable-capacity=true --default-fstype=ext4],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:ADDRESS,Value:/var/lib/csi/sockets/pluginproxy/csi.sock,ValueFrom:nil,},EnvVar{Name:NAMESPACE,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},EnvVar{Name:POD_NAME,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.name,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:socket-dir,ReadOnly:false,MountPath:/var/lib/csi/sockets/pluginproxy/,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:kube-api-access-8tngl,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod openebs-zfs-controller-0_kube-system(a277d60e-69ba-4e90-9eb9-197ba5ebd72d): ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get "https://k8s.gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Jul 04 11:46:35 truenas k3s[12736]: E0704 11:46:35.755163   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ErrImagePull: \"rpc error: code = Unknown desc = Error response from daemon: Get \\\"https://k8s.gcr.io/v2/\\\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\", failed to \"StartContainer\" for \"csi-provisioner\" with ErrImagePull: \"rpc error: code = Unknown desc = Error response from daemon: Get \\\"https://k8s.gcr.io/v2/\\\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:46:37 truenas k3s[12736]: E0704 11:46:37.377268   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:46:39 truenas k3s[12736]: E0704 11:46:39.378020   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:46:41 truenas k3s[12736]: E0704 11:46:41.351368   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:46:41 truenas k3s[12736]: E0704 11:46:41.379247   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:46:43 truenas k3s[12736]: E0704 11:46:43.369183   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:46:45 truenas k3s[12736]: E0704 11:46:45.381800   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:46:47 truenas k3s[12736]: E0704 11:46:47.365746   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:46:48 truenas k3s[12736]: E0704 11:46:48.352203   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:46:49 truenas k3s[12736]: E0704 11:46:49.378009   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:46:51 truenas k3s[12736]: E0704 11:46:51.379563   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:46:52 truenas k3s[12736]: E0704 11:46:52.350714   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:46:53 truenas k3s[12736]: E0704 11:46:53.369421   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:46:55 truenas k3s[12736]: E0704 11:46:55.384865   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:46:57 truenas k3s[12736]: E0704 11:46:57.377912   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:46:59 truenas k3s[12736]: E0704 11:46:59.353192   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:46:59 truenas k3s[12736]: E0704 11:46:59.368030   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:47:01 truenas k3s[12736]: E0704 11:47:01.379220   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:47:03 truenas k3s[12736]: E0704 11:47:03.370192   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:47:05 truenas k3s[12736]: E0704 11:47:05.378185   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:47:06 truenas k3s[12736]: E0704 11:47:06.351121   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:47:07 truenas k3s[12736]: E0704 11:47:07.369980   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:47:09 truenas k3s[12736]: E0704 11:47:09.378752   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:47:09 truenas k3s[12736]: {"level":"warn","ts":"2022-07-04T11:47:09.688+0800","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ed6c40/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Jul 04 11:47:11 truenas k3s[12736]: E0704 11:47:11.378446   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:47:12 truenas k3s[12736]: E0704 11:47:12.352254   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:47:13 truenas k3s[12736]: E0704 11:47:13.379768   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:47:15 truenas k3s[12736]: E0704 11:47:15.383092   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:47:17 truenas k3s[12736]: E0704 11:47:17.351164   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:47:17 truenas k3s[12736]: E0704 11:47:17.369528   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:47:19 truenas k3s[12736]: E0704 11:47:19.376925   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:47:21 truenas k3s[12736]: E0704 11:47:21.377957   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:47:23 truenas k3s[12736]: E0704 11:47:23.364665   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:47:24 truenas k3s[12736]: E0704 11:47:24.352580   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:47:25 truenas k3s[12736]: E0704 11:47:25.377133   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:47:27 truenas k3s[12736]: E0704 11:47:27.377823   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:47:29 truenas k3s[12736]: E0704 11:47:29.369950   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:47:31 truenas k3s[12736]: E0704 11:47:31.351454   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:47:31 truenas k3s[12736]: E0704 11:47:31.377992   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:47:33 truenas k3s[12736]: E0704 11:47:33.369486   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:47:35 truenas k3s[12736]: E0704 11:47:35.379720   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:47:36 truenas k3s[12736]: E0704 11:47:36.353276   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:47:37 truenas k3s[12736]: E0704 11:47:37.378223   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:47:39 truenas k3s[12736]: E0704 11:47:39.370349   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:47:39 truenas k3s[12736]: {"level":"warn","ts":"2022-07-04T11:47:39.861+0800","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ed6c40/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Jul 04 11:47:41 truenas k3s[12736]: E0704 11:47:41.378111   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:47:43 truenas k3s[12736]: E0704 11:47:43.369256   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:47:45 truenas k3s[12736]: E0704 11:47:45.376981   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:47:46 truenas k3s[12736]: E0704 11:47:46.350862   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:47:47 truenas k3s[12736]: E0704 11:47:47.353669   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:47:47 truenas k3s[12736]: E0704 11:47:47.378177   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:47:49 truenas k3s[12736]: E0704 11:47:49.369440   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:47:51 truenas k3s[12736]: E0704 11:47:51.377853   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:47:53 truenas k3s[12736]: E0704 11:47:53.363153   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:47:55 truenas k3s[12736]: E0704 11:47:55.383233   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:47:57 truenas k3s[12736]: E0704 11:47:57.351239   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:47:57 truenas k3s[12736]: E0704 11:47:57.377452   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:47:58 truenas k3s[12736]: E0704 11:47:58.352481   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:47:59 truenas k3s[12736]: E0704 11:47:59.369317   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:48:01 truenas k3s[12736]: E0704 11:48:01.378227   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:48:03 truenas k3s[12736]: E0704 11:48:03.370993   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:48:05 truenas k3s[12736]: E0704 11:48:05.379813   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:48:07 truenas k3s[12736]: E0704 11:48:07.377325   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:48:09 truenas k3s[12736]: E0704 11:48:09.376877   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:48:09 truenas k3s[12736]: {"level":"warn","ts":"2022-07-04T11:48:09.869+0800","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ed6c40/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Jul 04 11:48:11 truenas k3s[12736]: E0704 11:48:11.379343   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:48:12 truenas k3s[12736]: E0704 11:48:12.350789   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:48:12 truenas k3s[12736]: E0704 11:48:12.352483   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:48:13 truenas k3s[12736]: E0704 11:48:13.370020   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:48:15 truenas k3s[12736]: E0704 11:48:15.380091   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:48:17 truenas k3s[12736]: E0704 11:48:17.388148   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:48:19 truenas k3s[12736]: E0704 11:48:19.371498   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:48:21 truenas k3s[12736]: E0704 11:48:21.380241   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:48:23 truenas k3s[12736]: E0704 11:48:23.369975   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:48:24 truenas k3s[12736]: E0704 11:48:24.350957   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:48:24 truenas k3s[12736]: E0704 11:48:24.352463   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:48:25 truenas k3s[12736]: E0704 11:48:25.380213   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:48:27 truenas k3s[12736]: E0704 11:48:27.378762   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:48:29 truenas k3s[12736]: E0704 11:48:29.371260   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:48:31 truenas k3s[12736]: E0704 11:48:31.377413   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:48:33 truenas k3s[12736]: E0704 11:48:33.368707   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:48:35 truenas k3s[12736]: E0704 11:48:35.383517   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:48:37 truenas k3s[12736]: E0704 11:48:37.377721   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:48:38 truenas k3s[12736]: E0704 11:48:38.351351   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:48:38 truenas k3s[12736]: E0704 11:48:38.352966   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:48:39 truenas k3s[12736]: E0704 11:48:39.378545   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:48:41 truenas k3s[12736]: E0704 11:48:41.379591   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:48:42 truenas k3s[12736]: E0704 11:48:42.832191   12736 goroutinemap.go:150] Operation for "/var/lib/kubelet/plugins_registry/zfs.csi.openebs.io-reg.sock" failed. No retries permitted until 2022-07-04 11:50:44.832151221 +0800 CST m=+34164.231299540 (durationBeforeRetry 2m2s). Error: RegisterPlugin error -- dial failed at socket /var/lib/kubelet/plugins_registry/zfs.csi.openebs.io-reg.sock, err: failed to dial socket /var/lib/kubelet/plugins_registry/zfs.csi.openebs.io-reg.sock, err: context deadline exceeded
Jul 04 11:48:43 truenas k3s[12736]: E0704 11:48:43.379328   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:48:45 truenas k3s[12736]: E0704 11:48:45.367810   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:48:47 truenas k3s[12736]: E0704 11:48:47.372751   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:48:47 truenas k3s[12736]: {"level":"warn","ts":"2022-07-04T11:48:47.699+0800","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ed6c40/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Jul 04 11:48:49 truenas k3s[12736]: E0704 11:48:49.352785   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:48:49 truenas k3s[12736]: E0704 11:48:49.377243   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:48:51 truenas k3s[12736]: E0704 11:48:51.378652   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:48:53 truenas k3s[12736]: E0704 11:48:53.352145   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:48:53 truenas k3s[12736]: E0704 11:48:53.377524   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:48:55 truenas k3s[12736]: E0704 11:48:55.369189   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:48:57 truenas k3s[12736]: E0704 11:48:57.375804   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:48:59 truenas k3s[12736]: E0704 11:48:59.378899   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:49:00 truenas k3s[12736]: E0704 11:49:00.353228   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:49:01 truenas k3s[12736]: E0704 11:49:01.369422   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:49:03 truenas k3s[12736]: E0704 11:49:03.381935   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:49:05 truenas k3s[12736]: E0704 11:49:05.379743   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:49:06 truenas k3s[12736]: E0704 11:49:06.350990   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:49:07 truenas k3s[12736]: E0704 11:49:07.367507   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:49:09 truenas k3s[12736]: E0704 11:49:09.377937   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:49:11 truenas k3s[12736]: E0704 11:49:11.377691   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:49:13 truenas k3s[12736]: E0704 11:49:13.383054   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:49:14 truenas k3s[12736]: E0704 11:49:14.353242   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:49:15 truenas k3s[12736]: E0704 11:49:15.369268   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:49:17 truenas k3s[12736]: E0704 11:49:17.378427   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:49:18 truenas k3s[12736]: E0704 11:49:18.350926   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:49:19 truenas k3s[12736]: E0704 11:49:19.377742   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:49:19 truenas k3s[12736]: {"level":"warn","ts":"2022-07-04T11:49:19.460+0800","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ed6c40/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Jul 04 11:49:21 truenas k3s[12736]: E0704 11:49:21.369092   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:49:23 truenas k3s[12736]: E0704 11:49:23.378058   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:49:25 truenas k3s[12736]: E0704 11:49:25.377944   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:49:27 truenas k3s[12736]: E0704 11:49:27.379110   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:49:28 truenas k3s[12736]: E0704 11:49:28.352910   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:49:29 truenas k3s[12736]: E0704 11:49:29.369926   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:49:31 truenas k3s[12736]: E0704 11:49:31.377963   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:49:33 truenas k3s[12736]: E0704 11:49:33.351375   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:49:33 truenas k3s[12736]: E0704 11:49:33.381378   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:49:35 truenas k3s[12736]: E0704 11:49:35.377925   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:49:37 truenas k3s[12736]: E0704 11:49:37.368650   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:49:39 truenas k3s[12736]: E0704 11:49:39.377229   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:49:40 truenas k3s[12736]: E0704 11:49:40.352407   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:49:41 truenas k3s[12736]: E0704 11:49:41.378469   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:49:43 truenas k3s[12736]: E0704 11:49:43.379392   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:49:44 truenas k3s[12736]: E0704 11:49:44.350799   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:49:45 truenas k3s[12736]: E0704 11:49:45.369526   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:49:47 truenas k3s[12736]: E0704 11:49:47.376742   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:49:49 truenas k3s[12736]: E0704 11:49:49.377935   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:49:51 truenas k3s[12736]: E0704 11:49:51.370789   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:49:53 truenas k3s[12736]: E0704 11:49:53.383619   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:49:55 truenas k3s[12736]: E0704 11:49:55.351607   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:49:55 truenas k3s[12736]: E0704 11:49:55.353045   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:49:55 truenas k3s[12736]: E0704 11:49:55.379440   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:49:56 truenas k3s[12736]: {"level":"warn","ts":"2022-07-04T11:49:56.174+0800","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ed6c40/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Jul 04 11:49:57 truenas k3s[12736]: E0704 11:49:57.370363   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:49:59 truenas k3s[12736]: E0704 11:49:59.376846   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:50:01 truenas k3s[12736]: E0704 11:50:01.378713   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:50:03 truenas k3s[12736]: E0704 11:50:03.370281   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:50:05 truenas k3s[12736]: E0704 11:50:05.378430   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:50:07 truenas k3s[12736]: E0704 11:50:07.380521   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:50:08 truenas k3s[12736]: E0704 11:50:08.352785   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:50:09 truenas k3s[12736]: E0704 11:50:09.378442   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:50:10 truenas k3s[12736]: E0704 11:50:10.350493   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:50:11 truenas k3s[12736]: E0704 11:50:11.366820   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:50:13 truenas k3s[12736]: E0704 11:50:13.377964   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:50:15 truenas k3s[12736]: E0704 11:50:15.377564   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:50:17 truenas k3s[12736]: E0704 11:50:17.369293   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:50:19 truenas k3s[12736]: E0704 11:50:19.379194   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:50:21 truenas k3s[12736]: E0704 11:50:21.370302   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:50:22 truenas k3s[12736]: E0704 11:50:22.352724   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:50:23 truenas k3s[12736]: E0704 11:50:23.403855   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:50:25 truenas k3s[12736]: E0704 11:50:25.352117   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:50:25 truenas k3s[12736]: E0704 11:50:25.377828   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:50:27 truenas k3s[12736]: E0704 11:50:27.370170   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:50:29 truenas k3s[12736]: E0704 11:50:29.378764   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:50:31 truenas k3s[12736]: E0704 11:50:31.369275   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:50:33 truenas k3s[12736]: E0704 11:50:33.382142   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:50:35 truenas k3s[12736]: E0704 11:50:35.378206   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:50:37 truenas k3s[12736]: E0704 11:50:37.353039   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:50:37 truenas k3s[12736]: E0704 11:50:37.370932   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:50:39 truenas k3s[12736]: E0704 11:50:39.378696   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:50:40 truenas k3s[12736]: {"level":"warn","ts":"2022-07-04T11:50:40.794+0800","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ed6c40/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Jul 04 11:50:41 truenas k3s[12736]: E0704 11:50:41.369900   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:50:43 truenas k3s[12736]: E0704 11:50:43.377646   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:50:45 truenas k3s[12736]: E0704 11:50:45.378327   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:50:47 truenas k3s[12736]: E0704 11:50:47.379243   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:50:48 truenas k3s[12736]: E0704 11:50:48.352824   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:50:49 truenas k3s[12736]: E0704 11:50:49.370017   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:50:51 truenas k3s[12736]: E0704 11:50:51.377728   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:50:53 truenas k3s[12736]: E0704 11:50:53.378366   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:50:53 truenas k3s[12736]: E0704 11:50:53.505634   12736 remote_image.go:216] "PullImage from image service failed" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0"
Jul 04 11:50:53 truenas k3s[12736]: E0704 11:50:53.505712   12736 kuberuntime_image.go:51] "Failed to pull image" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0"
Jul 04 11:50:53 truenas k3s[12736]: E0704 11:50:53.505916   12736 kuberuntime_manager.go:919] container &Container{Name:csi-node-driver-registrar,Image:k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0,Command:[],Args:[--v=5 --csi-address=$(ADDRESS) --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:ADDRESS,Value:/plugin/csi.sock,ValueFrom:nil,},EnvVar{Name:DRIVER_REG_SOCK_PATH,Value:/var/lib/kubelet/plugins/zfs-localpv/csi.sock,ValueFrom:nil,},EnvVar{Name:KUBE_NODE_NAME,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:spec.nodeName,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},EnvVar{Name:NODE_DRIVER,Value:openebs-zfs,ValueFrom:nil,},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:plugin-dir,ReadOnly:false,MountPath:/plugin,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:registration-dir,ReadOnly:false,MountPath:/registration,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:kube-api-access-89mwk,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:&Lifecycle{PostStart:nil,PreStop:&LifecycleHandler{Exec:&ExecAction{Command:[/bin/sh -c rm -rf /registration/zfs-localpv /registration/zfs-localpv-reg.sock],},HTTPGet:nil,TCPSocket:nil,},},TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod openebs-zfs-node-5zm8n_kube-system(6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a): ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get "https://k8s.gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Jul 04 11:50:53 truenas k3s[12736]: E0704 11:50:53.505994   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ErrImagePull: \"rpc error: code = Unknown desc = Error response from daemon: Get \\\"https://k8s.gcr.io/v2/\\\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:50:54 truenas k3s[12736]: E0704 11:50:54.911216   12736 goroutinemap.go:150] Operation for "/var/lib/kubelet/plugins_registry/zfs.csi.openebs.io-reg.sock" failed. No retries permitted until 2022-07-04 11:52:56.911178592 +0800 CST m=+34296.310326901 (durationBeforeRetry 2m2s). Error: RegisterPlugin error -- dial failed at socket /var/lib/kubelet/plugins_registry/zfs.csi.openebs.io-reg.sock, err: failed to dial socket /var/lib/kubelet/plugins_registry/zfs.csi.openebs.io-reg.sock, err: context deadline exceeded
Jul 04 11:50:55 truenas k3s[12736]: E0704 11:50:55.371081   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:50:57 truenas k3s[12736]: E0704 11:50:57.377664   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:50:59 truenas k3s[12736]: E0704 11:50:59.352610   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:50:59 truenas k3s[12736]: E0704 11:50:59.367723   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:51:01 truenas k3s[12736]: E0704 11:51:01.378300   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:51:03 truenas k3s[12736]: E0704 11:51:03.378049   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:51:05 truenas k3s[12736]: E0704 11:51:05.371506   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:51:07 truenas k3s[12736]: E0704 11:51:07.377553   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:51:08 truenas k3s[12736]: E0704 11:51:08.350867   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:51:09 truenas k3s[12736]: E0704 11:51:09.370014   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:51:11 truenas k3s[12736]: E0704 11:51:11.378442   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:51:13 truenas k3s[12736]: E0704 11:51:13.352824   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:51:13 truenas k3s[12736]: E0704 11:51:13.369195   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:51:13 truenas k3s[12736]: {"level":"warn","ts":"2022-07-04T11:51:13.742+0800","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ed6c40/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Jul 04 11:51:15 truenas k3s[12736]: E0704 11:51:15.378957   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:51:17 truenas k3s[12736]: E0704 11:51:17.363722   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:51:19 truenas k3s[12736]: E0704 11:51:19.350743   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:51:19 truenas k3s[12736]: E0704 11:51:19.376967   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:51:21 truenas k3s[12736]: E0704 11:51:21.378889   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:51:23 truenas k3s[12736]: E0704 11:51:23.370095   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:51:25 truenas k3s[12736]: E0704 11:51:25.377281   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:51:27 truenas k3s[12736]: E0704 11:51:27.377670   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:51:29 truenas k3s[12736]: E0704 11:51:29.378531   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:51:31 truenas k3s[12736]: E0704 11:51:31.369795   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:51:32 truenas k3s[12736]: E0704 11:51:32.350982   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:51:33 truenas k3s[12736]: E0704 11:51:33.380296   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:51:35 truenas k3s[12736]: E0704 11:51:35.377755   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:51:37 truenas k3s[12736]: E0704 11:51:37.370477   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:51:39 truenas k3s[12736]: E0704 11:51:39.379532   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:51:39 truenas k3s[12736]: E0704 11:51:39.645814   12736 remote_image.go:216] "PullImage from image service failed" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-resizer:v1.2.0"
Jul 04 11:51:39 truenas k3s[12736]: E0704 11:51:39.645891   12736 kuberuntime_image.go:51] "Failed to pull image" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-resizer:v1.2.0"
Jul 04 11:51:39 truenas k3s[12736]: E0704 11:51:39.646050   12736 kuberuntime_manager.go:919] container &Container{Name:csi-resizer,Image:k8s.gcr.io/sig-storage/csi-resizer:v1.2.0,Command:[],Args:[--v=5 --csi-address=$(ADDRESS) --leader-election],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:ADDRESS,Value:/var/lib/csi/sockets/pluginproxy/csi.sock,ValueFrom:nil,},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:socket-dir,ReadOnly:false,MountPath:/var/lib/csi/sockets/pluginproxy/,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:kube-api-access-8tngl,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod openebs-zfs-controller-0_kube-system(a277d60e-69ba-4e90-9eb9-197ba5ebd72d): ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get "https://k8s.gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Jul 04 11:51:41 truenas k3s[12736]: E0704 11:51:41.369242   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:51:43 truenas k3s[12736]: E0704 11:51:43.378455   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:51:45 truenas k3s[12736]: E0704 11:51:45.369853   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:51:47 truenas k3s[12736]: E0704 11:51:47.351604   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:51:47 truenas k3s[12736]: E0704 11:51:47.378778   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:51:49 truenas k3s[12736]: E0704 11:51:49.369407   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:51:49 truenas k3s[12736]: {"level":"warn","ts":"2022-07-04T11:51:49.630+0800","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ed6c40/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Jul 04 11:51:51 truenas k3s[12736]: E0704 11:51:51.378436   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:51:53 truenas k3s[12736]: E0704 11:51:53.378207   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:51:54 truenas k3s[12736]: E0704 11:51:54.997252   12736 remote_image.go:216] "PullImage from image service failed" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0"
Jul 04 11:51:54 truenas k3s[12736]: E0704 11:51:54.997321   12736 kuberuntime_image.go:51] "Failed to pull image" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0"
Jul 04 11:51:54 truenas k3s[12736]: E0704 11:51:54.997508   12736 kuberuntime_manager.go:919] container &Container{Name:csi-provisioner,Image:k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0,Command:[],Args:[--csi-address=$(ADDRESS) --v=5 --feature-gates=Topology=true --strict-topology --leader-election --extra-create-metadata=true --enable-capacity=true --default-fstype=ext4],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:ADDRESS,Value:/var/lib/csi/sockets/pluginproxy/csi.sock,ValueFrom:nil,},EnvVar{Name:NAMESPACE,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},EnvVar{Name:POD_NAME,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.name,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:socket-dir,ReadOnly:false,MountPath:/var/lib/csi/sockets/pluginproxy/,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:kube-api-access-8tngl,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod openebs-zfs-controller-0_kube-system(a277d60e-69ba-4e90-9eb9-197ba5ebd72d): ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get "https://k8s.gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Jul 04 11:51:54 truenas k3s[12736]: E0704 11:51:54.997649   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ErrImagePull: \"rpc error: code = Unknown desc = Error response from daemon: Get \\\"https://k8s.gcr.io/v2/\\\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\", failed to \"StartContainer\" for \"csi-provisioner\" with ErrImagePull: \"rpc error: code = Unknown desc = Error response from daemon: Get \\\"https://k8s.gcr.io/v2/\\\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:51:55 truenas k3s[12736]: E0704 11:51:55.379516   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:51:57 truenas k3s[12736]: E0704 11:51:57.369444   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:51:59 truenas k3s[12736]: E0704 11:51:59.350248   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:51:59 truenas k3s[12736]: E0704 11:51:59.375901   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:52:01 truenas k3s[12736]: E0704 11:52:01.378985   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:52:03 truenas k3s[12736]: E0704 11:52:03.368610   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:52:05 truenas k3s[12736]: E0704 11:52:05.378211   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:52:06 truenas k3s[12736]: E0704 11:52:06.353029   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:52:07 truenas k3s[12736]: E0704 11:52:07.369789   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:52:09 truenas k3s[12736]: E0704 11:52:09.378705   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:52:11 truenas k3s[12736]: E0704 11:52:11.373568   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:52:12 truenas k3s[12736]: E0704 11:52:12.350373   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:52:13 truenas k3s[12736]: E0704 11:52:13.377107   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:52:15 truenas k3s[12736]: E0704 11:52:15.377949   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:52:17 truenas k3s[12736]: E0704 11:52:17.371015   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:52:19 truenas k3s[12736]: E0704 11:52:19.353279   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:52:19 truenas k3s[12736]: E0704 11:52:19.379357   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:52:21 truenas k3s[12736]: E0704 11:52:21.369579   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:52:23 truenas k3s[12736]: E0704 11:52:23.378076   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:52:25 truenas k3s[12736]: E0704 11:52:25.379846   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:52:26 truenas k3s[12736]: E0704 11:52:26.350778   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:52:27 truenas k3s[12736]: E0704 11:52:27.369666   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:52:29 truenas k3s[12736]: E0704 11:52:29.377324   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:52:30 truenas k3s[12736]: {"level":"warn","ts":"2022-07-04T11:52:30.362+0800","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ed6c40/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Jul 04 11:52:31 truenas k3s[12736]: E0704 11:52:31.381996   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:52:33 truenas k3s[12736]: E0704 11:52:33.353102   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:52:33 truenas k3s[12736]: E0704 11:52:33.378607   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:52:35 truenas k3s[12736]: E0704 11:52:35.370350   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:52:37 truenas k3s[12736]: E0704 11:52:37.377523   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:52:39 truenas k3s[12736]: E0704 11:52:39.378383   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:52:41 truenas k3s[12736]: E0704 11:52:41.351376   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:52:41 truenas k3s[12736]: E0704 11:52:41.377980   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:52:43 truenas k3s[12736]: E0704 11:52:43.370218   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:52:45 truenas k3s[12736]: E0704 11:52:45.378828   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:52:47 truenas k3s[12736]: E0704 11:52:47.378607   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:52:48 truenas k3s[12736]: E0704 11:52:48.353350   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:52:49 truenas k3s[12736]: E0704 11:52:49.371510   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:52:51 truenas k3s[12736]: E0704 11:52:51.384782   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:52:53 truenas k3s[12736]: E0704 11:52:53.369425   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:52:55 truenas k3s[12736]: E0704 11:52:55.351079   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:52:55 truenas k3s[12736]: E0704 11:52:55.377268   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:52:57 truenas k3s[12736]: E0704 11:52:57.368976   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:52:59 truenas k3s[12736]: E0704 11:52:59.377265   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:53:00 truenas k3s[12736]: E0704 11:53:00.352554   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:53:01 truenas k3s[12736]: E0704 11:53:01.379038   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:53:03 truenas k3s[12736]: E0704 11:53:03.381876   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:53:05 truenas k3s[12736]: E0704 11:53:05.370237   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:53:06 truenas k3s[12736]: E0704 11:53:06.987162   12736 goroutinemap.go:150] Operation for "/var/lib/kubelet/plugins_registry/zfs.csi.openebs.io-reg.sock" failed. No retries permitted until 2022-07-04 11:55:08.987117676 +0800 CST m=+34428.386265995 (durationBeforeRetry 2m2s). Error: RegisterPlugin error -- dial failed at socket /var/lib/kubelet/plugins_registry/zfs.csi.openebs.io-reg.sock, err: failed to dial socket /var/lib/kubelet/plugins_registry/zfs.csi.openebs.io-reg.sock, err: context deadline exceeded
Jul 04 11:53:07 truenas k3s[12736]: E0704 11:53:07.377862   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:53:07 truenas k3s[12736]: {"level":"warn","ts":"2022-07-04T11:53:07.560+0800","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ed6c40/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Jul 04 11:53:09 truenas k3s[12736]: E0704 11:53:09.376954   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:53:10 truenas k3s[12736]: E0704 11:53:10.351066   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:53:11 truenas k3s[12736]: E0704 11:53:11.372588   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:53:12 truenas k3s[12736]: E0704 11:53:12.352605   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:53:13 truenas k3s[12736]: E0704 11:53:13.377237   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:53:15 truenas k3s[12736]: E0704 11:53:15.378011   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:53:17 truenas k3s[12736]: E0704 11:53:17.378473   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:53:19 truenas k3s[12736]: E0704 11:53:19.368746   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:53:21 truenas k3s[12736]: E0704 11:53:21.378523   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:53:23 truenas k3s[12736]: E0704 11:53:23.353935   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:53:23 truenas k3s[12736]: E0704 11:53:23.381082   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:53:25 truenas k3s[12736]: E0704 11:53:25.350882   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:53:25 truenas k3s[12736]: E0704 11:53:25.369502   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:53:27 truenas k3s[12736]: E0704 11:53:27.378805   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:53:29 truenas k3s[12736]: E0704 11:53:29.384496   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:53:31 truenas k3s[12736]: E0704 11:53:31.370534   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:53:33 truenas k3s[12736]: E0704 11:53:33.381911   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:53:35 truenas k3s[12736]: E0704 11:53:35.353871   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:53:35 truenas k3s[12736]: E0704 11:53:35.378931   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:53:37 truenas k3s[12736]: E0704 11:53:37.370532   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:53:38 truenas k3s[12736]: E0704 11:53:38.350741   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:53:39 truenas k3s[12736]: E0704 11:53:39.377054   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:53:40 truenas k3s[12736]: {"level":"warn","ts":"2022-07-04T11:53:40.114+0800","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ed6c40/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Jul 04 11:53:41 truenas k3s[12736]: E0704 11:53:41.379042   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:53:43 truenas k3s[12736]: E0704 11:53:43.368873   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:53:45 truenas k3s[12736]: E0704 11:53:45.378482   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:53:47 truenas k3s[12736]: E0704 11:53:47.352555   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:53:47 truenas k3s[12736]: E0704 11:53:47.369343   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:53:49 truenas k3s[12736]: E0704 11:53:49.350975   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:53:49 truenas k3s[12736]: E0704 11:53:49.377272   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:53:51 truenas k3s[12736]: E0704 11:53:51.384616   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:53:53 truenas k3s[12736]: E0704 11:53:53.370283   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:53:55 truenas k3s[12736]: E0704 11:53:55.415469   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:53:57 truenas k3s[12736]: E0704 11:53:57.379222   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:53:59 truenas k3s[12736]: E0704 11:53:59.353425   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:53:59 truenas k3s[12736]: E0704 11:53:59.380239   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:54:01 truenas k3s[12736]: E0704 11:54:01.369682   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:54:03 truenas k3s[12736]: E0704 11:54:03.377454   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:54:04 truenas k3s[12736]: E0704 11:54:04.351082   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:54:05 truenas k3s[12736]: E0704 11:54:05.369126   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:54:07 truenas k3s[12736]: E0704 11:54:07.379086   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:54:09 truenas k3s[12736]: E0704 11:54:09.380685   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:54:11 truenas k3s[12736]: E0704 11:54:11.353451   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:54:11 truenas k3s[12736]: E0704 11:54:11.375780   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:54:13 truenas k3s[12736]: E0704 11:54:13.378771   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:54:15 truenas k3s[12736]: E0704 11:54:15.370001   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:54:16 truenas k3s[12736]: E0704 11:54:16.351069   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:54:17 truenas k3s[12736]: E0704 11:54:17.377989   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:54:19 truenas k3s[12736]: {"level":"warn","ts":"2022-07-04T11:54:19.140+0800","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ed6c40/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Jul 04 11:54:19 truenas k3s[12736]: E0704 11:54:19.369385   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:54:21 truenas k3s[12736]: E0704 11:54:21.378321   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:54:23 truenas k3s[12736]: E0704 11:54:23.369756   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:54:25 truenas k3s[12736]: E0704 11:54:25.353374   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:54:25 truenas k3s[12736]: E0704 11:54:25.378183   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:54:26 truenas k3s[12736]: I0704 11:54:26.197596   12736 trace.go:205] Trace[1535102563]: "GuaranteedUpdate etcd3" type:*coordination.Lease (04-Jul-2022 11:54:25.676) (total time: 520ms):
Jul 04 11:54:26 truenas k3s[12736]: Trace[1535102563]: ---"Transaction committed" 520ms (11:54:26.197)
Jul 04 11:54:26 truenas k3s[12736]: Trace[1535102563]: [520.677563ms] [520.677563ms] END
Jul 04 11:54:26 truenas k3s[12736]: I0704 11:54:26.197750   12736 trace.go:205] Trace[1541060866]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/snapshot-controller-leader,user-agent:snapshot-controller/v0.0.0 (linux/amd64) kubernetes/$Format,audit-id:0133c348-586d-454d-b7e5-976da319046a,client:172.16.0.25,accept:application/json, */*,protocol:HTTP/2.0 (04-Jul-2022 11:54:25.676) (total time: 521ms):
Jul 04 11:54:26 truenas k3s[12736]: Trace[1541060866]: ---"Object stored in database" 520ms (11:54:26.197)
Jul 04 11:54:26 truenas k3s[12736]: Trace[1541060866]: [521.074074ms] [521.074074ms] END
Jul 04 11:54:27 truenas k3s[12736]: E0704 11:54:27.369254   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:54:29 truenas k3s[12736]: E0704 11:54:29.351809   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:54:29 truenas k3s[12736]: E0704 11:54:29.382491   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:54:31 truenas k3s[12736]: I0704 11:54:31.257174   12736 trace.go:205] Trace[1833496763]: "GuaranteedUpdate etcd3" type:*coordination.Lease (04-Jul-2022 11:54:30.724) (total time: 532ms):
Jul 04 11:54:31 truenas k3s[12736]: Trace[1833496763]: ---"Transaction committed" 531ms (11:54:31.257)
Jul 04 11:54:31 truenas k3s[12736]: Trace[1833496763]: [532.610076ms] [532.610076ms] END
Jul 04 11:54:31 truenas k3s[12736]: I0704 11:54:31.257407   12736 trace.go:205] Trace[774145002]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/external-snapshotter-leader-zfs-csi-openebs-io,user-agent:csi-snapshotter/v0.0.0 (linux/amd64) kubernetes/$Format,audit-id:6e4600ed-8056-41e4-934f-7e7a647633b9,client:172.16.0.25,accept:application/json, */*,protocol:HTTP/2.0 (04-Jul-2022 11:54:30.724) (total time: 533ms):
Jul 04 11:54:31 truenas k3s[12736]: Trace[774145002]: ---"Object stored in database" 532ms (11:54:31.257)
Jul 04 11:54:31 truenas k3s[12736]: Trace[774145002]: [533.249288ms] [533.249288ms] END
Jul 04 11:54:31 truenas k3s[12736]: E0704 11:54:31.369520   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:54:33 truenas k3s[12736]: E0704 11:54:33.384335   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:54:35 truenas k3s[12736]: E0704 11:54:35.379333   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:54:36 truenas k3s[12736]: I0704 11:54:36.897122   12736 trace.go:205] Trace[178663010]: "GuaranteedUpdate etcd3" type:*coordination.Lease (04-Jul-2022 11:54:36.264) (total time: 632ms):
Jul 04 11:54:36 truenas k3s[12736]: Trace[178663010]: ---"Transaction committed" 631ms (11:54:36.897)
Jul 04 11:54:36 truenas k3s[12736]: Trace[178663010]: [632.915987ms] [632.915987ms] END
Jul 04 11:54:36 truenas k3s[12736]: I0704 11:54:36.897392   12736 trace.go:205] Trace[1701616304]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/snapshot-controller-leader,user-agent:snapshot-controller/v0.0.0 (linux/amd64) kubernetes/$Format,audit-id:d5e29d7d-f006-4abe-bcd4-0cba52b496dd,client:172.16.0.25,accept:application/json, */*,protocol:HTTP/2.0 (04-Jul-2022 11:54:36.263) (total time: 633ms):
Jul 04 11:54:36 truenas k3s[12736]: Trace[1701616304]: ---"Object stored in database" 633ms (11:54:36.897)
Jul 04 11:54:36 truenas k3s[12736]: Trace[1701616304]: [633.592929ms] [633.592929ms] END
Jul 04 11:54:36 truenas k3s[12736]: I0704 11:54:36.898378   12736 trace.go:205] Trace[934274860]: "GuaranteedUpdate etcd3" type:*coordination.Lease (04-Jul-2022 11:54:36.264) (total time: 633ms):
Jul 04 11:54:36 truenas k3s[12736]: Trace[934274860]: ---"Transaction committed" 632ms (11:54:36.898)
Jul 04 11:54:36 truenas k3s[12736]: Trace[934274860]: [633.771229ms] [633.771229ms] END
Jul 04 11:54:36 truenas k3s[12736]: I0704 11:54:36.898610   12736 trace.go:205] Trace[1486742999]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/external-snapshotter-leader-zfs-csi-openebs-io,user-agent:csi-snapshotter/v0.0.0 (linux/amd64) kubernetes/$Format,audit-id:dcb6a66c-cb49-4cf0-9489-d53af553d012,client:172.16.0.25,accept:application/json, */*,protocol:HTTP/2.0 (04-Jul-2022 11:54:36.264) (total time: 634ms):
Jul 04 11:54:36 truenas k3s[12736]: Trace[1486742999]: ---"Object stored in database" 633ms (11:54:36.898)
Jul 04 11:54:36 truenas k3s[12736]: Trace[1486742999]: [634.33315ms] [634.33315ms] END
Jul 04 11:54:36 truenas k3s[12736]: I0704 11:54:36.971471   12736 trace.go:205] Trace[1681253568]: "GuaranteedUpdate etcd3" type:*coordination.Lease (04-Jul-2022 11:54:36.437) (total time: 534ms):
Jul 04 11:54:36 truenas k3s[12736]: Trace[1681253568]: ---"Transaction committed" 532ms (11:54:36.971)
Jul 04 11:54:36 truenas k3s[12736]: Trace[1681253568]: [534.086468ms] [534.086468ms] END
Jul 04 11:54:36 truenas k3s[12736]: I0704 11:54:36.971739   12736 trace.go:205] Trace[1594220517]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ix-truenas,user-agent:Go-http-client/2.0,audit-id:76bd9f76-3da5-4507-8ccb-ce0eb4901948,client:127.0.0.1,accept:application/vnd.kubernetes.protobuf,application/json,protocol:HTTP/2.0 (04-Jul-2022 11:54:36.437) (total time: 534ms):
Jul 04 11:54:36 truenas k3s[12736]: Trace[1594220517]: ---"Object stored in database" 534ms (11:54:36.971)
Jul 04 11:54:36 truenas k3s[12736]: Trace[1594220517]: [534.61487ms] [534.61487ms] END
Jul 04 11:54:37 truenas k3s[12736]: E0704 11:54:37.379049   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:54:39 truenas k3s[12736]: E0704 11:54:39.370026   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:54:40 truenas k3s[12736]: E0704 11:54:40.351146   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:54:40 truenas k3s[12736]: E0704 11:54:40.352787   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:54:41 truenas k3s[12736]: E0704 11:54:41.377135   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:54:43 truenas k3s[12736]: E0704 11:54:43.369760   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:54:45 truenas k3s[12736]: E0704 11:54:45.379006   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:54:47 truenas k3s[12736]: E0704 11:54:47.370071   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:54:49 truenas k3s[12736]: E0704 11:54:49.378135   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:54:51 truenas k3s[12736]: E0704 11:54:51.364077   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:54:53 truenas k3s[12736]: E0704 11:54:53.353452   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:54:53 truenas k3s[12736]: E0704 11:54:53.384278   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:54:54 truenas k3s[12736]: E0704 11:54:54.351185   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:54:55 truenas k3s[12736]: E0704 11:54:55.370048   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:54:57 truenas k3s[12736]: {"level":"warn","ts":"2022-07-04T11:54:57.132+0800","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ed6c40/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Jul 04 11:54:57 truenas k3s[12736]: E0704 11:54:57.377456   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:54:59 truenas k3s[12736]: E0704 11:54:59.377832   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:55:01 truenas k3s[12736]: E0704 11:55:01.364647   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:55:03 truenas k3s[12736]: E0704 11:55:03.381782   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:55:05 truenas k3s[12736]: E0704 11:55:05.351617   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:55:05 truenas k3s[12736]: E0704 11:55:05.369563   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:55:07 truenas k3s[12736]: E0704 11:55:07.378400   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:55:08 truenas k3s[12736]: E0704 11:55:08.352345   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:55:09 truenas k3s[12736]: E0704 11:55:09.369215   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:55:11 truenas k3s[12736]: E0704 11:55:11.378364   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:55:13 truenas k3s[12736]: E0704 11:55:13.370216   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:55:15 truenas k3s[12736]: E0704 11:55:15.397698   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:55:17 truenas k3s[12736]: E0704 11:55:17.368724   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:55:18 truenas k3s[12736]: E0704 11:55:18.351934   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:55:19 truenas k3s[12736]: E0704 11:55:19.064391   12736 goroutinemap.go:150] Operation for "/var/lib/kubelet/plugins_registry/zfs.csi.openebs.io-reg.sock" failed. No retries permitted until 2022-07-04 11:57:21.064350483 +0800 CST m=+34560.463498792 (durationBeforeRetry 2m2s). Error: RegisterPlugin error -- dial failed at socket /var/lib/kubelet/plugins_registry/zfs.csi.openebs.io-reg.sock, err: failed to dial socket /var/lib/kubelet/plugins_registry/zfs.csi.openebs.io-reg.sock, err: context deadline exceeded
Jul 04 11:55:19 truenas k3s[12736]: E0704 11:55:19.353046   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:55:19 truenas k3s[12736]: E0704 11:55:19.384401   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:55:21 truenas k3s[12736]: E0704 11:55:21.369406   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:55:23 truenas k3s[12736]: E0704 11:55:23.378327   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:55:25 truenas k3s[12736]: E0704 11:55:25.369520   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:55:27 truenas k3s[12736]: E0704 11:55:27.378852   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:55:29 truenas k3s[12736]: E0704 11:55:29.369866   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:55:31 truenas k3s[12736]: E0704 11:55:31.353640   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:55:31 truenas k3s[12736]: E0704 11:55:31.377954   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:55:32 truenas k3s[12736]: E0704 11:55:32.351072   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:55:33 truenas k3s[12736]: E0704 11:55:33.378377   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:55:35 truenas k3s[12736]: E0704 11:55:35.370231   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:55:36 truenas k3s[12736]: {"level":"warn","ts":"2022-07-04T11:55:36.772+0800","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ed6c40/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Jul 04 11:55:37 truenas k3s[12736]: E0704 11:55:37.376187   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:55:39 truenas k3s[12736]: E0704 11:55:39.382316   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:55:41 truenas k3s[12736]: E0704 11:55:41.372730   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:55:43 truenas k3s[12736]: E0704 11:55:43.378133   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:55:44 truenas k3s[12736]: E0704 11:55:44.353357   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:55:45 truenas k3s[12736]: E0704 11:55:45.377842   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:55:46 truenas k3s[12736]: E0704 11:55:46.350985   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:55:47 truenas k3s[12736]: E0704 11:55:47.378163   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:55:49 truenas k3s[12736]: E0704 11:55:49.367020   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:55:51 truenas k3s[12736]: E0704 11:55:51.378402   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:55:53 truenas k3s[12736]: E0704 11:55:53.370477   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:55:55 truenas k3s[12736]: E0704 11:55:55.377768   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:55:57 truenas k3s[12736]: E0704 11:55:57.379084   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:55:59 truenas k3s[12736]: E0704 11:55:59.353584   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:55:59 truenas k3s[12736]: E0704 11:55:59.379034   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:56:01 truenas k3s[12736]: E0704 11:56:01.368579   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:56:03 truenas k3s[12736]: E0704 11:56:03.386200   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:56:05 truenas k3s[12736]: E0704 11:56:05.370204   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:56:07 truenas k3s[12736]: E0704 11:56:07.377939   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:56:08 truenas k3s[12736]: {"level":"warn","ts":"2022-07-04T11:56:08.378+0800","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ed6c40/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Jul 04 11:56:09 truenas k3s[12736]: E0704 11:56:09.369318   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:56:10 truenas k3s[12736]: E0704 11:56:10.352889   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:56:11 truenas k3s[12736]: E0704 11:56:11.379670   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:56:13 truenas k3s[12736]: E0704 11:56:13.369481   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:56:15 truenas k3s[12736]: E0704 11:56:15.383815   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:56:15 truenas k3s[12736]: E0704 11:56:15.539191   12736 remote_image.go:216] "PullImage from image service failed" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0"
Jul 04 11:56:15 truenas k3s[12736]: E0704 11:56:15.539257   12736 kuberuntime_image.go:51] "Failed to pull image" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0"
Jul 04 11:56:15 truenas k3s[12736]: E0704 11:56:15.539429   12736 kuberuntime_manager.go:919] container &Container{Name:csi-node-driver-registrar,Image:k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0,Command:[],Args:[--v=5 --csi-address=$(ADDRESS) --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:ADDRESS,Value:/plugin/csi.sock,ValueFrom:nil,},EnvVar{Name:DRIVER_REG_SOCK_PATH,Value:/var/lib/kubelet/plugins/zfs-localpv/csi.sock,ValueFrom:nil,},EnvVar{Name:KUBE_NODE_NAME,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:spec.nodeName,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},EnvVar{Name:NODE_DRIVER,Value:openebs-zfs,ValueFrom:nil,},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:plugin-dir,ReadOnly:false,MountPath:/plugin,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:registration-dir,ReadOnly:false,MountPath:/registration,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:kube-api-access-89mwk,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:&Lifecycle{PostStart:nil,PreStop:&LifecycleHandler{Exec:&ExecAction{Command:[/bin/sh -c rm -rf /registration/zfs-localpv /registration/zfs-localpv-reg.sock],},HTTPGet:nil,TCPSocket:nil,},},TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod openebs-zfs-node-5zm8n_kube-system(6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a): ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get "https://k8s.gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Jul 04 11:56:15 truenas k3s[12736]: E0704 11:56:15.539506   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ErrImagePull: \"rpc error: code = Unknown desc = Error response from daemon: Get \\\"https://k8s.gcr.io/v2/\\\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:56:17 truenas k3s[12736]: E0704 11:56:17.379275   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:56:19 truenas k3s[12736]: E0704 11:56:19.369509   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:56:21 truenas k3s[12736]: E0704 11:56:21.378323   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:56:23 truenas k3s[12736]: E0704 11:56:23.353573   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:56:23 truenas k3s[12736]: E0704 11:56:23.379113   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:56:25 truenas k3s[12736]: E0704 11:56:25.369349   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:56:27 truenas k3s[12736]: E0704 11:56:27.351801   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:56:27 truenas k3s[12736]: E0704 11:56:27.379257   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:56:29 truenas k3s[12736]: E0704 11:56:29.369790   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:56:31 truenas k3s[12736]: E0704 11:56:31.367267   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:56:33 truenas k3s[12736]: E0704 11:56:33.378576   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:56:35 truenas k3s[12736]: E0704 11:56:35.369554   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:56:37 truenas k3s[12736]: E0704 11:56:37.377733   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:56:38 truenas k3s[12736]: E0704 11:56:38.353250   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:56:39 truenas k3s[12736]: E0704 11:56:39.379888   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:56:41 truenas k3s[12736]: E0704 11:56:41.351084   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:56:41 truenas k3s[12736]: E0704 11:56:41.370321   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:56:43 truenas k3s[12736]: E0704 11:56:43.378323   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:56:45 truenas k3s[12736]: E0704 11:56:45.369101   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:56:45 truenas k3s[12736]: {"level":"warn","ts":"2022-07-04T11:56:45.890+0800","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ed6c40/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Jul 04 11:56:47 truenas k3s[12736]: E0704 11:56:47.379154   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:56:49 truenas k3s[12736]: E0704 11:56:49.379938   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:56:51 truenas k3s[12736]: E0704 11:56:51.370367   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:56:53 truenas k3s[12736]: E0704 11:56:53.382195   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:56:55 truenas k3s[12736]: E0704 11:56:55.369582   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:56:56 truenas k3s[12736]: E0704 11:56:56.351303   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:56:57 truenas k3s[12736]: E0704 11:56:57.370596   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:56:59 truenas k3s[12736]: E0704 11:56:59.363368   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:57:01 truenas k3s[12736]: E0704 11:57:01.378024   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:57:03 truenas k3s[12736]: E0704 11:57:03.364780   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:57:05 truenas k3s[12736]: E0704 11:57:05.378692   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:57:05 truenas k3s[12736]: E0704 11:57:05.518642   12736 remote_image.go:216] "PullImage from image service failed" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": context deadline exceeded" image="k8s.gcr.io/sig-storage/csi-resizer:v1.2.0"
Jul 04 11:57:05 truenas k3s[12736]: E0704 11:57:05.518715   12736 kuberuntime_image.go:51] "Failed to pull image" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": context deadline exceeded" image="k8s.gcr.io/sig-storage/csi-resizer:v1.2.0"
Jul 04 11:57:05 truenas k3s[12736]: E0704 11:57:05.518867   12736 kuberuntime_manager.go:919] container &Container{Name:csi-resizer,Image:k8s.gcr.io/sig-storage/csi-resizer:v1.2.0,Command:[],Args:[--v=5 --csi-address=$(ADDRESS) --leader-election],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:ADDRESS,Value:/var/lib/csi/sockets/pluginproxy/csi.sock,ValueFrom:nil,},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:socket-dir,ReadOnly:false,MountPath:/var/lib/csi/sockets/pluginproxy/,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:kube-api-access-8tngl,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod openebs-zfs-controller-0_kube-system(a277d60e-69ba-4e90-9eb9-197ba5ebd72d): ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get "https://k8s.gcr.io/v2/": context deadline exceeded
Jul 04 11:57:07 truenas k3s[12736]: E0704 11:57:07.376972   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:57:09 truenas k3s[12736]: E0704 11:57:09.351410   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:57:09 truenas k3s[12736]: E0704 11:57:09.378051   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:57:11 truenas k3s[12736]: E0704 11:57:11.369565   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:57:13 truenas k3s[12736]: E0704 11:57:13.379762   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:57:15 truenas k3s[12736]: E0704 11:57:15.379642   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:57:16 truenas k3s[12736]: {"level":"warn","ts":"2022-07-04T11:57:16.385+0800","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ed6c40/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Jul 04 11:57:17 truenas k3s[12736]: E0704 11:57:17.369793   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:57:19 truenas k3s[12736]: E0704 11:57:19.377938   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:57:20 truenas k3s[12736]: E0704 11:57:20.767546   12736 remote_image.go:216] "PullImage from image service failed" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0"
Jul 04 11:57:20 truenas k3s[12736]: E0704 11:57:20.767623   12736 kuberuntime_image.go:51] "Failed to pull image" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0"
Jul 04 11:57:20 truenas k3s[12736]: E0704 11:57:20.767792   12736 kuberuntime_manager.go:919] container &Container{Name:csi-provisioner,Image:k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0,Command:[],Args:[--csi-address=$(ADDRESS) --v=5 --feature-gates=Topology=true --strict-topology --leader-election --extra-create-metadata=true --enable-capacity=true --default-fstype=ext4],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:ADDRESS,Value:/var/lib/csi/sockets/pluginproxy/csi.sock,ValueFrom:nil,},EnvVar{Name:NAMESPACE,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},EnvVar{Name:POD_NAME,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.name,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:socket-dir,ReadOnly:false,MountPath:/var/lib/csi/sockets/pluginproxy/,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:kube-api-access-8tngl,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod openebs-zfs-controller-0_kube-system(a277d60e-69ba-4e90-9eb9-197ba5ebd72d): ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get "https://k8s.gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Jul 04 11:57:20 truenas k3s[12736]: E0704 11:57:20.767900   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ErrImagePull: \"rpc error: code = Unknown desc = Error response from daemon: Get \\\"https://k8s.gcr.io/v2/\\\": context deadline exceeded\", failed to \"StartContainer\" for \"csi-provisioner\" with ErrImagePull: \"rpc error: code = Unknown desc = Error response from daemon: Get \\\"https://k8s.gcr.io/v2/\\\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:57:21 truenas k3s[12736]: E0704 11:57:21.368945   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:57:22 truenas k3s[12736]: E0704 11:57:22.351492   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:57:23 truenas k3s[12736]: E0704 11:57:23.377388   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:57:25 truenas k3s[12736]: E0704 11:57:25.377477   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:57:27 truenas k3s[12736]: E0704 11:57:27.369928   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:57:29 truenas k3s[12736]: E0704 11:57:29.383041   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:57:31 truenas k3s[12736]: E0704 11:57:31.143500   12736 goroutinemap.go:150] Operation for "/var/lib/kubelet/plugins_registry/zfs.csi.openebs.io-reg.sock" failed. No retries permitted until 2022-07-04 11:59:33.143458643 +0800 CST m=+34692.542606962 (durationBeforeRetry 2m2s). Error: RegisterPlugin error -- dial failed at socket /var/lib/kubelet/plugins_registry/zfs.csi.openebs.io-reg.sock, err: failed to dial socket /var/lib/kubelet/plugins_registry/zfs.csi.openebs.io-reg.sock, err: context deadline exceeded
Jul 04 11:57:31 truenas k3s[12736]: E0704 11:57:31.370101   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:57:32 truenas k3s[12736]: E0704 11:57:32.352723   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:57:33 truenas k3s[12736]: E0704 11:57:33.384123   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:57:35 truenas k3s[12736]: E0704 11:57:35.377886   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:57:36 truenas k3s[12736]: E0704 11:57:36.350619   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:57:37 truenas k3s[12736]: E0704 11:57:37.369631   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:57:39 truenas k3s[12736]: E0704 11:57:39.377699   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:57:41 truenas k3s[12736]: E0704 11:57:41.378593   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:57:43 truenas k3s[12736]: E0704 11:57:43.371086   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:57:45 truenas k3s[12736]: E0704 11:57:45.378953   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:57:47 truenas k3s[12736]: E0704 11:57:47.353042   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:57:47 truenas k3s[12736]: E0704 11:57:47.370399   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:57:49 truenas k3s[12736]: E0704 11:57:49.377518   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:57:51 truenas k3s[12736]: E0704 11:57:51.351262   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:57:51 truenas k3s[12736]: E0704 11:57:51.377435   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:57:53 truenas k3s[12736]: E0704 11:57:53.384322   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:57:55 truenas k3s[12736]: E0704 11:57:55.370375   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:57:56 truenas k3s[12736]: {"level":"warn","ts":"2022-07-04T11:57:56.126+0800","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ed6c40/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Jul 04 11:57:57 truenas k3s[12736]: E0704 11:57:57.378807   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:57:59 truenas k3s[12736]: E0704 11:57:59.377993   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:58:00 truenas k3s[12736]: E0704 11:58:00.352805   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:58:01 truenas k3s[12736]: E0704 11:58:01.378675   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:58:03 truenas k3s[12736]: E0704 11:58:03.370097   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:58:04 truenas k3s[12736]: E0704 11:58:04.351254   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:58:05 truenas k3s[12736]: E0704 11:58:05.377396   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:58:07 truenas k3s[12736]: E0704 11:58:07.377261   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:58:09 truenas k3s[12736]: E0704 11:58:09.365889   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:58:11 truenas k3s[12736]: E0704 11:58:11.378268   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:58:13 truenas k3s[12736]: E0704 11:58:13.369501   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:58:14 truenas k3s[12736]: E0704 11:58:14.352640   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:58:15 truenas k3s[12736]: E0704 11:58:15.378568   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:58:17 truenas k3s[12736]: E0704 11:58:17.351419   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:58:17 truenas k3s[12736]: E0704 11:58:17.370517   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:58:19 truenas k3s[12736]: E0704 11:58:19.377521   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:58:21 truenas k3s[12736]: E0704 11:58:21.377540   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:58:23 truenas k3s[12736]: E0704 11:58:23.369963   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:58:25 truenas k3s[12736]: E0704 11:58:25.378742   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:58:27 truenas k3s[12736]: E0704 11:58:27.369021   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:58:28 truenas k3s[12736]: E0704 11:58:28.351061   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:58:28 truenas k3s[12736]: E0704 11:58:28.352822   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:58:29 truenas k3s[12736]: E0704 11:58:29.377643   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:58:31 truenas k3s[12736]: E0704 11:58:31.378518   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:58:33 truenas k3s[12736]: E0704 11:58:33.378045   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:58:35 truenas k3s[12736]: E0704 11:58:35.378617   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:58:37 truenas k3s[12736]: E0704 11:58:37.369448   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:58:39 truenas k3s[12736]: E0704 11:58:39.378988   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:58:40 truenas k3s[12736]: {"level":"warn","ts":"2022-07-04T11:58:40.004+0800","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ed6c40/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Jul 04 11:58:41 truenas k3s[12736]: E0704 11:58:41.353589   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:58:41 truenas k3s[12736]: E0704 11:58:41.379511   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:58:42 truenas k3s[12736]: E0704 11:58:42.351358   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:58:43 truenas k3s[12736]: E0704 11:58:43.381182   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:58:45 truenas k3s[12736]: E0704 11:58:45.368984   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:58:47 truenas k3s[12736]: E0704 11:58:47.376493   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:58:49 truenas k3s[12736]: E0704 11:58:49.377261   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:58:51 truenas k3s[12736]: E0704 11:58:51.369665   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:58:53 truenas k3s[12736]: E0704 11:58:53.353573   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:58:53 truenas k3s[12736]: E0704 11:58:53.378067   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:58:55 truenas k3s[12736]: E0704 11:58:55.351262   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:58:55 truenas k3s[12736]: E0704 11:58:55.378500   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:58:57 truenas k3s[12736]: E0704 11:58:57.379005   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:58:59 truenas k3s[12736]: E0704 11:58:59.369558   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:59:01 truenas k3s[12736]: E0704 11:59:01.377284   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:59:03 truenas k3s[12736]: E0704 11:59:03.377978   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:59:05 truenas k3s[12736]: E0704 11:59:05.378854   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:59:06 truenas k3s[12736]: E0704 11:59:06.350417   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:59:07 truenas k3s[12736]: E0704 11:59:07.369386   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:59:08 truenas k3s[12736]: E0704 11:59:08.353665   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:59:09 truenas k3s[12736]: E0704 11:59:09.378141   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:59:11 truenas k3s[12736]: E0704 11:59:11.378650   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:59:13 truenas k3s[12736]: E0704 11:59:13.369688   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:59:15 truenas k3s[12736]: E0704 11:59:15.377298   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:59:17 truenas k3s[12736]: E0704 11:59:17.370140   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:59:17 truenas k3s[12736]: {"level":"warn","ts":"2022-07-04T11:59:17.516+0800","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ed6c40/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Jul 04 11:59:19 truenas k3s[12736]: E0704 11:59:19.352523   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:59:19 truenas k3s[12736]: E0704 11:59:19.379501   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:59:20 truenas k3s[12736]: E0704 11:59:20.350705   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:59:21 truenas k3s[12736]: E0704 11:59:21.369043   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:59:23 truenas k3s[12736]: E0704 11:59:23.373516   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:59:25 truenas k3s[12736]: E0704 11:59:25.379195   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:59:27 truenas k3s[12736]: E0704 11:59:27.369275   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:59:29 truenas k3s[12736]: E0704 11:59:29.381249   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:59:30 truenas k3s[12736]: E0704 11:59:30.352417   12736 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=a277d60e-69ba-4e90-9eb9-197ba5ebd72d
Jul 04 11:59:31 truenas k3s[12736]: E0704 11:59:31.369821   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:59:33 truenas k3s[12736]: E0704 11:59:33.352408   12736 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-5zm8n" podUID=6ffcb2e0-af42-46dd-8edd-7a7790ce5c4a
Jul 04 11:59:33 truenas k3s[12736]: E0704 11:59:33.380759   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
Jul 04 11:59:35 truenas k3s[12736]: E0704 11:59:35.369220   12736 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"03df0930-1cd0-4aaf-90c7-7b5aa0e06f4c\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=7
debug finished in 0 seconds for K3s Logs (last 1000 lines)


