+--------------------------------------------------------------------------------+
+                      Kubernetes Configuration @1631988307                      +
+--------------------------------------------------------------------------------+
{
  "id": 1,
  "pool": "Gamma",
  "cluster_cidr": "172.16.0.0/16",
  "service_cidr": "172.17.0.0/16",
  "cluster_dns_ip": "172.17.0.10",
  "route_v4_interface": null,
  "route_v4_gateway": null,
  "route_v6_interface": null,
  "route_v6_gateway": null,
  "node_ip": "0.0.0.0",
  "configure_gpus": true,
  "dataset": "Gamma/ix-applications"
}
debug finished in 0 seconds for Kubernetes Configuration


+--------------------------------------------------------------------------------+
+                     k3s kubectl describe nodes @1631988307                     +
+--------------------------------------------------------------------------------+
Name:               ix-truenas
Roles:              control-plane,master
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=ix-truenas
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/control-plane=true
                    node-role.kubernetes.io/master=true
                    openebs.io/nodeid=ix-truenas
                    openebs.io/nodename=ix-truenas
Annotations:        csi.volume.kubernetes.io/nodeid: {"zfs.csi.openebs.io":"ix-truenas"}
                    k3s.io/node-args:
                      ["server","--flannel-backend","none","--disable","traefik,metrics-server,local-storage","--disable-kube-proxy","--disable-network-policy",...
                    k3s.io/node-config-hash: KGQ5P7Q3OAGSEEHWD34OA36OI22LCY44AY4APE4BMRJXUPDPV4XA====
                    k3s.io/node-env: {"K3S_DATA_DIR":"/mnt/Gamma/ix-applications/k3s/data/b923ec15745249e9ba73deb8dc22f52cc5b7615239be700fbc14a9e6d80314aa"}
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Tue, 14 Sep 2021 19:36:50 -0700
Taints:             <none>
Unschedulable:      false
Lease:
  HolderIdentity:  ix-truenas
  AcquireTime:     <unset>
  RenewTime:       Sat, 18 Sep 2021 11:05:07 -0700
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Sat, 18 Sep 2021 11:01:05 -0700   Tue, 14 Sep 2021 19:36:48 -0700   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Sat, 18 Sep 2021 11:01:05 -0700   Tue, 14 Sep 2021 19:36:48 -0700   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Sat, 18 Sep 2021 11:01:05 -0700   Tue, 14 Sep 2021 19:36:48 -0700   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Sat, 18 Sep 2021 11:01:05 -0700   Tue, 14 Sep 2021 22:41:26 -0700   KubeletReady                 kubelet is posting ready status. AppArmor enabled
Addresses:
  InternalIP:  192.168.55.210
  Hostname:    ix-truenas
Capacity:
  cpu:                12
  ephemeral-storage:  59919104Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             65227648Ki
  pods:               110
Allocatable:
  cpu:                12
  ephemeral-storage:  58289304326
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             65227648Ki
  pods:               110
System Info:
  Machine ID:                 09fe4309acec4bb4a5a561e6d36d60b2
  System UUID:                03c00218-044d-0585-5d06-000700080009
  Boot ID:                    f071eec1-a32d-45c2-9fb0-8292bfd626ac
  Kernel Version:             5.10.42+truenas
  OS Image:                   Debian GNU/Linux 11 (bullseye)
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  docker://20.10.8
  Kubelet Version:            v1.21.0-k3s1
  Kube-Proxy Version:         v1.21.0-k3s1
PodCIDR:                      172.16.0.0/16
PodCIDRs:                     172.16.0.0/16
Non-terminated Pods:          (3 in total)
  Namespace                   Name                        CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                        ------------  ----------  ---------------  -------------  ---
  kube-system                 coredns-7448499f4d-fmc97    100m (0%)     0 (0%)      70Mi (0%)        170Mi (0%)     3d15h
  kube-system                 openebs-zfs-node-xm2jq      0 (0%)        0 (0%)      0 (0%)           0 (0%)         3d15h
  kube-system                 openebs-zfs-controller-0    0 (0%)        0 (0%)      0 (0%)           0 (0%)         3d15h
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests   Limits
  --------           --------   ------
  cpu                100m (0%)  0 (0%)
  memory             70Mi (0%)  170Mi (0%)
  ephemeral-storage  0 (0%)     0 (0%)
  hugepages-1Gi      0 (0%)     0 (0%)
  hugepages-2Mi      0 (0%)     0 (0%)
Events:              <none>
debug finished in 0 seconds for k3s kubectl describe nodes


+--------------------------------------------------------------------------------+
+k3s kubectl get pods,svc,daemonsets,deployments,statefulset,sc,pvc,ns,job --all-namespaces -o wide @1631988307+
+--------------------------------------------------------------------------------+
NAMESPACE     NAME                           READY   STATUS    RESTARTS   AGE     IP               NODE         NOMINATED NODE   READINESS GATES
kube-system   pod/coredns-7448499f4d-fmc97   1/1     Running   3          3d15h   172.16.0.9       ix-truenas   <none>           <none>
kube-system   pod/openebs-zfs-node-xm2jq     2/2     Running   8          3d15h   192.168.55.210   ix-truenas   <none>           <none>
kube-system   pod/openebs-zfs-controller-0   5/5     Running   16         3d15h   172.16.0.8       ix-truenas   <none>           <none>

NAMESPACE     NAME                 TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)                  AGE     SELECTOR
default       service/kubernetes   ClusterIP   172.17.0.1    <none>        443/TCP                  3d15h   <none>
kube-system   service/kube-dns     ClusterIP   172.17.0.10   <none>        53/UDP,53/TCP,9153/TCP   3d15h   k8s-app=kube-dns

NAMESPACE     NAME                              DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE     CONTAINERS                                     IMAGES                                                                          SELECTOR
kube-system   daemonset.apps/openebs-zfs-node   1         1         1       1            1           <none>          3d15h   csi-node-driver-registrar,openebs-zfs-plugin   k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0,openebs/zfs-driver:ci   app=openebs-zfs-node

NAMESPACE     NAME                      READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS   IMAGES                          SELECTOR
kube-system   deployment.apps/coredns   1/1     1            1           3d15h   coredns      rancher/coredns-coredns:1.8.3   k8s-app=kube-dns

NAMESPACE     NAME                                      READY   AGE     CONTAINERS                                                                           IMAGES
kube-system   statefulset.apps/openebs-zfs-controller   1/1     3d15h   csi-resizer,csi-snapshotter,snapshot-controller,csi-provisioner,openebs-zfs-plugin   k8s.gcr.io/sig-storage/csi-resizer:v1.1.0,k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0,k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0,k8s.gcr.io/sig-storage/csi-provisioner:v2.1.0,openebs/zfs-driver:ci

NAMESPACE   NAME                                                          PROVISIONER          RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
            storageclass.storage.k8s.io/openebs-zfspv-default (default)   zfs.csi.openebs.io   Delete          Immediate           true                   3d15h

NAMESPACE   NAME                        STATUS   AGE
            namespace/default           Active   3d15h
            namespace/kube-system       Active   3d15h
            namespace/kube-public       Active   3d15h
            namespace/kube-node-lease   Active   3d15h
            namespace/openebs           Active   3d15h
debug finished in 1 seconds for k3s kubectl get pods,svc,daemonsets,deployments,statefulset,sc,pvc,ns,job --all-namespaces -o wide


+--------------------------------------------------------------------------------+
+         k3s kubectl describe deployments --all-namespaces @1631988308          +
+--------------------------------------------------------------------------------+
Name:                   coredns
Namespace:              kube-system
CreationTimestamp:      Tue, 14 Sep 2021 19:36:53 -0700
Labels:                 k8s-app=kube-dns
                        kubernetes.io/name=CoreDNS
                        objectset.rio.cattle.io/hash=bce283298811743a0386ab510f2f67ef74240c57
Annotations:            deployment.kubernetes.io/revision: 1
                        objectset.rio.cattle.io/applied:
                          H4sIAAAAAAAA/6xV32/bNhD+V4Z7ln8obhNXwB66OFuDLZ6RH30pjOJMnSzOFI8jKTdGoP99OEl2nDRNu2FPpsnvjt/38e70AOj0R/JBs4UM0Lkw2qaQwEbbHDKYkTO8q8hGSKCiiD...
                        objectset.rio.cattle.io/id: 
                        objectset.rio.cattle.io/owner-gvk: k3s.cattle.io/v1, Kind=Addon
                        objectset.rio.cattle.io/owner-name: coredns
                        objectset.rio.cattle.io/owner-namespace: kube-system
Selector:               k8s-app=kube-dns
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  1 max unavailable, 25% max surge
Pod Template:
  Labels:           k8s-app=kube-dns
  Service Account:  coredns
  Containers:
   coredns:
    Image:       rancher/coredns-coredns:1.8.3
    Ports:       53/UDP, 53/TCP, 9153/TCP
    Host Ports:  0/UDP, 0/TCP, 0/TCP
    Args:
      -conf
      /etc/coredns/Corefile
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=1s period=10s #success=1 #failure=3
    Readiness:    http-get http://:8181/ready delay=0s timeout=1s period=2s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (ro)
  Volumes:
   config-volume:
    Type:               ConfigMap (a volume populated by a ConfigMap)
    Name:               coredns
    Optional:           false
  Priority Class Name:  system-cluster-critical
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   coredns-7448499f4d (1/1 replicas created)
Events:          <none>
debug finished in 0 seconds for k3s kubectl describe deployments --all-namespaces


+--------------------------------------------------------------------------------+
+             k3s kubectl describe pods --all-namespaces @1631988308             +
+--------------------------------------------------------------------------------+
Name:                 coredns-7448499f4d-fmc97
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 ix-truenas/192.168.55.210
Start Time:           Tue, 14 Sep 2021 19:37:06 -0700
Labels:               k8s-app=kube-dns
                      pod-template-hash=7448499f4d
Annotations:          k8s.v1.cni.cncf.io/network-status:
                        [{
                            "name": "ix-net",
                            "interface": "eth0",
                            "ips": [
                                "172.16.0.9"
                            ],
                            "mac": "7e:a2:28:92:4d:a1",
                            "default": true,
                            "dns": {}
                        }]
                      k8s.v1.cni.cncf.io/networks-status:
                        [{
                            "name": "ix-net",
                            "interface": "eth0",
                            "ips": [
                                "172.16.0.9"
                            ],
                            "mac": "7e:a2:28:92:4d:a1",
                            "default": true,
                            "dns": {}
                        }]
Status:               Running
IP:                   172.16.0.9
IPs:
  IP:           172.16.0.9
Controlled By:  ReplicaSet/coredns-7448499f4d
Containers:
  coredns:
    Container ID:  docker://b699e723eb6abe5acbdcddafaafa0550fb6bc6fe41bf41f592aa3c76cec94643
    Image:         rancher/coredns-coredns:1.8.3
    Image ID:      docker://sha256:3885a5b7f138c3e8779a18f672fd55c3c7cf0166b67cf5143ab0cd137eb38418
    Ports:         53/UDP, 53/TCP, 9153/TCP
    Host Ports:    0/UDP, 0/TCP, 0/TCP
    Args:
      -conf
      /etc/coredns/Corefile
    State:          Running
      Started:      Tue, 14 Sep 2021 22:41:21 -0700
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Tue, 14 Sep 2021 21:25:04 -0700
      Finished:     Tue, 14 Sep 2021 22:39:05 -0700
    Ready:          True
    Restart Count:  3
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=1s period=10s #success=1 #failure=3
    Readiness:    http-get http://:8181/ready delay=0s timeout=1s period=2s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bkqvw (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      coredns
    Optional:  false
  kube-api-access-bkqvw:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              beta.kubernetes.io/os=linux
Tolerations:                 CriticalAddonsOnly op=Exists
                             node-role.kubernetes.io/control-plane:NoSchedule op=Exists
                             node-role.kubernetes.io/master:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>


Name:                 openebs-zfs-node-xm2jq
Namespace:            kube-system
Priority:             2000001000
Priority Class Name:  system-node-critical
Node:                 ix-truenas/192.168.55.210
Start Time:           Tue, 14 Sep 2021 19:37:06 -0700
Labels:               app=openebs-zfs-node
                      controller-revision-hash=568c578775
                      openebs.io/component-name=openebs-zfs-node
                      openebs.io/version=ci
                      pod-template-generation=1
                      role=openebs-zfs
Annotations:          <none>
Status:               Running
IP:                   192.168.55.210
IPs:
  IP:           192.168.55.210
Controlled By:  DaemonSet/openebs-zfs-node
Containers:
  csi-node-driver-registrar:
    Container ID:  docker://72842482eb1ee57dca08ddb3beec03c691f43184ea23181a70e36b7f3b657c1d
    Image:         k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0
    Image ID:      docker-pullable://k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f
    Port:          <none>
    Host Port:     <none>
    Args:
      --v=5
      --csi-address=$(ADDRESS)
      --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)
    State:          Running
      Started:      Tue, 14 Sep 2021 22:41:18 -0700
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Tue, 14 Sep 2021 21:25:00 -0700
      Finished:     Tue, 14 Sep 2021 22:39:05 -0700
    Ready:          True
    Restart Count:  3
    Environment:
      ADDRESS:               /plugin/csi.sock
      DRIVER_REG_SOCK_PATH:  /var/lib/kubelet/plugins/zfs-localpv/csi.sock
      KUBE_NODE_NAME:         (v1:spec.nodeName)
      NODE_DRIVER:           openebs-zfs
    Mounts:
      /plugin from plugin-dir (rw)
      /registration from registration-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-kj2xl (ro)
  openebs-zfs-plugin:
    Container ID:  docker://41a852f775e7af9baefd84f83ab6d14f09b09adc6a7282632296ea2866c4d2b7
    Image:         openebs/zfs-driver:ci
    Image ID:      docker-pullable://openebs/zfs-driver@sha256:b3cd33326c68242cbd21b92fc8b78e768fa899b7b91f1e3fcad2f2a7013b4ce9
    Port:          <none>
    Host Port:     <none>
    Args:
      --nodename=$(OPENEBS_NODE_NAME)
      --endpoint=$(OPENEBS_CSI_ENDPOINT)
      --plugin=$(OPENEBS_NODE_DRIVER)
    State:          Running
      Started:      Tue, 14 Sep 2021 22:41:35 -0700
    Last State:     Terminated
      Reason:       Error
      Exit Code:    255
      Started:      Tue, 14 Sep 2021 22:41:19 -0700
      Finished:     Tue, 14 Sep 2021 22:41:21 -0700
    Ready:          True
    Restart Count:  5
    Environment:
      OPENEBS_NODE_NAME:      (v1:spec.nodeName)
      OPENEBS_CSI_ENDPOINT:  unix:///plugin/csi.sock
      OPENEBS_NODE_DRIVER:   agent
      OPENEBS_NAMESPACE:     openebs
    Mounts:
      /dev from device-dir (rw)
      /home/keys from encr-keys (rw)
      /host from host-root (ro)
      /plugin from plugin-dir (rw)
      /sbin/zfs from chroot-zfs (rw,path="zfs")
      /var/lib/kubelet/ from pods-mount-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-kj2xl (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  device-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /dev
    HostPathType:  Directory
  encr-keys:
    Type:          HostPath (bare host directory volume)
    Path:          /home/keys
    HostPathType:  DirectoryOrCreate
  chroot-zfs:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      openebs-zfspv-bin
    Optional:  false
  host-root:
    Type:          HostPath (bare host directory volume)
    Path:          /
    HostPathType:  Directory
  registration-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/plugins_registry/
    HostPathType:  DirectoryOrCreate
  plugin-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/plugins/zfs-localpv/
    HostPathType:  DirectoryOrCreate
  pods-mount-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/
    HostPathType:  Directory
  kube-api-access-kj2xl:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/network-unavailable:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:                      <none>


Name:                 openebs-zfs-controller-0
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 ix-truenas/192.168.55.210
Start Time:           Tue, 14 Sep 2021 19:37:05 -0700
Labels:               app=openebs-zfs-controller
                      controller-revision-hash=openebs-zfs-controller-56b9b7c4c7
                      openebs.io/component-name=openebs-zfs-controller
                      openebs.io/version=ci
                      role=openebs-zfs
                      statefulset.kubernetes.io/pod-name=openebs-zfs-controller-0
Annotations:          k8s.v1.cni.cncf.io/network-status:
                        [{
                            "name": "ix-net",
                            "interface": "eth0",
                            "ips": [
                                "172.16.0.8"
                            ],
                            "mac": "a2:13:16:19:90:f5",
                            "default": true,
                            "dns": {}
                        }]
                      k8s.v1.cni.cncf.io/networks-status:
                        [{
                            "name": "ix-net",
                            "interface": "eth0",
                            "ips": [
                                "172.16.0.8"
                            ],
                            "mac": "a2:13:16:19:90:f5",
                            "default": true,
                            "dns": {}
                        }]
Status:               Running
IP:                   172.16.0.8
IPs:
  IP:           172.16.0.8
Controlled By:  StatefulSet/openebs-zfs-controller
Containers:
  csi-resizer:
    Container ID:  docker://d0acf0c5c93ecf43bffc8bde48a078db85eaa5c56e552595c864260844ef3a82
    Image:         k8s.gcr.io/sig-storage/csi-resizer:v1.1.0
    Image ID:      docker-pullable://k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a
    Port:          <none>
    Host Port:     <none>
    Args:
      --v=5
      --csi-address=$(ADDRESS)
      --leader-election
    State:          Running
      Started:      Tue, 14 Sep 2021 22:41:20 -0700
    Last State:     Terminated
      Reason:       Error
      Exit Code:    2
      Started:      Tue, 14 Sep 2021 21:25:04 -0700
      Finished:     Tue, 14 Sep 2021 22:39:05 -0700
    Ready:          True
    Restart Count:  3
    Environment:
      ADDRESS:  /var/lib/csi/sockets/pluginproxy/csi.sock
    Mounts:
      /var/lib/csi/sockets/pluginproxy/ from socket-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-56p6r (ro)
  csi-snapshotter:
    Container ID:  docker://7c1392f0a16673e8ff571a002eced62c5d174c3d911dd6076d0f07d0767fc17a
    Image:         k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0
    Image ID:      docker-pullable://k8s.gcr.io/sig-storage/csi-snapshotter@sha256:51f2dfde5bccac7854b3704689506aeecfb793328427b91115ba253a93e60782
    Port:          <none>
    Host Port:     <none>
    Args:
      --csi-address=$(ADDRESS)
      --leader-election
    State:          Running
      Started:      Tue, 14 Sep 2021 22:41:21 -0700
    Last State:     Terminated
      Reason:       Error
      Exit Code:    2
      Started:      Tue, 14 Sep 2021 21:25:04 -0700
      Finished:     Tue, 14 Sep 2021 22:39:05 -0700
    Ready:          True
    Restart Count:  3
    Environment:
      ADDRESS:  /var/lib/csi/sockets/pluginproxy/csi.sock
    Mounts:
      /var/lib/csi/sockets/pluginproxy/ from socket-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-56p6r (ro)
  snapshot-controller:
    Container ID:  docker://6f3a3d823bea8863def9fcc9f15135dd8bddb4f53bbad800fdf6093756eff343
    Image:         k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0
    Image ID:      docker-pullable://k8s.gcr.io/sig-storage/snapshot-controller@sha256:00fcc441ea9f72899c25eed61d602272a2a58c5f0014332bdcb5ac24acef08e4
    Port:          <none>
    Host Port:     <none>
    Args:
      --v=5
      --leader-election=true
    State:          Running
      Started:      Tue, 14 Sep 2021 22:41:21 -0700
    Last State:     Terminated
      Reason:       Error
      Exit Code:    2
      Started:      Tue, 14 Sep 2021 21:25:04 -0700
      Finished:     Tue, 14 Sep 2021 22:39:05 -0700
    Ready:          True
    Restart Count:  3
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-56p6r (ro)
  csi-provisioner:
    Container ID:  docker://8d7b5b7333d978861f456d17be943beba54cee84db4663bc0cad32920e360b4b
    Image:         k8s.gcr.io/sig-storage/csi-provisioner:v2.1.0
    Image ID:      docker-pullable://k8s.gcr.io/sig-storage/csi-provisioner@sha256:20c828075d1e36f679d6a91e905b0927141eef5e15be0c9a1ca4a6a0ed9313d2
    Port:          <none>
    Host Port:     <none>
    Args:
      --csi-address=$(ADDRESS)
      --v=5
      --feature-gates=Topology=true
      --strict-topology
      --leader-election
      --extra-create-metadata=true
      --default-fstype=ext4
    State:          Running
      Started:      Tue, 14 Sep 2021 22:41:36 -0700
    Last State:     Terminated
      Reason:       Error
      Exit Code:    255
      Started:      Tue, 14 Sep 2021 22:41:21 -0700
      Finished:     Tue, 14 Sep 2021 22:41:21 -0700
    Ready:          True
    Restart Count:  4
    Environment:
      ADDRESS:  /var/lib/csi/sockets/pluginproxy/csi.sock
    Mounts:
      /var/lib/csi/sockets/pluginproxy/ from socket-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-56p6r (ro)
  openebs-zfs-plugin:
    Container ID:  docker://ea912e99036c84ddceb164f1afb6de81999819bd174c99e1b3d2cba823aa7f06
    Image:         openebs/zfs-driver:ci
    Image ID:      docker-pullable://openebs/zfs-driver@sha256:b3cd33326c68242cbd21b92fc8b78e768fa899b7b91f1e3fcad2f2a7013b4ce9
    Port:          <none>
    Host Port:     <none>
    Args:
      --endpoint=$(OPENEBS_CSI_ENDPOINT)
      --plugin=$(OPENEBS_CONTROLLER_DRIVER)
    State:          Running
      Started:      Tue, 14 Sep 2021 22:41:21 -0700
    Last State:     Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Tue, 14 Sep 2021 21:25:04 -0700
      Finished:     Tue, 14 Sep 2021 22:39:15 -0700
    Ready:          True
    Restart Count:  3
    Environment:
      OPENEBS_CONTROLLER_DRIVER:    controller
      OPENEBS_CSI_ENDPOINT:         unix:///var/lib/csi/sockets/pluginproxy/csi.sock
      OPENEBS_NAMESPACE:            openebs
      OPENEBS_IO_INSTALLER_TYPE:    zfs-operator
      OPENEBS_IO_ENABLE_ANALYTICS:  true
    Mounts:
      /var/lib/csi/sockets/pluginproxy/ from socket-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-56p6r (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  socket-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-56p6r:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>
debug finished in 0 seconds for k3s kubectl describe pods --all-namespaces


+--------------------------------------------------------------------------------+
+           k3s kubectl describe services --all-namespaces @1631988308           +
+--------------------------------------------------------------------------------+
Name:              kubernetes
Namespace:         default
Labels:            component=apiserver
                   provider=kubernetes
Annotations:       <none>
Selector:          <none>
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                172.17.0.1
IPs:               172.17.0.1
Port:              https  443/TCP
TargetPort:        6443/TCP
Endpoints:         192.168.55.210:6443
Session Affinity:  None
Events:            <none>


Name:              kube-dns
Namespace:         kube-system
Labels:            k8s-app=kube-dns
                   kubernetes.io/cluster-service=true
                   kubernetes.io/name=CoreDNS
                   objectset.rio.cattle.io/hash=bce283298811743a0386ab510f2f67ef74240c57
Annotations:       objectset.rio.cattle.io/applied:
                     {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"objectset.rio.cattle.io/id":"","objectset.rio.cattle.io/owner-gvk":"k3s.ca...
                   objectset.rio.cattle.io/id: 
                   objectset.rio.cattle.io/owner-gvk: k3s.cattle.io/v1, Kind=Addon
                   objectset.rio.cattle.io/owner-name: coredns
                   objectset.rio.cattle.io/owner-namespace: kube-system
                   prometheus.io/port: 9153
                   prometheus.io/scrape: true
Selector:          k8s-app=kube-dns
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                172.17.0.10
IPs:               172.17.0.10
Port:              dns  53/UDP
TargetPort:        53/UDP
Endpoints:         172.16.0.9:53
Port:              dns-tcp  53/TCP
TargetPort:        53/TCP
Endpoints:         172.16.0.9:53
Port:              metrics  9153/TCP
TargetPort:        9153/TCP
Endpoints:         172.16.0.9:9153
Session Affinity:  None
Events:            <none>
debug finished in 0 seconds for k3s kubectl describe services --all-namespaces


+--------------------------------------------------------------------------------+
+         k3s kubectl describe statefulset --all-namespaces @1631988308          +
+--------------------------------------------------------------------------------+
Name:               openebs-zfs-controller
Namespace:          kube-system
CreationTimestamp:  Tue, 14 Sep 2021 19:36:56 -0700
Selector:           app=openebs-zfs-controller,role=openebs-zfs
Labels:             objectset.rio.cattle.io/hash=4b1efd47d7f37713706abe9f8b0de3f0adf02e2c
                    openebs.io/component-name=openebs-zfs-controller
                    openebs.io/version=ci
Annotations:        objectset.rio.cattle.io/applied:
                      H4sIAAAAAAAA/8xWX2/bthf9Kj9c9KEFRP9p8ls6AX5QYz0YTR0jNgoUhRHQ1JXMmSY58kqzG/i7D5RsV03j/Bm2bm82yXt4zuG5Iu+AW/kJnZdGQwzcWt+t+hDBSuoMYpgSJ8xLNU...
                    objectset.rio.cattle.io/id: 
                    objectset.rio.cattle.io/owner-gvk: k3s.cattle.io/v1, Kind=Addon
                    objectset.rio.cattle.io/owner-name: zfs-operator
                    objectset.rio.cattle.io/owner-namespace: kube-system
Replicas:           1 desired | 1 total
Update Strategy:    RollingUpdate
  Partition:        0
Pods Status:        1 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           app=openebs-zfs-controller
                    openebs.io/component-name=openebs-zfs-controller
                    openebs.io/version=ci
                    role=openebs-zfs
  Service Account:  openebs-zfs-controller-sa
  Containers:
   csi-resizer:
    Image:      k8s.gcr.io/sig-storage/csi-resizer:v1.1.0
    Port:       <none>
    Host Port:  <none>
    Args:
      --v=5
      --csi-address=$(ADDRESS)
      --leader-election
    Environment:
      ADDRESS:  /var/lib/csi/sockets/pluginproxy/csi.sock
    Mounts:
      /var/lib/csi/sockets/pluginproxy/ from socket-dir (rw)
   csi-snapshotter:
    Image:      k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0
    Port:       <none>
    Host Port:  <none>
    Args:
      --csi-address=$(ADDRESS)
      --leader-election
    Environment:
      ADDRESS:  /var/lib/csi/sockets/pluginproxy/csi.sock
    Mounts:
      /var/lib/csi/sockets/pluginproxy/ from socket-dir (rw)
   snapshot-controller:
    Image:      k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0
    Port:       <none>
    Host Port:  <none>
    Args:
      --v=5
      --leader-election=true
    Environment:  <none>
    Mounts:       <none>
   csi-provisioner:
    Image:      k8s.gcr.io/sig-storage/csi-provisioner:v2.1.0
    Port:       <none>
    Host Port:  <none>
    Args:
      --csi-address=$(ADDRESS)
      --v=5
      --feature-gates=Topology=true
      --strict-topology
      --leader-election
      --extra-create-metadata=true
      --default-fstype=ext4
    Environment:
      ADDRESS:  /var/lib/csi/sockets/pluginproxy/csi.sock
    Mounts:
      /var/lib/csi/sockets/pluginproxy/ from socket-dir (rw)
   openebs-zfs-plugin:
    Image:      openebs/zfs-driver:ci
    Port:       <none>
    Host Port:  <none>
    Args:
      --endpoint=$(OPENEBS_CSI_ENDPOINT)
      --plugin=$(OPENEBS_CONTROLLER_DRIVER)
    Environment:
      OPENEBS_CONTROLLER_DRIVER:    controller
      OPENEBS_CSI_ENDPOINT:         unix:///var/lib/csi/sockets/pluginproxy/csi.sock
      OPENEBS_NAMESPACE:            openebs
      OPENEBS_IO_INSTALLER_TYPE:    zfs-operator
      OPENEBS_IO_ENABLE_ANALYTICS:  true
    Mounts:
      /var/lib/csi/sockets/pluginproxy/ from socket-dir (rw)
  Volumes:
   socket-dir:
    Type:               EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:             
    SizeLimit:          <unset>
  Priority Class Name:  system-cluster-critical
Volume Claims:          <none>
Events:                 <none>
debug finished in 0 seconds for k3s kubectl describe statefulset --all-namespaces


+--------------------------------------------------------------------------------+
+             k3s kubectl describe job --all-namespaces @1631988308              +
+--------------------------------------------------------------------------------+
No resources found
debug finished in 0 seconds for k3s kubectl describe job --all-namespaces


+--------------------------------------------------------------------------------+
+           k3s kubectl describe cronjob --all-namespaces @1631988308            +
+--------------------------------------------------------------------------------+
No resources found
debug finished in 1 seconds for k3s kubectl describe cronjob --all-namespaces


+--------------------------------------------------------------------------------+
+          k3s kubectl describe daemonsets --all-namespaces @1631988309          +
+--------------------------------------------------------------------------------+
Name:           openebs-zfs-node
Selector:       app=openebs-zfs-node
Node-Selector:  <none>
Labels:         objectset.rio.cattle.io/hash=4b1efd47d7f37713706abe9f8b0de3f0adf02e2c
                openebs.io/component-name=openebs-zfs-node
                openebs.io/version=ci
Annotations:    deprecated.daemonset.template.generation: 1
                objectset.rio.cattle.io/applied:
                  H4sIAAAAAAAA/6xW0W7buBL9lYLoBVpAsuymbXoF5MG1tdugG9uwk74UgUGRI5lrihRISok30L8vSMm2bMtJFsibRM6cmTlzOOQTwjn7BUozKVCIcJ7roBwgD62ZoChEYwyZFAswyE...
                objectset.rio.cattle.io/id: 
                objectset.rio.cattle.io/owner-gvk: k3s.cattle.io/v1, Kind=Addon
                objectset.rio.cattle.io/owner-name: zfs-operator
                objectset.rio.cattle.io/owner-namespace: kube-system
Desired Number of Nodes Scheduled: 1
Current Number of Nodes Scheduled: 1
Number of Nodes Scheduled with Up-to-date Pods: 1
Number of Nodes Scheduled with Available Pods: 1
Number of Nodes Misscheduled: 0
Pods Status:  1 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           app=openebs-zfs-node
                    openebs.io/component-name=openebs-zfs-node
                    openebs.io/version=ci
                    role=openebs-zfs
  Service Account:  openebs-zfs-node-sa
  Containers:
   csi-node-driver-registrar:
    Image:      k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0
    Port:       <none>
    Host Port:  <none>
    Args:
      --v=5
      --csi-address=$(ADDRESS)
      --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)
    Environment:
      ADDRESS:               /plugin/csi.sock
      DRIVER_REG_SOCK_PATH:  /var/lib/kubelet/plugins/zfs-localpv/csi.sock
      KUBE_NODE_NAME:         (v1:spec.nodeName)
      NODE_DRIVER:           openebs-zfs
    Mounts:
      /plugin from plugin-dir (rw)
      /registration from registration-dir (rw)
   openebs-zfs-plugin:
    Image:      openebs/zfs-driver:ci
    Port:       <none>
    Host Port:  <none>
    Args:
      --nodename=$(OPENEBS_NODE_NAME)
      --endpoint=$(OPENEBS_CSI_ENDPOINT)
      --plugin=$(OPENEBS_NODE_DRIVER)
    Environment:
      OPENEBS_NODE_NAME:      (v1:spec.nodeName)
      OPENEBS_CSI_ENDPOINT:  unix:///plugin/csi.sock
      OPENEBS_NODE_DRIVER:   agent
      OPENEBS_NAMESPACE:     openebs
    Mounts:
      /dev from device-dir (rw)
      /home/keys from encr-keys (rw)
      /host from host-root (ro)
      /plugin from plugin-dir (rw)
      /sbin/zfs from chroot-zfs (rw,path="zfs")
      /var/lib/kubelet/ from pods-mount-dir (rw)
  Volumes:
   device-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /dev
    HostPathType:  Directory
   encr-keys:
    Type:          HostPath (bare host directory volume)
    Path:          /home/keys
    HostPathType:  DirectoryOrCreate
   chroot-zfs:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      openebs-zfspv-bin
    Optional:  false
   host-root:
    Type:          HostPath (bare host directory volume)
    Path:          /
    HostPathType:  Directory
   registration-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/plugins_registry/
    HostPathType:  DirectoryOrCreate
   plugin-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/plugins/zfs-localpv/
    HostPathType:  DirectoryOrCreate
   pods-mount-dir:
    Type:               HostPath (bare host directory volume)
    Path:               /var/lib/kubelet/
    HostPathType:       Directory
  Priority Class Name:  system-node-critical
Events:                 <none>
debug finished in 0 seconds for k3s kubectl describe daemonsets --all-namespaces


+--------------------------------------------------------------------------------+
+                            docker ps -a @1631988309                            +
+--------------------------------------------------------------------------------+
CONTAINER ID   IMAGE               COMMAND                  CREATED      STATUS                    PORTS     NAMES
8d7b5b7333d9   e0d187f105d6        "/csi-provisioner --…"   3 days ago   Up 3 days                           k8s_csi-provisioner_openebs-zfs-controller-0_kube-system_c967a0a1-ac3a-4a75-bd4a-b0c3b13f7512_4
41a852f775e7   ec422619cddf        "/usr/local/bin/zfs-…"   3 days ago   Up 3 days                           k8s_openebs-zfs-plugin_openebs-zfs-node-xm2jq_kube-system_19ff9693-df92-45a0-a5f6-513f87ec2431_5
ea912e99036c   ec422619cddf        "/usr/local/bin/zfs-…"   3 days ago   Up 3 days                           k8s_openebs-zfs-plugin_openebs-zfs-controller-0_kube-system_c967a0a1-ac3a-4a75-bd4a-b0c3b13f7512_3
b699e723eb6a   3885a5b7f138        "/coredns -conf /etc…"   3 days ago   Up 3 days                           k8s_coredns_coredns-7448499f4d-fmc97_kube-system_397f7f6c-930c-41f0-8d65-c85deac56607_3
1593d48c919b   e0d187f105d6        "/csi-provisioner --…"   3 days ago   Exited (255) 3 days ago             k8s_csi-provisioner_openebs-zfs-controller-0_kube-system_c967a0a1-ac3a-4a75-bd4a-b0c3b13f7512_3
6f3a3d823bea   f1d8a00ae690        "/snapshot-controlle…"   3 days ago   Up 3 days                           k8s_snapshot-controller_openebs-zfs-controller-0_kube-system_c967a0a1-ac3a-4a75-bd4a-b0c3b13f7512_3
fb0eacd1d769   rancher/pause:3.1   "/pause"                 3 days ago   Up 3 days                           k8s_POD_coredns-7448499f4d-fmc97_kube-system_397f7f6c-930c-41f0-8d65-c85deac56607_3
7c1392f0a166   da32a49a903a        "/csi-snapshotter --…"   3 days ago   Up 3 days                           k8s_csi-snapshotter_openebs-zfs-controller-0_kube-system_c967a0a1-ac3a-4a75-bd4a-b0c3b13f7512_3
d0acf0c5c93e   a8fe79377034        "/csi-resizer --v=5 …"   3 days ago   Up 3 days                           k8s_csi-resizer_openebs-zfs-controller-0_kube-system_c967a0a1-ac3a-4a75-bd4a-b0c3b13f7512_3
4b47b7630420   rancher/pause:3.1   "/pause"                 3 days ago   Up 3 days                           k8s_POD_openebs-zfs-controller-0_kube-system_c967a0a1-ac3a-4a75-bd4a-b0c3b13f7512_3
20b9f7a939ef   ec422619cddf        "/usr/local/bin/zfs-…"   3 days ago   Exited (255) 3 days ago             k8s_openebs-zfs-plugin_openebs-zfs-node-xm2jq_kube-system_19ff9693-df92-45a0-a5f6-513f87ec2431_4
72842482eb1e   ef2b13b2a066        "/csi-node-driver-re…"   3 days ago   Up 3 days                           k8s_csi-node-driver-registrar_openebs-zfs-node-xm2jq_kube-system_19ff9693-df92-45a0-a5f6-513f87ec2431_3
865ff064fb37   rancher/pause:3.1   "/pause"                 3 days ago   Up 3 days                           k8s_POD_openebs-zfs-node-xm2jq_kube-system_19ff9693-df92-45a0-a5f6-513f87ec2431_3
7395f7e8d762   ec422619cddf        "/usr/local/bin/zfs-…"   3 days ago   Exited (137) 3 days ago             k8s_openebs-zfs-plugin_openebs-zfs-controller-0_kube-system_c967a0a1-ac3a-4a75-bd4a-b0c3b13f7512_2
a51da35f29ff   f1d8a00ae690        "/snapshot-controlle…"   3 days ago   Exited (2) 3 days ago               k8s_snapshot-controller_openebs-zfs-controller-0_kube-system_c967a0a1-ac3a-4a75-bd4a-b0c3b13f7512_2
07c0a7328e29   da32a49a903a        "/csi-snapshotter --…"   3 days ago   Exited (2) 3 days ago               k8s_csi-snapshotter_openebs-zfs-controller-0_kube-system_c967a0a1-ac3a-4a75-bd4a-b0c3b13f7512_2
6d985f4c825a   a8fe79377034        "/csi-resizer --v=5 …"   3 days ago   Exited (2) 3 days ago               k8s_csi-resizer_openebs-zfs-controller-0_kube-system_c967a0a1-ac3a-4a75-bd4a-b0c3b13f7512_2
05d4c47d312f   3885a5b7f138        "/coredns -conf /etc…"   3 days ago   Exited (0) 3 days ago               k8s_coredns_coredns-7448499f4d-fmc97_kube-system_397f7f6c-930c-41f0-8d65-c85deac56607_2
feebc253d054   rancher/pause:3.1   "/pause"                 3 days ago   Exited (0) 3 days ago               k8s_POD_coredns-7448499f4d-fmc97_kube-system_397f7f6c-930c-41f0-8d65-c85deac56607_2
a5189c4385c7   rancher/pause:3.1   "/pause"                 3 days ago   Exited (0) 3 days ago               k8s_POD_openebs-zfs-controller-0_kube-system_c967a0a1-ac3a-4a75-bd4a-b0c3b13f7512_2
0a91990b0e6f   ef2b13b2a066        "/csi-node-driver-re…"   3 days ago   Exited (0) 3 days ago               k8s_csi-node-driver-registrar_openebs-zfs-node-xm2jq_kube-system_19ff9693-df92-45a0-a5f6-513f87ec2431_2
135628a79128   rancher/pause:3.1   "/pause"                 3 days ago   Exited (0) 3 days ago               k8s_POD_openebs-zfs-node-xm2jq_kube-system_19ff9693-df92-45a0-a5f6-513f87ec2431_2
debug finished in 0 seconds for docker ps -a


+--------------------------------------------------------------------------------+
+                          docker images -a @1631988309                          +
+--------------------------------------------------------------------------------+
REPOSITORY                                         TAG                     IMAGE ID       CREATED        SIZE
openebs/zfs-driver                                 ci                      ec422619cddf   5 days ago     175MB
plexinc/pms-docker                                 1.24.2.4973-2b1b51db9   9149b97bd2ad   3 weeks ago    643MB
rancher/coredns-coredns                            1.8.3                   3885a5b7f138   6 months ago   43.5MB
k8s.gcr.io/sig-storage/csi-snapshotter             v4.0.0                  da32a49a903a   9 months ago   49.5MB
k8s.gcr.io/sig-storage/snapshot-controller         v4.0.0                  f1d8a00ae690   9 months ago   46.6MB
k8s.gcr.io/sig-storage/csi-resizer                 v1.1.0                  a8fe79377034   9 months ago   49.2MB
k8s.gcr.io/sig-storage/csi-provisioner             v2.1.0                  e0d187f105d6   9 months ago   51.6MB
k8s.gcr.io/sig-storage/csi-node-driver-registrar   v2.1.0                  ef2b13b2a066   9 months ago   19.6MB
rancher/klipper-lb                                 v0.1.2                  897ce3c5fc8f   2 years ago    6.1MB
rancher/pause                                      3.1                     da86e6ba6ca1   3 years ago    742kB
debug finished in 0 seconds for docker images -a


