Jul 24 08:51:17 bluth k3s[11030]: time="2022-07-24T08:51:17+01:00" level=info msg="Starting k3s v1.23.5+k3s-fbfa51e5-dirty (fbfa51e5)" Jul 24 08:51:17 bluth k3s[11030]: time="2022-07-24T08:51:17+01:00" level=info msg="Configuring sqlite3 database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s" Jul 24 08:51:17 bluth k3s[11030]: time="2022-07-24T08:51:17+01:00" level=info msg="Configuring database table schema and indexes, this may take a moment..." Jul 24 08:51:17 bluth k3s[11030]: time="2022-07-24T08:51:17+01:00" level=info msg="Database tables and indexes are up to date" Jul 24 08:51:18 bluth k3s[11030]: time="2022-07-24T08:51:18+01:00" level=info msg="Kine available at unix://kine.sock" Jul 24 08:51:19 bluth k3s[11030]: time="2022-07-24T08:51:19+01:00" level=info msg="Reconciling bootstrap data between datastore and disk" Jul 24 08:51:19 bluth k3s[11030]: time="2022-07-24T08:51:19+01:00" level=info msg="Running kube-apiserver --advertise-address=0.0.0.0 --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/log/k3s_server_audit.log --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/mnt/nagmash/ix-applications/k3s/server/tls/temporary-certs --client-ca-file=/mnt/nagmash/ix-applications/k3s/server/tls/client-ca.crt --enable-admission-plugins=NodeRestriction,NamespaceLifecycle,ServiceAccount --etcd-servers=unix://kine.sock --feature-gates=JobTrackingWithFinalizers=true --insecure-port=0 --kubelet-certificate-authority=/mnt/nagmash/ix-applications/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/mnt/nagmash/ix-applications/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/mnt/nagmash/ix-applications/k3s/server/tls/client-kube-apiserver.key --profiling=false --proxy-client-cert-file=/mnt/nagmash/ix-applications/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/mnt/nagmash/ix-applications/k3s/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/mnt/nagmash/ix-applications/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/mnt/nagmash/ix-applications/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/mnt/nagmash/ix-applications/k3s/server/tls/service.key --service-cluster-ip-range=172.17.0.0/16 --service-node-port-range=9000-65535 --storage-backend=etcd3 --tls-cert-file=/mnt/nagmash/ix-applications/k3s/server/tls/serving-kube-apiserver.crt --tls-private-key-file=/mnt/nagmash/ix-applications/k3s/server/tls/serving-kube-apiserver.key" Jul 24 08:51:19 bluth k3s[11030]: Flag --insecure-port has been deprecated, This flag has no effect now and will be removed in v1.24. Jul 24 08:51:19 bluth k3s[11030]: I0724 08:51:19.632486 11030 server.go:593] external host was not specified, using 192.168.0.143 Jul 24 08:51:19 bluth k3s[11030]: I0724 08:51:19.633251 11030 server.go:186] Version: v1.23.5+k3s-fbfa51e5-dirty Jul 24 08:51:19 bluth k3s[11030]: time="2022-07-24T08:51:19+01:00" level=info msg="Running kube-scheduler --authentication-kubeconfig=/mnt/nagmash/ix-applications/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/mnt/nagmash/ix-applications/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --kubeconfig=/mnt/nagmash/ix-applications/k3s/server/cred/scheduler.kubeconfig --leader-elect=false --profiling=false --secure-port=10259" Jul 24 08:51:19 bluth k3s[11030]: time="2022-07-24T08:51:19+01:00" level=info msg="Waiting for API server to become available" Jul 24 08:51:19 bluth k3s[11030]: time="2022-07-24T08:51:19+01:00" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/mnt/nagmash/ix-applications/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/mnt/nagmash/ix-applications/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cluster-cidr=172.16.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/mnt/nagmash/ix-applications/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/mnt/nagmash/ix-applications/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/mnt/nagmash/ix-applications/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/mnt/nagmash/ix-applications/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/mnt/nagmash/ix-applications/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/mnt/nagmash/ix-applications/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/mnt/nagmash/ix-applications/k3s/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/mnt/nagmash/ix-applications/k3s/server/tls/server-ca.key --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/mnt/nagmash/ix-applications/k3s/server/cred/controller.kubeconfig --leader-elect=false --node-cidr-mask-size=16 --profiling=false --root-ca-file=/mnt/nagmash/ix-applications/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/mnt/nagmash/ix-applications/k3s/server/tls/service.key --terminated-pod-gc-threshold=5 --use-service-account-credentials=true" Jul 24 08:51:19 bluth k3s[11030]: time="2022-07-24T08:51:19+01:00" level=info msg="Node token is available at /mnt/nagmash/ix-applications/k3s/server/token" Jul 24 08:51:19 bluth k3s[11030]: time="2022-07-24T08:51:19+01:00" level=info msg="To join node to cluster: k3s agent -s https://192.168.0.143:6443 -t ${NODE_TOKEN}" Jul 24 08:51:19 bluth k3s[11030]: time="2022-07-24T08:51:19+01:00" level=info msg="Wrote kubeconfig /etc/rancher/k3s/k3s.yaml" Jul 24 08:51:19 bluth k3s[11030]: time="2022-07-24T08:51:19+01:00" level=info msg="Run: k3s kubectl" Jul 24 08:51:19 bluth k3s[11030]: I0724 08:51:19.772740 11030 shared_informer.go:240] Waiting for caches to sync for node_authorizer Jul 24 08:51:19 bluth k3s[11030]: I0724 08:51:19.780785 11030 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook. Jul 24 08:51:19 bluth k3s[11030]: I0724 08:51:19.780838 11030 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota. Jul 24 08:51:19 bluth k3s[11030]: I0724 08:51:19.792492 11030 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook. Jul 24 08:51:19 bluth k3s[11030]: I0724 08:51:19.792534 11030 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota. Jul 24 08:51:19 bluth k3s[11030]: time="2022-07-24T08:51:19+01:00" level=info msg="certificate CN=ix-truenas signed by CN=k3s-server-ca@1655583878: notBefore=2022-06-18 20:24:38 +0000 UTC notAfter=2023-07-24 07:51:19 +0000 UTC" Jul 24 08:51:19 bluth k3s[11030]: time="2022-07-24T08:51:19+01:00" level=info msg="certificate CN=system:node:ix-truenas,O=system:nodes signed by CN=k3s-client-ca@1655583878: notBefore=2022-06-18 20:24:38 +0000 UTC notAfter=2023-07-24 07:51:19 +0000 UTC" Jul 24 08:51:19 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T08:51:19.814+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 08:51:19 bluth k3s[11030]: time="2022-07-24T08:51:19+01:00" level=info msg="Module overlay was already loaded" Jul 24 08:51:19 bluth k3s[11030]: time="2022-07-24T08:51:19+01:00" level=info msg="Module br_netfilter was already loaded" Jul 24 08:51:19 bluth k3s[11030]: time="2022-07-24T08:51:19+01:00" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_max' to 196608" Jul 24 08:51:19 bluth k3s[11030]: time="2022-07-24T08:51:19+01:00" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400" Jul 24 08:51:19 bluth k3s[11030]: time="2022-07-24T08:51:19+01:00" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600" Jul 24 08:51:19 bluth k3s[11030]: time="2022-07-24T08:51:19+01:00" level=info msg="Connecting to proxy" url="wss://127.0.0.1:6443/v1-k3s/connect" Jul 24 08:51:19 bluth k3s[11030]: time="2022-07-24T08:51:19+01:00" level=error msg="Failed to connect to proxy" error="websocket: bad handshake" Jul 24 08:51:19 bluth k3s[11030]: time="2022-07-24T08:51:19+01:00" level=error msg="Remotedialer proxy error" error="websocket: bad handshake" Jul 24 08:51:19 bluth k3s[11030]: W0724 08:51:19.903421 11030 genericapiserver.go:538] Skipping API apiextensions.k8s.io/v1beta1 because it has no resources. Jul 24 08:51:19 bluth k3s[11030]: I0724 08:51:19.905023 11030 instance.go:274] Using reconciler: lease Jul 24 08:51:20 bluth k3s[11030]: W0724 08:51:20.238824 11030 genericapiserver.go:538] Skipping API authentication.k8s.io/v1beta1 because it has no resources. Jul 24 08:51:20 bluth k3s[11030]: W0724 08:51:20.240113 11030 genericapiserver.go:538] Skipping API authorization.k8s.io/v1beta1 because it has no resources. Jul 24 08:51:20 bluth k3s[11030]: W0724 08:51:20.253058 11030 genericapiserver.go:538] Skipping API certificates.k8s.io/v1beta1 because it has no resources. Jul 24 08:51:20 bluth k3s[11030]: W0724 08:51:20.254245 11030 genericapiserver.go:538] Skipping API coordination.k8s.io/v1beta1 because it has no resources. Jul 24 08:51:20 bluth k3s[11030]: W0724 08:51:20.258117 11030 genericapiserver.go:538] Skipping API networking.k8s.io/v1beta1 because it has no resources. Jul 24 08:51:20 bluth k3s[11030]: W0724 08:51:20.260533 11030 genericapiserver.go:538] Skipping API node.k8s.io/v1alpha1 because it has no resources. Jul 24 08:51:20 bluth k3s[11030]: W0724 08:51:20.264691 11030 genericapiserver.go:538] Skipping API rbac.authorization.k8s.io/v1beta1 because it has no resources. Jul 24 08:51:20 bluth k3s[11030]: W0724 08:51:20.264700 11030 genericapiserver.go:538] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources. Jul 24 08:51:20 bluth k3s[11030]: W0724 08:51:20.265689 11030 genericapiserver.go:538] Skipping API scheduling.k8s.io/v1beta1 because it has no resources. Jul 24 08:51:20 bluth k3s[11030]: W0724 08:51:20.265697 11030 genericapiserver.go:538] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources. Jul 24 08:51:20 bluth k3s[11030]: W0724 08:51:20.268319 11030 genericapiserver.go:538] Skipping API storage.k8s.io/v1alpha1 because it has no resources. Jul 24 08:51:20 bluth k3s[11030]: W0724 08:51:20.271082 11030 genericapiserver.go:538] Skipping API flowcontrol.apiserver.k8s.io/v1alpha1 because it has no resources. Jul 24 08:51:20 bluth k3s[11030]: W0724 08:51:20.273926 11030 genericapiserver.go:538] Skipping API apps/v1beta2 because it has no resources. Jul 24 08:51:20 bluth k3s[11030]: W0724 08:51:20.273935 11030 genericapiserver.go:538] Skipping API apps/v1beta1 because it has no resources. Jul 24 08:51:20 bluth k3s[11030]: W0724 08:51:20.275332 11030 genericapiserver.go:538] Skipping API admissionregistration.k8s.io/v1beta1 because it has no resources. Jul 24 08:51:20 bluth k3s[11030]: I0724 08:51:20.277715 11030 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook. Jul 24 08:51:20 bluth k3s[11030]: I0724 08:51:20.277724 11030 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota. Jul 24 08:51:20 bluth k3s[11030]: W0724 08:51:20.285421 11030 genericapiserver.go:538] Skipping API apiregistration.k8s.io/v1beta1 because it has no resources. Jul 24 08:51:20 bluth k3s[11030]: I0724 08:51:20.921667 11030 secure_serving.go:266] Serving securely on 127.0.0.1:6444 Jul 24 08:51:20 bluth k3s[11030]: I0724 08:51:20.921674 11030 dynamic_cafile_content.go:156] "Starting controller" name="request-header::/mnt/nagmash/ix-applications/k3s/server/tls/request-header-ca.crt" Jul 24 08:51:20 bluth k3s[11030]: I0724 08:51:20.921753 11030 tlsconfig.go:240] "Starting DynamicServingCertificateController" Jul 24 08:51:20 bluth k3s[11030]: I0724 08:51:20.921672 11030 dynamic_cafile_content.go:156] "Starting controller" name="client-ca-bundle::/mnt/nagmash/ix-applications/k3s/server/tls/client-ca.crt" Jul 24 08:51:20 bluth k3s[11030]: I0724 08:51:20.921913 11030 dynamic_serving_content.go:131] "Starting controller" name="aggregator-proxy-cert::/mnt/nagmash/ix-applications/k3s/server/tls/client-auth-proxy.crt::/mnt/nagmash/ix-applications/k3s/server/tls/client-auth-proxy.key" Jul 24 08:51:20 bluth k3s[11030]: I0724 08:51:20.921718 11030 dynamic_serving_content.go:131] "Starting controller" name="serving-cert::/mnt/nagmash/ix-applications/k3s/server/tls/serving-kube-apiserver.crt::/mnt/nagmash/ix-applications/k3s/server/tls/serving-kube-apiserver.key" Jul 24 08:51:20 bluth k3s[11030]: I0724 08:51:20.922949 11030 available_controller.go:491] Starting AvailableConditionController Jul 24 08:51:20 bluth k3s[11030]: I0724 08:51:20.922959 11030 cache.go:32] Waiting for caches to sync for AvailableConditionController controller Jul 24 08:51:20 bluth k3s[11030]: I0724 08:51:20.925825 11030 apf_controller.go:317] Starting API Priority and Fairness config controller Jul 24 08:51:20 bluth k3s[11030]: I0724 08:51:20.925857 11030 apiservice_controller.go:97] Starting APIServiceRegistrationController Jul 24 08:51:20 bluth k3s[11030]: I0724 08:51:20.925862 11030 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller Jul 24 08:51:20 bluth k3s[11030]: I0724 08:51:20.925888 11030 controller.go:83] Starting OpenAPI AggregationController Jul 24 08:51:20 bluth k3s[11030]: I0724 08:51:20.933271 11030 customresource_discovery_controller.go:209] Starting DiscoveryController Jul 24 08:51:20 bluth k3s[11030]: I0724 08:51:20.935549 11030 cluster_authentication_trust_controller.go:440] Starting cluster_authentication_trust_controller controller Jul 24 08:51:20 bluth k3s[11030]: I0724 08:51:20.935574 11030 shared_informer.go:240] Waiting for caches to sync for cluster_authentication_trust_controller Jul 24 08:51:20 bluth k3s[11030]: I0724 08:51:20.935731 11030 autoregister_controller.go:141] Starting autoregister controller Jul 24 08:51:20 bluth k3s[11030]: I0724 08:51:20.935742 11030 cache.go:32] Waiting for caches to sync for autoregister controller Jul 24 08:51:20 bluth k3s[11030]: I0724 08:51:20.935911 11030 apiapproval_controller.go:186] Starting KubernetesAPIApprovalPolicyConformantConditionController Jul 24 08:51:20 bluth k3s[11030]: I0724 08:51:20.935930 11030 establishing_controller.go:76] Starting EstablishingController Jul 24 08:51:20 bluth k3s[11030]: I0724 08:51:20.935942 11030 crd_finalizer.go:266] Starting CRDFinalizer Jul 24 08:51:20 bluth k3s[11030]: I0724 08:51:20.935968 11030 dynamic_cafile_content.go:156] "Starting controller" name="client-ca-bundle::/mnt/nagmash/ix-applications/k3s/server/tls/client-ca.crt" Jul 24 08:51:20 bluth k3s[11030]: I0724 08:51:20.936093 11030 naming_controller.go:291] Starting NamingConditionController Jul 24 08:51:20 bluth k3s[11030]: I0724 08:51:20.936162 11030 dynamic_cafile_content.go:156] "Starting controller" name="request-header::/mnt/nagmash/ix-applications/k3s/server/tls/request-header-ca.crt" Jul 24 08:51:20 bluth k3s[11030]: I0724 08:51:20.936223 11030 controller.go:85] Starting OpenAPI controller Jul 24 08:51:20 bluth k3s[11030]: I0724 08:51:20.935912 11030 nonstructuralschema_controller.go:192] Starting NonStructuralSchemaConditionController Jul 24 08:51:20 bluth k3s[11030]: I0724 08:51:20.938043 11030 crdregistration_controller.go:111] Starting crd-autoregister controller Jul 24 08:51:20 bluth k3s[11030]: I0724 08:51:20.938055 11030 shared_informer.go:240] Waiting for caches to sync for crd-autoregister Jul 24 08:51:20 bluth k3s[11030]: I0724 08:51:20.973300 11030 shared_informer.go:247] Caches are synced for node_authorizer Jul 24 08:51:21 bluth k3s[11030]: E0724 08:51:21.016492 11030 controller.go:161] Error removing old endpoints from kubernetes service: no master IPs were listed in storage, refusing to erase all endpoints for the kubernetes service Jul 24 08:51:21 bluth k3s[11030]: I0724 08:51:21.023622 11030 cache.go:39] Caches are synced for AvailableConditionController controller Jul 24 08:51:21 bluth k3s[11030]: I0724 08:51:21.026200 11030 cache.go:39] Caches are synced for APIServiceRegistrationController controller Jul 24 08:51:21 bluth k3s[11030]: I0724 08:51:21.026204 11030 apf_controller.go:322] Running API Priority and Fairness config worker Jul 24 08:51:21 bluth k3s[11030]: I0724 08:51:21.035690 11030 shared_informer.go:247] Caches are synced for cluster_authentication_trust_controller Jul 24 08:51:21 bluth k3s[11030]: I0724 08:51:21.036463 11030 cache.go:39] Caches are synced for autoregister controller Jul 24 08:51:21 bluth k3s[11030]: I0724 08:51:21.038114 11030 shared_informer.go:247] Caches are synced for crd-autoregister Jul 24 08:51:21 bluth k3s[11030]: I0724 08:51:21.921904 11030 controller.go:132] OpenAPI AggregationController: action for item : Nothing (removed from the queue). Jul 24 08:51:21 bluth k3s[11030]: I0724 08:51:21.921931 11030 controller.go:132] OpenAPI AggregationController: action for item k8s_internal_local_delegation_chain_0000000000: Nothing (removed from the queue). Jul 24 08:51:21 bluth k3s[11030]: I0724 08:51:21.938026 11030 storage_scheduling.go:109] all system priority classes are created successfully or already exist. Jul 24 08:51:22 bluth k3s[11030]: time="2022-07-24T08:51:22+01:00" level=info msg="Kube API server is now running" Jul 24 08:51:22 bluth k3s[11030]: time="2022-07-24T08:51:22+01:00" level=info msg="k3s is up and running" Jul 24 08:51:22 bluth k3s[11030]: time="2022-07-24T08:51:22+01:00" level=info msg="Applying CRD addons.k3s.cattle.io" Jul 24 08:51:22 bluth k3s[11030]: time="2022-07-24T08:51:22+01:00" level=info msg="Applying CRD helmcharts.helm.cattle.io" Jul 24 08:51:22 bluth k3s[11030]: time="2022-07-24T08:51:22+01:00" level=info msg="Applying CRD helmchartconfigs.helm.cattle.io" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Writing static file: /mnt/nagmash/ix-applications/k3s/server/static/charts/traefik-10.14.100.tgz" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Writing static file: /mnt/nagmash/ix-applications/k3s/server/static/charts/traefik-crd-10.14.100.tgz" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Writing manifest: /mnt/nagmash/ix-applications/k3s/server/manifests/rolebindings.yaml" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Writing manifest: /mnt/nagmash/ix-applications/k3s/server/manifests/coredns.yaml" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Writing manifest: /mnt/nagmash/ix-applications/k3s/server/manifests/kuberouter.yaml" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Writing manifest: /mnt/nagmash/ix-applications/k3s/server/manifests/multus-daemonset.yaml" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Writing manifest: /mnt/nagmash/ix-applications/k3s/server/manifests/zfs-operator.yaml" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Starting k3s.cattle.io/v1, Kind=Addon controller" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Creating deploy event broadcaster" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"coredns\", UID:\"25c6f7a3-77bf-4cf6-8aa6-c0826c717f90\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"435505\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyingManifest' Applying manifest at \"/mnt/nagmash/ix-applications/k3s/server/manifests/coredns.yaml\"" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Creating svccontroller event broadcaster" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Cluster dns configmap already exists" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Labels and annotations have been set successfully on node: ix-truenas" Jul 24 08:51:23 bluth k3s[11030]: I0724 08:51:23.206988 11030 controller.go:611] quota admission added evaluator for: deployments.apps Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"coredns\", UID:\"25c6f7a3-77bf-4cf6-8aa6-c0826c717f90\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"435505\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedManifest' Applied manifest at \"/mnt/nagmash/ix-applications/k3s/server/manifests/coredns.yaml\"" Jul 24 08:51:23 bluth k3s[11030]: I0724 08:51:23.213898 11030 controller.go:611] quota admission added evaluator for: addons.k3s.cattle.io Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"kuberouter\", UID:\"87a33a18-0310-41d6-b81f-d1017f1cec45\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"435508\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyingManifest' Applying manifest at \"/mnt/nagmash/ix-applications/k3s/server/manifests/kuberouter.yaml\"" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"kuberouter\", UID:\"87a33a18-0310-41d6-b81f-d1017f1cec45\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"435508\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedManifest' Applied manifest at \"/mnt/nagmash/ix-applications/k3s/server/manifests/kuberouter.yaml\"" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"multus-daemonset\", UID:\"92cc4fe6-4bbc-42b9-8489-2b1386b5edf1\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"435510\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyingManifest' Applying manifest at \"/mnt/nagmash/ix-applications/k3s/server/manifests/multus-daemonset.yaml\"" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"multus-daemonset\", UID:\"92cc4fe6-4bbc-42b9-8489-2b1386b5edf1\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"435510\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedManifest' Applied manifest at \"/mnt/nagmash/ix-applications/k3s/server/manifests/multus-daemonset.yaml\"" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"rolebindings\", UID:\"c0393f7b-ac59-43b4-aafc-8093e8b315a5\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"435513\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyingManifest' Applying manifest at \"/mnt/nagmash/ix-applications/k3s/server/manifests/rolebindings.yaml\"" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"rolebindings\", UID:\"c0393f7b-ac59-43b4-aafc-8093e8b315a5\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"435513\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedManifest' Applied manifest at \"/mnt/nagmash/ix-applications/k3s/server/manifests/rolebindings.yaml\"" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Starting helm.cattle.io/v1, Kind=HelmChart controller" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Starting helm.cattle.io/v1, Kind=HelmChartConfig controller" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"zfs-operator\", UID:\"bba54994-b1b3-45ef-b00e-c5446bea5856\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"435518\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyingManifest' Applying manifest at \"/mnt/nagmash/ix-applications/k3s/server/manifests/zfs-operator.yaml\"" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Starting apps/v1, Kind=Deployment controller" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Starting apps/v1, Kind=DaemonSet controller" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Starting rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding controller" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Starting batch/v1, Kind=Job controller" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Starting /v1, Kind=ServiceAccount controller" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Starting /v1, Kind=Pod controller" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Starting /v1, Kind=Node controller" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Starting /v1, Kind=ConfigMap controller" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Starting /v1, Kind=Service controller" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Starting /v1, Kind=Endpoints controller" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-sonarr\", Name:\"sonarr\", UID:\"b24571a2-a018-4716-ab7d-d6ab31e66e44\", APIVersion:\"v1\", ResourceVersion:\"2334145\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-sonarr/svclb-sonarr" Jul 24 08:51:23 bluth k3s[11030]: I0724 08:51:23.293046 11030 serving.go:355] Generated self-signed cert in-memory Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-prowlarr\", Name:\"prowlarr\", UID:\"bd43faf9-73db-4bcd-a678-c673a7b54098\", APIVersion:\"v1\", ResourceVersion:\"2334148\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-prowlarr/svclb-prowlarr" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-qbittorrent\", Name:\"qbittorrent-torrent\", UID:\"29b172f8-d799-4e0c-aac7-78ec5af5c01b\", APIVersion:\"v1\", ResourceVersion:\"2334164\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-qbittorrent/svclb-qbittorrent-torrent" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-esphome\", Name:\"esphome\", UID:\"9d7f07bb-0ed3-4a9b-b9b3-ce707e119669\", APIVersion:\"v1\", ResourceVersion:\"2334230\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-esphome/svclb-esphome" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-qbittorrent\", Name:\"qbittorrent\", UID:\"728a4977-c72e-46a0-96f2-a9279b009171\", APIVersion:\"v1\", ResourceVersion:\"2334264\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-qbittorrent/svclb-qbittorrent" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-tdarr\", Name:\"tdarr-comm\", UID:\"c3b33312-b11b-4e16-915e-205202cc0a2d\", APIVersion:\"v1\", ResourceVersion:\"2504455\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-tdarr/svclb-tdarr-comm" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-tdarr\", Name:\"tdarr\", UID:\"90709eb9-3d84-4b91-b6ab-73ece8ffcb86\", APIVersion:\"v1\", ResourceVersion:\"2504489\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-tdarr/svclb-tdarr" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-qbittorrent\", Name:\"qbittorrent-torrentudp\", UID:\"1d92e74f-b912-46f1-b780-c740f80bb9ca\", APIVersion:\"v1\", ResourceVersion:\"2504584\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-qbittorrent/svclb-qbittorrent-torrentudp" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-radarr\", Name:\"radarr\", UID:\"2ceba47d-fa27-462f-9816-af4352781795\", APIVersion:\"v1\", ResourceVersion:\"2504670\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-radarr/svclb-radarr" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-qbittorrent\", Name:\"qbittorrent\", UID:\"728a4977-c72e-46a0-96f2-a9279b009171\", APIVersion:\"v1\", ResourceVersion:\"2334264\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-qbittorrent/svclb-qbittorrent" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-qbittorrent\", Name:\"qbittorrent-torrent\", UID:\"29b172f8-d799-4e0c-aac7-78ec5af5c01b\", APIVersion:\"v1\", ResourceVersion:\"2334164\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-qbittorrent/svclb-qbittorrent-torrent" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-esphome\", Name:\"esphome\", UID:\"9d7f07bb-0ed3-4a9b-b9b3-ce707e119669\", APIVersion:\"v1\", ResourceVersion:\"2334230\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-esphome/svclb-esphome" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-prowlarr\", Name:\"prowlarr\", UID:\"bd43faf9-73db-4bcd-a678-c673a7b54098\", APIVersion:\"v1\", ResourceVersion:\"2334148\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-prowlarr/svclb-prowlarr" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-sonarr\", Name:\"sonarr\", UID:\"b24571a2-a018-4716-ab7d-d6ab31e66e44\", APIVersion:\"v1\", ResourceVersion:\"2334145\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-sonarr/svclb-sonarr" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-tdarr\", Name:\"tdarr-comm\", UID:\"c3b33312-b11b-4e16-915e-205202cc0a2d\", APIVersion:\"v1\", ResourceVersion:\"2504455\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-tdarr/svclb-tdarr-comm" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-tdarr\", Name:\"tdarr\", UID:\"90709eb9-3d84-4b91-b6ab-73ece8ffcb86\", APIVersion:\"v1\", ResourceVersion:\"2504489\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-tdarr/svclb-tdarr" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-qbittorrent\", Name:\"qbittorrent-torrentudp\", UID:\"1d92e74f-b912-46f1-b780-c740f80bb9ca\", APIVersion:\"v1\", ResourceVersion:\"2504584\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-qbittorrent/svclb-qbittorrent-torrentudp" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-radarr\", Name:\"radarr\", UID:\"2ceba47d-fa27-462f-9816-af4352781795\", APIVersion:\"v1\", ResourceVersion:\"2504670\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-radarr/svclb-radarr" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"zfs-operator\", UID:\"bba54994-b1b3-45ef-b00e-c5446bea5856\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"435518\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedManifest' Applied manifest at \"/mnt/nagmash/ix-applications/k3s/server/manifests/zfs-operator.yaml\"" Jul 24 08:51:23 bluth k3s[11030]: I0724 08:51:23.433375 11030 serving.go:355] Generated self-signed cert in-memory Jul 24 08:51:23 bluth k3s[11030]: I0724 08:51:23.579891 11030 controllermanager.go:196] Version: v1.23.5+k3s-fbfa51e5-dirty Jul 24 08:51:23 bluth k3s[11030]: I0724 08:51:23.581701 11030 secure_serving.go:200] Serving securely on 127.0.0.1:10257 Jul 24 08:51:23 bluth k3s[11030]: I0724 08:51:23.581763 11030 configmap_cafile_content.go:201] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" Jul 24 08:51:23 bluth k3s[11030]: I0724 08:51:23.581772 11030 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file Jul 24 08:51:23 bluth k3s[11030]: I0724 08:51:23.581799 11030 tlsconfig.go:240] "Starting DynamicServingCertificateController" Jul 24 08:51:23 bluth k3s[11030]: I0724 08:51:23.581806 11030 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController Jul 24 08:51:23 bluth k3s[11030]: I0724 08:51:23.581814 11030 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController Jul 24 08:51:23 bluth k3s[11030]: I0724 08:51:23.581848 11030 configmap_cafile_content.go:201] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" Jul 24 08:51:23 bluth k3s[11030]: I0724 08:51:23.581854 11030 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file Jul 24 08:51:23 bluth k3s[11030]: I0724 08:51:23.682151 11030 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController Jul 24 08:51:23 bluth k3s[11030]: I0724 08:51:23.682151 11030 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file Jul 24 08:51:23 bluth k3s[11030]: I0724 08:51:23.682228 11030 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Starting /v1, Kind=Secret controller" Jul 24 08:51:23 bluth k3s[11030]: time="2022-07-24T08:51:23+01:00" level=info msg="Updating TLS secret for k3s-serving (count: 11): map[listener.cattle.io/cn-0.0.0.0:0.0.0.0 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-172.17.0.1:172.17.0.1 listener.cattle.io/cn-192.168.0.143:192.168.0.143 listener.cattle.io/cn-__1-f16284:::1 listener.cattle.io/cn-ix-truenas:ix-truenas listener.cattle.io/cn-kubernetes:kubernetes listener.cattle.io/cn-kubernetes.default:kubernetes.default listener.cattle.io/cn-kubernetes.default.svc:kubernetes.default.svc listener.cattle.io/cn-kubernetes.default.svc.cluster.local:kubernetes.default.svc.cluster.local listener.cattle.io/cn-localhost:localhost listener.cattle.io/fingerprint:SHA1=74C8245641D6058D0CB28FD7B1211C2824BE8CDE]" Jul 24 08:51:23 bluth k3s[11030]: I0724 08:51:23.744622 11030 server.go:139] "Starting Kubernetes Scheduler" version="v1.23.5+k3s-fbfa51e5-dirty" Jul 24 08:51:23 bluth k3s[11030]: I0724 08:51:23.752988 11030 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController Jul 24 08:51:23 bluth k3s[11030]: I0724 08:51:23.753036 11030 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController Jul 24 08:51:23 bluth k3s[11030]: I0724 08:51:23.753062 11030 configmap_cafile_content.go:201] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" Jul 24 08:51:23 bluth k3s[11030]: I0724 08:51:23.753072 11030 configmap_cafile_content.go:201] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" Jul 24 08:51:23 bluth k3s[11030]: I0724 08:51:23.753101 11030 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file Jul 24 08:51:23 bluth k3s[11030]: I0724 08:51:23.753123 11030 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file Jul 24 08:51:23 bluth k3s[11030]: I0724 08:51:23.753406 11030 secure_serving.go:200] Serving securely on 127.0.0.1:10259 Jul 24 08:51:23 bluth k3s[11030]: I0724 08:51:23.753478 11030 tlsconfig.go:240] "Starting DynamicServingCertificateController" Jul 24 08:51:23 bluth k3s[11030]: I0724 08:51:23.853791 11030 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController Jul 24 08:51:23 bluth k3s[11030]: I0724 08:51:23.853872 11030 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file Jul 24 08:51:23 bluth k3s[11030]: I0724 08:51:23.853976 11030 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.045037 11030 shared_informer.go:240] Waiting for caches to sync for tokens Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.056301 11030 controllermanager.go:605] Started "deployment" Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.057717 11030 deployment_controller.go:153] "Starting controller" controller="deployment" Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.057763 11030 shared_informer.go:240] Waiting for caches to sync for deployment Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.063144 11030 controllermanager.go:605] Started "csrcleaner" Jul 24 08:51:24 bluth k3s[11030]: W0724 08:51:24.063189 11030 controllermanager.go:570] "tokencleaner" is disabled Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.065957 11030 cleaner.go:82] Starting CSR cleaner controller Jul 24 08:51:24 bluth k3s[11030]: E0724 08:51:24.069377 11030 core.go:92] Failed to start service controller: WARNING: no cloud provider provided, services of type LoadBalancer will fail Jul 24 08:51:24 bluth k3s[11030]: W0724 08:51:24.069413 11030 controllermanager.go:583] Skipping "service" Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.073113 11030 controllermanager.go:605] Started "endpointslice" Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.074825 11030 endpointslice_controller.go:257] Starting endpoint slice controller Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.074856 11030 shared_informer.go:240] Waiting for caches to sync for endpoint_slice Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.093911 11030 controllermanager.go:605] Started "namespace" Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.094701 11030 namespace_controller.go:200] Starting namespace controller Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.094716 11030 shared_informer.go:240] Waiting for caches to sync for namespace Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.095774 11030 node_lifecycle_controller.go:377] Sending events to api server. Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.096035 11030 taint_manager.go:163] "Sending events to api server" Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.096087 11030 node_lifecycle_controller.go:505] Controller will reconcile labels. Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.096115 11030 controllermanager.go:605] Started "nodelifecycle" Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.096245 11030 node_lifecycle_controller.go:539] Starting node controller Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.096255 11030 shared_informer.go:240] Waiting for caches to sync for taint Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.097621 11030 controllermanager.go:605] Started "endpoint" Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.097711 11030 endpoints_controller.go:193] Starting endpoint controller Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.097723 11030 shared_informer.go:240] Waiting for caches to sync for endpoint Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.098966 11030 controllermanager.go:605] Started "clusterrole-aggregation" Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.099058 11030 clusterroleaggregation_controller.go:194] Starting ClusterRoleAggregator Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.099065 11030 shared_informer.go:240] Waiting for caches to sync for ClusterRoleAggregator Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.100579 11030 controllermanager.go:605] Started "ephemeral-volume" Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.100692 11030 controller.go:170] Starting ephemeral volume controller Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.100703 11030 shared_informer.go:240] Waiting for caches to sync for ephemeral Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.102268 11030 controllermanager.go:605] Started "job" Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.102313 11030 job_controller.go:186] Starting job controller Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.102330 11030 shared_informer.go:240] Waiting for caches to sync for job Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.103978 11030 controllermanager.go:605] Started "root-ca-cert-publisher" Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.104069 11030 publisher.go:107] Starting root CA certificate configmap publisher Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.104077 11030 shared_informer.go:240] Waiting for caches to sync for crt configmap Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.105400 11030 controllermanager.go:605] Started "pv-protection" Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.105486 11030 pv_protection_controller.go:79] Starting PV protection controller Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.105492 11030 shared_informer.go:240] Waiting for caches to sync for PV protection Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.106754 11030 controllermanager.go:605] Started "cronjob" Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.106843 11030 cronjob_controllerv2.go:132] "Starting cronjob controller v2" Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.106853 11030 shared_informer.go:240] Waiting for caches to sync for cronjob Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.108261 11030 certificate_controller.go:118] Starting certificate controller "csrsigning-kubelet-serving" Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.108271 11030 shared_informer.go:240] Waiting for caches to sync for certificate-csrsigning-kubelet-serving Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.108287 11030 dynamic_serving_content.go:131] "Starting controller" name="csr-controller::/mnt/nagmash/ix-applications/k3s/server/tls/server-ca.crt::/mnt/nagmash/ix-applications/k3s/server/tls/server-ca.key" Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.108381 11030 certificate_controller.go:118] Starting certificate controller "csrsigning-kubelet-client" Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.108391 11030 shared_informer.go:240] Waiting for caches to sync for certificate-csrsigning-kubelet-client Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.108405 11030 dynamic_serving_content.go:131] "Starting controller" name="csr-controller::/mnt/nagmash/ix-applications/k3s/server/tls/client-ca.crt::/mnt/nagmash/ix-applications/k3s/server/tls/client-ca.key" Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.108530 11030 certificate_controller.go:118] Starting certificate controller "csrsigning-kube-apiserver-client" Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.108537 11030 shared_informer.go:240] Waiting for caches to sync for certificate-csrsigning-kube-apiserver-client Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.108549 11030 dynamic_serving_content.go:131] "Starting controller" name="csr-controller::/mnt/nagmash/ix-applications/k3s/server/tls/client-ca.crt::/mnt/nagmash/ix-applications/k3s/server/tls/client-ca.key" Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.108592 11030 controllermanager.go:605] Started "csrsigning" Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.108653 11030 certificate_controller.go:118] Starting certificate controller "csrsigning-legacy-unknown" Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.108660 11030 shared_informer.go:240] Waiting for caches to sync for certificate-csrsigning-legacy-unknown Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.108674 11030 dynamic_serving_content.go:131] "Starting controller" name="csr-controller::/mnt/nagmash/ix-applications/k3s/server/tls/server-ca.crt::/mnt/nagmash/ix-applications/k3s/server/tls/server-ca.key" Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.109751 11030 node_ipam_controller.go:91] Sending events to api server. Jul 24 08:51:24 bluth k3s[11030]: I0724 08:51:24.145194 11030 shared_informer.go:247] Caches are synced for tokens Jul 24 08:51:24 bluth k3s[11030]: time="2022-07-24T08:51:24+01:00" level=info msg="Connecting to proxy" url="wss://127.0.0.1:6443/v1-k3s/connect" Jul 24 08:51:24 bluth k3s[11030]: time="2022-07-24T08:51:24+01:00" level=info msg="Handling backend connection request [ix-truenas]" Jul 24 08:51:24 bluth k3s[11030]: time="2022-07-24T08:51:24+01:00" level=info msg="Running kubelet --address=0.0.0.0 --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=cgroupfs --client-ca-file=/mnt/nagmash/ix-applications/k3s/agent/client-ca.crt --cluster-dns=172.17.0.10 --cluster-domain=cluster.local --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ix-truenas --kubeconfig=/mnt/nagmash/ix-applications/k3s/agent/kubelet.kubeconfig --max-pods=250 --network-plugin=cni --node-ip=0.0.0.0 --node-labels= --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/mnt/nagmash/ix-applications/k3s/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/etc/resolv.conf --tls-cert-file=/mnt/nagmash/ix-applications/k3s/agent/serving-kubelet.crt --tls-private-key-file=/mnt/nagmash/ix-applications/k3s/agent/serving-kubelet.key" Jul 24 08:51:24 bluth k3s[11030]: Flag --network-plugin has been deprecated, will be removed along with dockershim. Jul 24 08:51:24 bluth k3s[11030]: time="2022-07-24T08:51:24+01:00" level=info msg="labels have already set on node: ix-truenas" Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.060259 11030 server.go:442] "Kubelet version" kubeletVersion="v1.23.5+k3s-fbfa51e5-dirty" Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.064606 11030 dynamic_cafile_content.go:156] "Starting controller" name="client-ca-bundle::/mnt/nagmash/ix-applications/k3s/agent/client-ca.crt" Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.270724 11030 server.go:693] "--cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to /" Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.273262 11030 container_manager_linux.go:281] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.273664 11030 container_manager_linux.go:286] "Creating Container Manager object based on Node Config" nodeConfig={RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:docker CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:true NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: ReservedSystemCPUs: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:nodefs.available Operator:LessThan Value:{Quantity: Percentage:0.05} GracePeriod:0s MinReclaim:} {Signal:imagefs.available Operator:LessThan Value:{Quantity: Percentage:0.05} GracePeriod:0s MinReclaim:}]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalCPUManagerPolicyOptions:map[] ExperimentalTopologyManagerScope:container ExperimentalCPUManagerReconcilePeriod:10s ExperimentalMemoryManagerPolicy:None ExperimentalMemoryManagerReservedMemory:[] ExperimentalPodPidsLimit:-1 EnforceCPULimits:true CPUCFSQuotaPeriod:100ms ExperimentalTopologyManagerPolicy:none} Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.277302 11030 topology_manager.go:133] "Creating topology manager with policy per scope" topologyPolicyName="none" topologyScopeName="container" Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.277369 11030 container_manager_linux.go:321] "Creating device plugin manager" devicePluginEnabled=true Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.337922 11030 state_mem.go:36] "Initialized new in-memory state store" Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.339978 11030 kubelet.go:313] "Using dockershim is deprecated, please consider using a full-fledged CRI implementation" Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.353068 11030 client.go:80] "Connecting to docker on the dockerEndpoint" endpoint="unix:///var/run/docker.sock" Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.353164 11030 client.go:99] "Start docker client with request timeout" timeout="2m0s" Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.373722 11030 docker_service.go:571] "Hairpin mode is set but kubenet is not enabled, falling back to HairpinVeth" hairpinMode=promiscuous-bridge Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.374237 11030 docker_service.go:243] "Hairpin mode is set" hairpinMode=hairpin-veth Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.375157 11030 cni.go:240] "Unable to update cni config" err="no networks found in /etc/cni/net.d" Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.409834 11030 cni.go:240] "Unable to update cni config" err="no networks found in /etc/cni/net.d" Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.410716 11030 docker_service.go:258] "Docker cri networking managed by the network plugin" networkPluginName="cni" Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.411120 11030 cni.go:240] "Unable to update cni config" err="no networks found in /etc/cni/net.d" Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.442828 11030 docker_service.go:264] "Docker Info" dockerInfo=&{ID:YAXQ:ZJH6:Z4VR:I76D:4GH3:QKGH:KTLZ:KMKD:QRAV:LABJ:PIKE:XNCR Containers:93 ContainersRunning:40 ContainersPaused:0 ContainersStopped:53 Images:49 Driver:zfs DriverStatus:[[Zpool nagmash] [Zpool Health ONLINE] [Parent Dataset nagmash/ix-applications/docker] [Space Used By Parent 12247334176] [Space Available 1934099693328] [Parent Quota no] [Compression lz4]] SystemStatus:[] Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:[] Log:[awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:true KernelMemoryTCP:true CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6tables:true Debug:false NFd:23 OomKillDisable:true NGoroutines:42 SystemTime:2022-07-24T08:51:25.41275102+01:00 LoggingDriver:json-file CgroupDriver:cgroupfs CgroupVersion:1 NEventsListener:0 KernelVersion:5.10.120+truenas OperatingSystem:Debian GNU/Linux 11 (bullseye) OSVersion:11 OSType:linux Architecture:x86_64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:0xc0048202a0 NCPU:6 MemTotal:16572170240 GenericResources:[] DockerRootDir:/mnt/nagmash/ix-applications/docker HTTPProxy: HTTPSProxy: NoProxy: Name:bluth Labels:[] ExperimentalBuild:false ServerVersion:dev ClusterStore: ClusterAdvertise: Runtimes:map[io.containerd.runc.v2:{Path:runc Args:[] Shim:} io.containerd.runtime.v1.linux:{Path:runc Args:[] Shim:} runc:{Path:runc Args:[] Shim:}] DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:[] Nodes:0 Managers:0 Cluster: Warnings:[]} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:7b11cfaabd73bb80907dd23182b9347b4245eb5d Expected:7b11cfaabd73bb80907dd23182b9347b4245eb5d} RuncCommit:{ID:v1.0.2-0-g52b36a2 Expected:v1.0.2-0-g52b36a2} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=apparmor] ProductLicense: DefaultAddressPools:[] Warnings:[]} Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.442883 11030 docker_service.go:279] "Setting cgroupDriver" cgroupDriver="cgroupfs" Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.563584 11030 kubelet.go:416] "Attempting to sync node with API server" Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.564159 11030 kubelet.go:278] "Adding static pod path" path="/mnt/nagmash/ix-applications/k3s/agent/pod-manifests" Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.564907 11030 kubelet.go:289] "Adding apiserver pod source" Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.565423 11030 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.585808 11030 kuberuntime_manager.go:249] "Container runtime initialized" containerRuntime="docker" version="dev" apiVersion="1.41.0" Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.589300 11030 server.go:1231] "Started kubelet" Jul 24 08:51:25 bluth k3s[11030]: E0724 08:51:25.589694 11030 kubelet.go:1351] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.590462 11030 server.go:150] "Starting to listen" address="0.0.0.0" port=10250 Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.592740 11030 server.go:410] "Adding debug handlers to kubelet server" Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.598649 11030 scope.go:110] "RemoveContainer" containerID="701697deca9bf7b8e1f4e7db07fd50948523f662557bda540a935e84cafcb74d" Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.615029 11030 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.615294 11030 volume_manager.go:291] "Starting Kubelet Volume Manager" Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.615443 11030 desired_state_of_world_populator.go:147] "Desired state populator starts to run" Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.619058 11030 controller.go:611] quota admission added evaluator for: leases.coordination.k8s.io Jul 24 08:51:25 bluth k3s[11030]: E0724 08:51:25.626154 11030 kubelet.go:2347] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.716126 11030 kuberuntime_manager.go:1105] "Updating runtime config through cri with podcidr" CIDR="172.16.0.0/16" Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.716432 11030 docker_service.go:364] "Docker cri received runtime config" runtimeConfig="&RuntimeConfig{NetworkConfig:&NetworkConfig{PodCidr:172.16.0.0/16,},}" Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.716696 11030 kubelet_network.go:76] "Updating Pod CIDR" originalPodCIDR="" newPodCIDR="172.16.0.0/16" Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.723085 11030 kubelet_node_status.go:70] "Attempting to register node" node="ix-truenas" Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.723847 11030 kubelet_network_linux.go:57] "Initialized protocol iptables rules." protocol=IPv4 Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.728784 11030 cpu_manager.go:213] "Starting CPU manager" policy="none" Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.728800 11030 cpu_manager.go:214] "Reconciling" reconcilePeriod="10s" Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.728820 11030 state_mem.go:36] "Initialized new in-memory state store" Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.728991 11030 kubelet_node_status.go:108] "Node was previously registered" node="ix-truenas" Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.741881 11030 state_mem.go:88] "Updated default CPUSet" cpuSet="" Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.741897 11030 state_mem.go:96] "Updated CPUSet assignments" assignments=map[] Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.741902 11030 kubelet_network_linux.go:57] "Initialized protocol iptables rules." protocol=IPv6 Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.741910 11030 policy_none.go:49] "None policy: Start" Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.741994 11030 status_manager.go:159] "Starting to sync pod status with apiserver" Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.742012 11030 kubelet.go:1977] "Starting kubelet main sync loop" Jul 24 08:51:25 bluth k3s[11030]: E0724 08:51:25.742057 11030 kubelet.go:2001] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.750237 11030 memory_manager.go:168] "Starting memorymanager" policy="None" Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.750276 11030 state_mem.go:35] "Initializing new in-memory state store" Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.750424 11030 state_mem.go:75] "Updated machine memory state" Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.752240 11030 docker_sandbox.go:402] "Failed to read pod IP from plugin/docker" err="networkPlugin cni failed on the status hook for pod \"status-html-ix-chart-765b966694-9clbp_ix-status-html\": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container \"d2f132034b94577c4f38f3822a64e2609c355bbe34a9181ea5e259031c0ac943\"" Jul 24 08:51:25 bluth k3s[11030]: E0724 08:51:25.842550 11030 kubelet.go:2001] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.961291 11030 kubelet_node_status.go:73] "Successfully registered node" node="ix-truenas" Jul 24 08:51:25 bluth k3s[11030]: I0724 08:51:25.985206 11030 setters.go:578] "Node became not ready" node="ix-truenas" condition={Type:Ready Status:False LastHeartbeatTime:2022-07-24 08:51:25.985118525 +0100 IST m=+12.472975432 LastTransitionTime:2022-07-24 08:51:25.985118525 +0100 IST m=+12.472975432 Reason:KubeletNotReady Message:[container runtime status check may not have completed yet, container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized]} Jul 24 08:51:26 bluth k3s[11030]: E0724 08:51:26.042730 11030 kubelet.go:2001] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Jul 24 08:51:26 bluth k3s[11030]: I0724 08:51:26.072542 11030 docker_sandbox.go:402] "Failed to read pod IP from plugin/docker" err="networkPlugin cni failed on the status hook for pod \"tdarr-746d65b466-b7whb_ix-tdarr\": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container \"6e77c264156395d54133c0402bc6e1ad7bbef3dc70443cd759055f59551884ee\"" Jul 24 08:51:26 bluth k3s[11030]: E0724 08:51:26.443615 11030 kubelet.go:2001] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Jul 24 08:51:26 bluth k3s[11030]: I0724 08:51:26.569863 11030 apiserver.go:52] "Watching apiserver" Jul 24 08:51:26 bluth k3s[11030]: I0724 08:51:26.814877 11030 plugin_manager.go:114] "Starting Kubelet Plugin Manager" Jul 24 08:51:26 bluth k3s[11030]: E0724 08:51:26.862159 11030 kubelet.go:2347] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" Jul 24 08:51:26 bluth k3s[11030]: I0724 08:51:26.989750 11030 docker_sandbox.go:402] "Failed to read pod IP from plugin/docker" err="networkPlugin cni failed on the status hook for pod \"svclb-qbittorrent-ldv82_ix-qbittorrent\": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container \"bf4c9c440473cecdcd094a4ebeea14778d786c81fa9f21f1b69c814963677f8f\"" Jul 24 08:51:27 bluth k3s[11030]: I0724 08:51:27.142875 11030 docker_sandbox.go:402] "Failed to read pod IP from plugin/docker" err="networkPlugin cni failed on the status hook for pod \"svclb-qbittorrent-torrent-8qgf8_ix-qbittorrent\": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container \"b6e3949c474adcffbc57d4c83242acb2528215cdb92ccaf02d777d568775734d\"" Jul 24 08:51:27 bluth k3s[11030]: I0724 08:51:27.146065 11030 docker_sandbox.go:402] "Failed to read pod IP from plugin/docker" err="networkPlugin cni failed on the status hook for pod \"svclb-qbittorrent-torrent-8qgf8_ix-qbittorrent\": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container \"14a15e5a147c12eddc4317a17434a059afa99b54bda399bd2d82ff6a5e07e9be\"" Jul 24 08:51:27 bluth k3s[11030]: I0724 08:51:27.245174 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="d2f132034b94577c4f38f3822a64e2609c355bbe34a9181ea5e259031c0ac943" Jul 24 08:51:27 bluth k3s[11030]: I0724 08:51:27.245432 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:51:27 bluth k3s[11030]: I0724 08:51:27.247442 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:51:27 bluth k3s[11030]: I0724 08:51:27.247748 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:51:27 bluth k3s[11030]: I0724 08:51:27.247963 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:51:27 bluth k3s[11030]: I0724 08:51:27.248114 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:51:27 bluth k3s[11030]: I0724 08:51:27.248262 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:51:27 bluth k3s[11030]: I0724 08:51:27.248549 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:51:27 bluth k3s[11030]: I0724 08:51:27.249003 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:51:27 bluth k3s[11030]: I0724 08:51:27.249420 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:51:27 bluth k3s[11030]: I0724 08:51:27.249822 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:51:27 bluth k3s[11030]: I0724 08:51:27.250276 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:51:27 bluth k3s[11030]: I0724 08:51:27.250802 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:51:27 bluth k3s[11030]: E0724 08:51:27.251644 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-qbittorrent/svclb-qbittorrent-ldv82" podUID=ad6ed51e-d3e1-430c-9bad-528f4ae492c5 Jul 24 08:51:27 bluth k3s[11030]: I0724 08:51:27.251646 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:51:27 bluth k3s[11030]: I0724 08:51:27.252091 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:51:27 bluth k3s[11030]: I0724 08:51:27.252586 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:51:27 bluth k3s[11030]: I0724 08:51:27.252826 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:51:27 bluth k3s[11030]: I0724 08:51:27.253182 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:51:27 bluth k3s[11030]: I0724 08:51:27.253912 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:51:27 bluth k3s[11030]: I0724 08:51:27.254241 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:51:27 bluth k3s[11030]: I0724 08:51:27.254564 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:51:27 bluth k3s[11030]: I0724 08:51:27.254792 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:51:27 bluth k3s[11030]: I0724 08:51:27.255182 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:51:27 bluth k3s[11030]: I0724 08:51:27.255472 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:51:27 bluth k3s[11030]: E0724 08:51:27.255771 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-tdarr/tdarr-746d65b466-b7whb" podUID=6e3e9791-9bb3-487f-ae50-048fefbef8be Jul 24 08:51:27 bluth k3s[11030]: I0724 08:51:27.255793 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:51:27 bluth k3s[11030]: I0724 08:51:27.256232 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:51:27 bluth k3s[11030]: I0724 08:51:27.256651 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:51:27 bluth k3s[11030]: I0724 08:51:27.257175 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="6e77c264156395d54133c0402bc6e1ad7bbef3dc70443cd759055f59551884ee" Jul 24 08:51:27 bluth k3s[11030]: I0724 08:51:27.257306 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="07dc78644c5b8a54aa576c34788ff45e78b00c27a12f5ab9b51a15d04483dedd" Jul 24 08:51:27 bluth k3s[11030]: I0724 08:51:27.257338 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="b2b3302d611f848ce57cb6b9a7c80e848963117806184965663832bad8e09718" Jul 24 08:51:27 bluth k3s[11030]: I0724 08:51:27.257374 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="bf4c9c440473cecdcd094a4ebeea14778d786c81fa9f21f1b69c814963677f8f" Jul 24 08:51:27 bluth k3s[11030]: I0724 08:51:27.257437 11030 scope.go:110] "RemoveContainer" containerID="0cc34af722b67d4e3567a59268d634aa0b99635c2e9bbaaa83c0ee28b84d7458" Jul 24 08:51:27 bluth k3s[11030]: I0724 08:51:27.448050 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="b6e3949c474adcffbc57d4c83242acb2528215cdb92ccaf02d777d568775734d" Jul 24 08:51:27 bluth k3s[11030]: I0724 08:51:27.449514 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="14a15e5a147c12eddc4317a17434a059afa99b54bda399bd2d82ff6a5e07e9be" Jul 24 08:51:27 bluth k3s[11030]: E0724 08:51:27.450386 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-qbittorrent/svclb-qbittorrent-torrent-8qgf8" podUID=19ef4922-0714-452c-b8ba-e3c955cabe50 Jul 24 08:51:27 bluth k3s[11030]: I0724 08:51:27.526744 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-root\" (UniqueName: \"kubernetes.io/host-path/d312e744-054d-4a0d-8f39-46a83c98c808-host-root\") pod \"openebs-zfs-node-bf7kk\" (UID: \"d312e744-054d-4a0d-8f39-46a83c98c808\") " pod="kube-system/openebs-zfs-node-bf7kk" Jul 24 08:51:27 bluth k3s[11030]: I0724 08:51:27.526807 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d312e744-054d-4a0d-8f39-46a83c98c808-registration-dir\") pod \"openebs-zfs-node-bf7kk\" (UID: \"d312e744-054d-4a0d-8f39-46a83c98c808\") " pod="kube-system/openebs-zfs-node-bf7kk" Jul 24 08:51:27 bluth k3s[11030]: I0724 08:51:27.526822 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-dir\" (UniqueName: \"kubernetes.io/host-path/d312e744-054d-4a0d-8f39-46a83c98c808-plugin-dir\") pod \"openebs-zfs-node-bf7kk\" (UID: \"d312e744-054d-4a0d-8f39-46a83c98c808\") " pod="kube-system/openebs-zfs-node-bf7kk" Jul 24 08:51:27 bluth k3s[11030]: I0724 08:51:27.526839 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pods-mount-dir\" (UniqueName: \"kubernetes.io/host-path/d312e744-054d-4a0d-8f39-46a83c98c808-pods-mount-dir\") pod \"openebs-zfs-node-bf7kk\" (UID: \"d312e744-054d-4a0d-8f39-46a83c98c808\") " pod="kube-system/openebs-zfs-node-bf7kk" Jul 24 08:51:27 bluth k3s[11030]: I0724 08:51:27.526855 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vv85\" (UniqueName: \"kubernetes.io/projected/d312e744-054d-4a0d-8f39-46a83c98c808-kube-api-access-8vv85\") pod \"openebs-zfs-node-bf7kk\" (UID: \"d312e744-054d-4a0d-8f39-46a83c98c808\") " pod="kube-system/openebs-zfs-node-bf7kk" Jul 24 08:51:27 bluth k3s[11030]: I0724 08:51:27.526871 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d312e744-054d-4a0d-8f39-46a83c98c808-device-dir\") pod \"openebs-zfs-node-bf7kk\" (UID: \"d312e744-054d-4a0d-8f39-46a83c98c808\") " pod="kube-system/openebs-zfs-node-bf7kk" Jul 24 08:51:27 bluth k3s[11030]: I0724 08:51:27.526884 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"encr-keys\" (UniqueName: \"kubernetes.io/host-path/d312e744-054d-4a0d-8f39-46a83c98c808-encr-keys\") pod \"openebs-zfs-node-bf7kk\" (UID: \"d312e744-054d-4a0d-8f39-46a83c98c808\") " pod="kube-system/openebs-zfs-node-bf7kk" Jul 24 08:51:27 bluth k3s[11030]: I0724 08:51:27.526899 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"chroot-zfs\" (UniqueName: \"kubernetes.io/configmap/d312e744-054d-4a0d-8f39-46a83c98c808-chroot-zfs\") pod \"openebs-zfs-node-bf7kk\" (UID: \"d312e744-054d-4a0d-8f39-46a83c98c808\") " pod="kube-system/openebs-zfs-node-bf7kk" Jul 24 08:51:27 bluth k3s[11030]: time="2022-07-24T08:51:27+01:00" level=info msg="Stopped tunnel to 127.0.0.1:6443" Jul 24 08:51:27 bluth k3s[11030]: time="2022-07-24T08:51:27+01:00" level=info msg="Proxy done" err="context canceled" url="wss://127.0.0.1:6443/v1-k3s/connect" Jul 24 08:51:27 bluth k3s[11030]: time="2022-07-24T08:51:27+01:00" level=info msg="Connecting to proxy" url="wss://192.168.0.143:6443/v1-k3s/connect" Jul 24 08:51:27 bluth k3s[11030]: time="2022-07-24T08:51:27+01:00" level=info msg="error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF" Jul 24 08:51:27 bluth k3s[11030]: time="2022-07-24T08:51:27+01:00" level=info msg="Handling backend connection request [ix-truenas]" Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.033946 11030 docker_sandbox.go:402] "Failed to read pod IP from plugin/docker" err="networkPlugin cni failed on the status hook for pod \"openebs-zfs-controller-0_kube-system\": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container \"b6d43670deb9d67684a6743dad54c0dd6c4e7c295084b0ac48903cf764b79c20\"" Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.034687 11030 docker_sandbox.go:402] "Failed to read pod IP from plugin/docker" err="networkPlugin cni failed on the status hook for pod \"openebs-zfs-controller-0_kube-system\": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container \"5364fb99bc658579081df87e1915ed88fda1de41488a83ba9c0154f4e691952d\"" Jul 24 08:51:28 bluth k3s[11030]: E0724 08:51:28.047396 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=1 Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.047444 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="26ea2c00c14f73e68ef0933456e8d1ad9ba1098e757a89d762d4817b7d02cc36" Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.047452 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="5c5e061c6867c79411e60eb61fb274c7ed28ad040f29326b5ff68f4aa8efce2b" Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.138629 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="b6d43670deb9d67684a6743dad54c0dd6c4e7c295084b0ac48903cf764b79c20" Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.138646 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="5364fb99bc658579081df87e1915ed88fda1de41488a83ba9c0154f4e691952d" Jul 24 08:51:28 bluth k3s[11030]: E0724 08:51:28.138756 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="kube-system/openebs-zfs-controller-0" podUID=508c7225-82e1-4311-9b86-6a6ad8bf3740 Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.140792 11030 docker_sandbox.go:402] "Failed to read pod IP from plugin/docker" err="networkPlugin cni failed on the status hook for pod \"svclb-esphome-q4lrj_ix-esphome\": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container \"69a8e880a3b2d17ecdaca01f72544c5de9832603ac3c68a4a4b668f13be89212\"" Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.141504 11030 docker_sandbox.go:402] "Failed to read pod IP from plugin/docker" err="networkPlugin cni failed on the status hook for pod \"svclb-esphome-q4lrj_ix-esphome\": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container \"ae53aab9115f1640d86df436ceacb7f96dae2fa96c88a8e0d57d08aa26cc25b1\"" Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.165009 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="69a8e880a3b2d17ecdaca01f72544c5de9832603ac3c68a4a4b668f13be89212" Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.165027 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="ae53aab9115f1640d86df436ceacb7f96dae2fa96c88a8e0d57d08aa26cc25b1" Jul 24 08:51:28 bluth k3s[11030]: E0724 08:51:28.165057 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-esphome/svclb-esphome-q4lrj" podUID=c41628dc-0533-406b-ac00-3fbf6cd22284 Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.166547 11030 docker_sandbox.go:402] "Failed to read pod IP from plugin/docker" err="networkPlugin cni failed on the status hook for pod \"radarr-7486f78d7f-q4pn4_ix-radarr\": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container \"46351dd20a65bcd3401b2ce2feaf6f4de7de1533f2dcc58d7dc4d09d8b7aacad\"" Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.276464 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="46351dd20a65bcd3401b2ce2feaf6f4de7de1533f2dcc58d7dc4d09d8b7aacad" Jul 24 08:51:28 bluth k3s[11030]: E0724 08:51:28.276537 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-radarr/radarr-7486f78d7f-q4pn4" podUID=be152b2b-fd1c-4a36-9081-ac6be2fb7901 Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.278133 11030 docker_sandbox.go:402] "Failed to read pod IP from plugin/docker" err="networkPlugin cni failed on the status hook for pod \"esphome-658957fddc-sb8c8_ix-esphome\": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container \"7c1dcc789affc3d314b965f028c17f94aba4566963ca77a10eab57d91b1a1972\"" Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.280917 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="7c1dcc789affc3d314b965f028c17f94aba4566963ca77a10eab57d91b1a1972" Jul 24 08:51:28 bluth k3s[11030]: E0724 08:51:28.281020 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-esphome/esphome-658957fddc-sb8c8" podUID=05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.282511 11030 docker_sandbox.go:402] "Failed to read pod IP from plugin/docker" err="networkPlugin cni failed on the status hook for pod \"esphome-658957fddc-bbcxl_ix-esphome\": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container \"6d2d1f41beed8c6cb0a510daf97a2b869656d3eb1f69580c3e40b7d603a6913d\"" Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.325259 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="6d2d1f41beed8c6cb0a510daf97a2b869656d3eb1f69580c3e40b7d603a6913d" Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.326606 11030 docker_sandbox.go:402] "Failed to read pod IP from plugin/docker" err="networkPlugin cni failed on the status hook for pod \"prowlarr-5bcc99cc47-rgrsl_ix-prowlarr\": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container \"9ac66d3a4484fab3c3fc091d5fb61f0f6cdd9b19f8d48a132bdea423b0ca1e77\"" Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.364303 11030 request.go:665] Waited for 1.107016441s due to client-side throttling, not priority and fairness, request: GET:https://127.0.0.1:6443/api/v1/namespaces/ix-home-assistant/configmaps?fieldSelector=metadata.name%3Dkube-root-ca.crt&limit=500&resourceVersion=0 Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.443531 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="9ac66d3a4484fab3c3fc091d5fb61f0f6cdd9b19f8d48a132bdea423b0ca1e77" Jul 24 08:51:28 bluth k3s[11030]: E0724 08:51:28.443593 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-prowlarr/prowlarr-5bcc99cc47-rgrsl" podUID=afd760a7-8b6b-470e-aa87-8ed16adb34bb Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.445218 11030 docker_sandbox.go:402] "Failed to read pod IP from plugin/docker" err="networkPlugin cni failed on the status hook for pod \"svclb-prowlarr-qhcm8_ix-prowlarr\": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container \"568c38abb65c3616d7624a6a1f8160bf3393ce5cce49c54dae9d81d75dbdf823\"" Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.455416 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="568c38abb65c3616d7624a6a1f8160bf3393ce5cce49c54dae9d81d75dbdf823" Jul 24 08:51:28 bluth k3s[11030]: E0724 08:51:28.455460 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-prowlarr/svclb-prowlarr-qhcm8" podUID=01a5e89a-9794-4261-ada8-82a5e21ccb7d Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.456974 11030 docker_sandbox.go:402] "Failed to read pod IP from plugin/docker" err="networkPlugin cni failed on the status hook for pod \"sonarr-867d7dd548-4bn8s_ix-sonarr\": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container \"788c819da6910895b7711adad84d6e1f1fdb4b436f2562f6c9b94ed9c7826957\"" Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.459979 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="788c819da6910895b7711adad84d6e1f1fdb4b436f2562f6c9b94ed9c7826957" Jul 24 08:51:28 bluth k3s[11030]: E0724 08:51:28.460082 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-sonarr/sonarr-867d7dd548-4bn8s" podUID=35ac75bf-b85f-4286-a228-ea8459dde305 Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.461322 11030 docker_sandbox.go:402] "Failed to read pod IP from plugin/docker" err="networkPlugin cni failed on the status hook for pod \"sonarr-867d7dd548-q6zsc_ix-sonarr\": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container \"9bf06bda197b74b428990270afa6fb445d8fd616bcf780b7af3d4862178e01bf\"" Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.462932 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="9bf06bda197b74b428990270afa6fb445d8fd616bcf780b7af3d4862178e01bf" Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.466758 11030 docker_sandbox.go:402] "Failed to read pod IP from plugin/docker" err="networkPlugin cni failed on the status hook for pod \"cloudflared-ix-chart-687fff6d55-t247w_ix-cloudflared\": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container \"a6327cc849b9115cec3b175ecb675c69b212d724d426e29b9149fcfcbe004693\"" Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.469601 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="a6327cc849b9115cec3b175ecb675c69b212d724d426e29b9149fcfcbe004693" Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.471139 11030 docker_sandbox.go:402] "Failed to read pod IP from plugin/docker" err="networkPlugin cni failed on the status hook for pod \"svclb-sonarr-69zgj_ix-sonarr\": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container \"554e89bab665d26a32b8de48179d718626c945bef4e0914bdb72348cb4b0424c\"" Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.471855 11030 docker_sandbox.go:402] "Failed to read pod IP from plugin/docker" err="networkPlugin cni failed on the status hook for pod \"svclb-sonarr-69zgj_ix-sonarr\": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container \"1fc67adce9f4e30d9b6b1eb0f3a6b4aaf5fec9ab91f920aa6866764e09b0fdcf\"" Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.474359 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="554e89bab665d26a32b8de48179d718626c945bef4e0914bdb72348cb4b0424c" Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.474373 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="1fc67adce9f4e30d9b6b1eb0f3a6b4aaf5fec9ab91f920aa6866764e09b0fdcf" Jul 24 08:51:28 bluth k3s[11030]: E0724 08:51:28.474407 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-sonarr/svclb-sonarr-69zgj" podUID=ecd248e1-4900-4e80-ba0b-7233f4b438b4 Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.475803 11030 docker_sandbox.go:402] "Failed to read pod IP from plugin/docker" err="networkPlugin cni failed on the status hook for pod \"intel-gpu-plugin-fxs6q_kube-system\": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container \"08296103f6b3137ee4836a8a22882c87ac8204632cd2fb900918cde27fc9f00d\"" Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.476424 11030 docker_sandbox.go:402] "Failed to read pod IP from plugin/docker" err="networkPlugin cni failed on the status hook for pod \"intel-gpu-plugin-fxs6q_kube-system\": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container \"1f9702e48cd10a6ed8d91715e373237e2c86c037931231de35857ee760f1237a\"" Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.573129 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="08296103f6b3137ee4836a8a22882c87ac8204632cd2fb900918cde27fc9f00d" Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.573147 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="1f9702e48cd10a6ed8d91715e373237e2c86c037931231de35857ee760f1237a" Jul 24 08:51:28 bluth k3s[11030]: E0724 08:51:28.573175 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="kube-system/intel-gpu-plugin-fxs6q" podUID=bb5993ab-6276-43d2-8017-a6fc01efb9a3 Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.574787 11030 docker_sandbox.go:402] "Failed to read pod IP from plugin/docker" err="networkPlugin cni failed on the status hook for pod \"coredns-d76bd69b-gk2vd_kube-system\": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container \"f1414372babc0c30df738691410c98290a02013f8804c21d718fbe797fbb3dec\"" Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.575412 11030 docker_sandbox.go:402] "Failed to read pod IP from plugin/docker" err="networkPlugin cni failed on the status hook for pod \"coredns-d76bd69b-gk2vd_kube-system\": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container \"dce4f4b3e8ca8548d50565bb42384f5836c7fc83b991f548646aab0519c46a9d\"" Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.613734 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="f1414372babc0c30df738691410c98290a02013f8804c21d718fbe797fbb3dec" Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.613752 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="dce4f4b3e8ca8548d50565bb42384f5836c7fc83b991f548646aab0519c46a9d" Jul 24 08:51:28 bluth k3s[11030]: E0724 08:51:28.613788 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="kube-system/coredns-d76bd69b-gk2vd" podUID=b1f2714d-382d-4638-83f4-2b2055748db3 Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.615354 11030 docker_sandbox.go:402] "Failed to read pod IP from plugin/docker" err="networkPlugin cni failed on the status hook for pod \"svclb-tdarr-comm-npvsl_ix-tdarr\": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container \"1c512175054884715ca92eb45b8a2a3ec64eb47b4580394d1ea00fff9a6c0ec5\"" Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.615975 11030 docker_sandbox.go:402] "Failed to read pod IP from plugin/docker" err="networkPlugin cni failed on the status hook for pod \"svclb-tdarr-comm-npvsl_ix-tdarr\": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container \"005e09ca79f65fcd916e189fde8a5b8aa1e12a236dcf8fa6e3bde9422eeb86ec\"" Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.618683 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="1c512175054884715ca92eb45b8a2a3ec64eb47b4580394d1ea00fff9a6c0ec5" Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.618696 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="005e09ca79f65fcd916e189fde8a5b8aa1e12a236dcf8fa6e3bde9422eeb86ec" Jul 24 08:51:28 bluth k3s[11030]: E0724 08:51:28.618747 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-tdarr/svclb-tdarr-comm-npvsl" podUID=69463f72-1255-4fe0-b5a2-59f859d48b92 Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.620190 11030 docker_sandbox.go:402] "Failed to read pod IP from plugin/docker" err="networkPlugin cni failed on the status hook for pod \"svclb-qbittorrent-torrentudp-62ksx_ix-qbittorrent\": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container \"9646db8ba80a34c2bef8a8af2f400a5605123c0d09435b55db342b1e27cfb9fa\"" Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.620771 11030 docker_sandbox.go:402] "Failed to read pod IP from plugin/docker" err="networkPlugin cni failed on the status hook for pod \"svclb-qbittorrent-torrentudp-62ksx_ix-qbittorrent\": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container \"717448f14cab88165abca913942f4948c690475046d08b5e08a8ce20182b5157\"" Jul 24 08:51:28 bluth k3s[11030]: E0724 08:51:28.661502 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-qbittorrent/svclb-qbittorrent-torrentudp-62ksx" podUID=c69bd8a5-5c5f-4a26-9204-b7d4036137ec Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.661572 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="9646db8ba80a34c2bef8a8af2f400a5605123c0d09435b55db342b1e27cfb9fa" Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.661586 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="717448f14cab88165abca913942f4948c690475046d08b5e08a8ce20182b5157" Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.664375 11030 docker_sandbox.go:402] "Failed to read pod IP from plugin/docker" err="networkPlugin cni failed on the status hook for pod \"svclb-radarr-nwf7v_ix-radarr\": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container \"e435127cb9f2eeb6d7d95a6f0ce58ca5db660af02044757de0c70f759204cb63\"" Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.693102 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="e435127cb9f2eeb6d7d95a6f0ce58ca5db660af02044757de0c70f759204cb63" Jul 24 08:51:28 bluth k3s[11030]: E0724 08:51:28.693150 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-radarr/svclb-radarr-nwf7v" podUID=51776aeb-f372-4d10-aa50-df5855b638e3 Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.694411 11030 docker_sandbox.go:402] "Failed to read pod IP from plugin/docker" err="networkPlugin cni failed on the status hook for pod \"home-assistant-ix-chart-54bf5d89c8-mz95k_ix-home-assistant\": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container \"93ce773435d99e6c171bcc350a4dec1fa738edda5dc537ff4e0e2ccb1dd443be\"" Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.696703 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="93ce773435d99e6c171bcc350a4dec1fa738edda5dc537ff4e0e2ccb1dd443be" Jul 24 08:51:28 bluth k3s[11030]: E0724 08:51:28.696794 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-mz95k" podUID=3f1f74fe-1d74-4b61-9bd2-083782908a35 Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.698239 11030 docker_sandbox.go:402] "Failed to read pod IP from plugin/docker" err="networkPlugin cni failed on the status hook for pod \"svclb-tdarr-44c97_ix-tdarr\": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container \"f3a735687a4e40b168169d3ff90f0743e4298fc2345713b0805896bbf025be6f\"" Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.698826 11030 docker_sandbox.go:402] "Failed to read pod IP from plugin/docker" err="networkPlugin cni failed on the status hook for pod \"svclb-tdarr-44c97_ix-tdarr\": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container \"18435955b67859313eb3b3436e5eacf2203618b8fce6d2a313d84bb2239758d7\"" Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.701186 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="f3a735687a4e40b168169d3ff90f0743e4298fc2345713b0805896bbf025be6f" Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.701199 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="18435955b67859313eb3b3436e5eacf2203618b8fce6d2a313d84bb2239758d7" Jul 24 08:51:28 bluth k3s[11030]: E0724 08:51:28.701229 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-tdarr/svclb-tdarr-44c97" podUID=e0e2f67d-a94d-4a52-a604-fb75e8d30555 Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.702461 11030 docker_sandbox.go:402] "Failed to read pod IP from plugin/docker" err="networkPlugin cni failed on the status hook for pod \"qbittorrent-6d7cf9f7cd-7n6c6_ix-qbittorrent\": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container \"e9b0b4c761ef9178a6c584628e4bad3640e0c7e17646220cc5483ae1d2d9f139\"" Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.743269 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="e9b0b4c761ef9178a6c584628e4bad3640e0c7e17646220cc5483ae1d2d9f139" Jul 24 08:51:28 bluth k3s[11030]: E0724 08:51:28.743345 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-qbittorrent/qbittorrent-6d7cf9f7cd-7n6c6" podUID=2fac2fcc-a503-4887-810c-c2ff11c07852 Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.744997 11030 docker_sandbox.go:402] "Failed to read pod IP from plugin/docker" err="networkPlugin cni failed on the status hook for pod \"cloudflared-ix-chart-687fff6d55-z78xq_ix-cloudflared\": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container \"2e4eeae2d2bc8561a0c65ea655e2d96c6388021de8a81b475bdaf0c3c2a7df72\"" Jul 24 08:51:28 bluth k3s[11030]: I0724 08:51:28.746553 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="2e4eeae2d2bc8561a0c65ea655e2d96c6388021de8a81b475bdaf0c3c2a7df72" Jul 24 08:51:28 bluth k3s[11030]: E0724 08:51:28.746642 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-cloudflared/cloudflared-ix-chart-687fff6d55-z78xq" podUID=1f5bcb97-e684-4d8a-904e-1fdbd19a42f9 Jul 24 08:51:29 bluth k3s[11030]: I0724 08:51:29.302768 11030 controller.go:611] quota admission added evaluator for: daemonsets.apps Jul 24 08:51:29 bluth k3s[11030]: I0724 08:51:29.640196 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"shared\" (UniqueName: \"kubernetes.io/empty-dir/afd760a7-8b6b-470e-aa87-8ed16adb34bb-shared\") pod \"prowlarr-5bcc99cc47-rgrsl\" (UID: \"afd760a7-8b6b-470e-aa87-8ed16adb34bb\") " pod="ix-prowlarr/prowlarr-5bcc99cc47-rgrsl" Jul 24 08:51:29 bluth k3s[11030]: I0724 08:51:29.640225 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp\" (UniqueName: \"kubernetes.io/empty-dir/afd760a7-8b6b-470e-aa87-8ed16adb34bb-temp\") pod \"prowlarr-5bcc99cc47-rgrsl\" (UID: \"afd760a7-8b6b-470e-aa87-8ed16adb34bb\") " pod="ix-prowlarr/prowlarr-5bcc99cc47-rgrsl" Jul 24 08:51:29 bluth k3s[11030]: I0724 08:51:29.640239 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"varlogs\" (UniqueName: \"kubernetes.io/empty-dir/afd760a7-8b6b-470e-aa87-8ed16adb34bb-varlogs\") pod \"prowlarr-5bcc99cc47-rgrsl\" (UID: \"afd760a7-8b6b-470e-aa87-8ed16adb34bb\") " pod="ix-prowlarr/prowlarr-5bcc99cc47-rgrsl" Jul 24 08:51:29 bluth k3s[11030]: I0724 08:51:29.640256 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274\") pod \"prowlarr-5bcc99cc47-rgrsl\" (UID: \"afd760a7-8b6b-470e-aa87-8ed16adb34bb\") " pod="ix-prowlarr/prowlarr-5bcc99cc47-rgrsl" Jul 24 08:51:29 bluth k3s[11030]: I0724 08:51:29.640271 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc4q9\" (UniqueName: \"kubernetes.io/projected/afd760a7-8b6b-470e-aa87-8ed16adb34bb-kube-api-access-mc4q9\") pod \"prowlarr-5bcc99cc47-rgrsl\" (UID: \"afd760a7-8b6b-470e-aa87-8ed16adb34bb\") " pod="ix-prowlarr/prowlarr-5bcc99cc47-rgrsl" Jul 24 08:51:29 bluth k3s[11030]: E0724 08:51:29.747968 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=1 Jul 24 08:51:29 bluth k3s[11030]: I0724 08:51:29.761961 11030 docker_sandbox.go:402] "Failed to read pod IP from plugin/docker" err="networkPlugin cni failed on the status hook for pod \"svclb-qbittorrent-torrent-8qgf8_ix-qbittorrent\": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container \"b6e3949c474adcffbc57d4c83242acb2528215cdb92ccaf02d777d568775734d\"" Jul 24 08:51:29 bluth k3s[11030]: I0724 08:51:29.762622 11030 docker_sandbox.go:402] "Failed to read pod IP from plugin/docker" err="networkPlugin cni failed on the status hook for pod \"svclb-qbittorrent-torrent-8qgf8_ix-qbittorrent\": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container \"14a15e5a147c12eddc4317a17434a059afa99b54bda399bd2d82ff6a5e07e9be\"" Jul 24 08:51:29 bluth k3s[11030]: E0724 08:51:29.765474 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-qbittorrent/svclb-qbittorrent-ldv82" podUID=ad6ed51e-d3e1-430c-9bad-528f4ae492c5 Jul 24 08:51:29 bluth k3s[11030]: E0724 08:51:29.765554 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-qbittorrent/svclb-qbittorrent-torrent-8qgf8" podUID=19ef4922-0714-452c-b8ba-e3c955cabe50 Jul 24 08:51:29 bluth k3s[11030]: E0724 08:51:29.765605 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-mz95k" podUID=3f1f74fe-1d74-4b61-9bd2-083782908a35 Jul 24 08:51:29 bluth k3s[11030]: E0724 08:51:29.765658 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="kube-system/coredns-d76bd69b-gk2vd" podUID=b1f2714d-382d-4638-83f4-2b2055748db3 Jul 24 08:51:29 bluth k3s[11030]: E0724 08:51:29.765710 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-prowlarr/svclb-prowlarr-qhcm8" podUID=01a5e89a-9794-4261-ada8-82a5e21ccb7d Jul 24 08:51:29 bluth k3s[11030]: E0724 08:51:29.765769 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="kube-system/intel-gpu-plugin-fxs6q" podUID=bb5993ab-6276-43d2-8017-a6fc01efb9a3 Jul 24 08:51:29 bluth k3s[11030]: E0724 08:51:29.765830 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-sonarr/sonarr-867d7dd548-4bn8s" podUID=35ac75bf-b85f-4286-a228-ea8459dde305 Jul 24 08:51:29 bluth k3s[11030]: E0724 08:51:29.765882 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-tdarr/svclb-tdarr-44c97" podUID=e0e2f67d-a94d-4a52-a604-fb75e8d30555 Jul 24 08:51:29 bluth k3s[11030]: E0724 08:51:29.765941 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-tdarr/tdarr-746d65b466-b7whb" podUID=6e3e9791-9bb3-487f-ae50-048fefbef8be Jul 24 08:51:29 bluth k3s[11030]: E0724 08:51:29.765993 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-qbittorrent/svclb-qbittorrent-torrentudp-62ksx" podUID=c69bd8a5-5c5f-4a26-9204-b7d4036137ec Jul 24 08:51:29 bluth k3s[11030]: E0724 08:51:29.766050 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-prowlarr/prowlarr-5bcc99cc47-rgrsl" podUID=afd760a7-8b6b-470e-aa87-8ed16adb34bb Jul 24 08:51:29 bluth k3s[11030]: E0724 08:51:29.766099 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-tdarr/svclb-tdarr-comm-npvsl" podUID=69463f72-1255-4fe0-b5a2-59f859d48b92 Jul 24 08:51:29 bluth k3s[11030]: E0724 08:51:29.766148 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-esphome/svclb-esphome-q4lrj" podUID=c41628dc-0533-406b-ac00-3fbf6cd22284 Jul 24 08:51:29 bluth k3s[11030]: E0724 08:51:29.766209 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-esphome/esphome-658957fddc-sb8c8" podUID=05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b Jul 24 08:51:29 bluth k3s[11030]: E0724 08:51:29.766257 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-cloudflared/cloudflared-ix-chart-687fff6d55-z78xq" podUID=1f5bcb97-e684-4d8a-904e-1fdbd19a42f9 Jul 24 08:51:29 bluth k3s[11030]: E0724 08:51:29.766367 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="kube-system/openebs-zfs-controller-0" podUID=508c7225-82e1-4311-9b86-6a6ad8bf3740 Jul 24 08:51:29 bluth k3s[11030]: E0724 08:51:29.766425 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-qbittorrent/qbittorrent-6d7cf9f7cd-7n6c6" podUID=2fac2fcc-a503-4887-810c-c2ff11c07852 Jul 24 08:51:29 bluth k3s[11030]: E0724 08:51:29.766483 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-sonarr/svclb-sonarr-69zgj" podUID=ecd248e1-4900-4e80-ba0b-7233f4b438b4 Jul 24 08:51:29 bluth k3s[11030]: E0724 08:51:29.766544 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-radarr/radarr-7486f78d7f-q4pn4" podUID=be152b2b-fd1c-4a36-9081-ac6be2fb7901 Jul 24 08:51:29 bluth k3s[11030]: E0724 08:51:29.766593 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-radarr/svclb-radarr-nwf7v" podUID=51776aeb-f372-4d10-aa50-df5855b638e3 Jul 24 08:51:30 bluth k3s[11030]: I0724 08:51:30.646830 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"list-0\" (UniqueName: \"kubernetes.io/host-path/be152b2b-fd1c-4a36-9081-ac6be2fb7901-list-0\") pod \"radarr-7486f78d7f-q4pn4\" (UID: \"be152b2b-fd1c-4a36-9081-ac6be2fb7901\") " pod="ix-radarr/radarr-7486f78d7f-q4pn4" Jul 24 08:51:30 bluth k3s[11030]: I0724 08:51:30.646862 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"shared\" (UniqueName: \"kubernetes.io/empty-dir/be152b2b-fd1c-4a36-9081-ac6be2fb7901-shared\") pod \"radarr-7486f78d7f-q4pn4\" (UID: \"be152b2b-fd1c-4a36-9081-ac6be2fb7901\") " pod="ix-radarr/radarr-7486f78d7f-q4pn4" Jul 24 08:51:30 bluth k3s[11030]: I0724 08:51:30.646879 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp\" (UniqueName: \"kubernetes.io/empty-dir/be152b2b-fd1c-4a36-9081-ac6be2fb7901-temp\") pod \"radarr-7486f78d7f-q4pn4\" (UID: \"be152b2b-fd1c-4a36-9081-ac6be2fb7901\") " pod="ix-radarr/radarr-7486f78d7f-q4pn4" Jul 24 08:51:30 bluth k3s[11030]: I0724 08:51:30.646920 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjp24\" (UniqueName: \"kubernetes.io/projected/be152b2b-fd1c-4a36-9081-ac6be2fb7901-kube-api-access-vjp24\") pod \"radarr-7486f78d7f-q4pn4\" (UID: \"be152b2b-fd1c-4a36-9081-ac6be2fb7901\") " pod="ix-radarr/radarr-7486f78d7f-q4pn4" Jul 24 08:51:30 bluth k3s[11030]: I0724 08:51:30.646935 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"varlogs\" (UniqueName: \"kubernetes.io/empty-dir/be152b2b-fd1c-4a36-9081-ac6be2fb7901-varlogs\") pod \"radarr-7486f78d7f-q4pn4\" (UID: \"be152b2b-fd1c-4a36-9081-ac6be2fb7901\") " pod="ix-radarr/radarr-7486f78d7f-q4pn4" Jul 24 08:51:30 bluth k3s[11030]: I0724 08:51:30.646958 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-195f1a32-9c93-4737-983a-8dacee32e453\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-195f1a32-9c93-4737-983a-8dacee32e453\") pod \"radarr-7486f78d7f-q4pn4\" (UID: \"be152b2b-fd1c-4a36-9081-ac6be2fb7901\") " pod="ix-radarr/radarr-7486f78d7f-q4pn4" Jul 24 08:51:30 bluth k3s[11030]: E0724 08:51:30.773992 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-tdarr/tdarr-746d65b466-b7whb" podUID=6e3e9791-9bb3-487f-ae50-048fefbef8be Jul 24 08:51:30 bluth k3s[11030]: E0724 08:51:30.774051 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-qbittorrent/qbittorrent-6d7cf9f7cd-7n6c6" podUID=2fac2fcc-a503-4887-810c-c2ff11c07852 Jul 24 08:51:30 bluth k3s[11030]: E0724 08:51:30.774115 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-radarr/radarr-7486f78d7f-q4pn4" podUID=be152b2b-fd1c-4a36-9081-ac6be2fb7901 Jul 24 08:51:30 bluth k3s[11030]: E0724 08:51:30.774168 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-prowlarr/prowlarr-5bcc99cc47-rgrsl" podUID=afd760a7-8b6b-470e-aa87-8ed16adb34bb Jul 24 08:51:30 bluth k3s[11030]: E0724 08:51:30.774212 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-sonarr/sonarr-867d7dd548-4bn8s" podUID=35ac75bf-b85f-4286-a228-ea8459dde305 Jul 24 08:51:30 bluth k3s[11030]: E0724 08:51:30.774260 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-esphome/svclb-esphome-q4lrj" podUID=c41628dc-0533-406b-ac00-3fbf6cd22284 Jul 24 08:51:30 bluth k3s[11030]: E0724 08:51:30.774301 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-qbittorrent/svclb-qbittorrent-ldv82" podUID=ad6ed51e-d3e1-430c-9bad-528f4ae492c5 Jul 24 08:51:30 bluth k3s[11030]: E0724 08:51:30.774358 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-qbittorrent/svclb-qbittorrent-torrentudp-62ksx" podUID=c69bd8a5-5c5f-4a26-9204-b7d4036137ec Jul 24 08:51:30 bluth k3s[11030]: E0724 08:51:30.774394 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-tdarr/svclb-tdarr-comm-npvsl" podUID=69463f72-1255-4fe0-b5a2-59f859d48b92 Jul 24 08:51:30 bluth k3s[11030]: E0724 08:51:30.774447 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-esphome/esphome-658957fddc-sb8c8" podUID=05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b Jul 24 08:51:30 bluth k3s[11030]: E0724 08:51:30.774522 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="kube-system/openebs-zfs-controller-0" podUID=508c7225-82e1-4311-9b86-6a6ad8bf3740 Jul 24 08:51:30 bluth k3s[11030]: E0724 08:51:30.774568 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-qbittorrent/svclb-qbittorrent-torrent-8qgf8" podUID=19ef4922-0714-452c-b8ba-e3c955cabe50 Jul 24 08:51:30 bluth k3s[11030]: E0724 08:51:30.774610 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-sonarr/svclb-sonarr-69zgj" podUID=ecd248e1-4900-4e80-ba0b-7233f4b438b4 Jul 24 08:51:30 bluth k3s[11030]: E0724 08:51:30.774653 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="kube-system/coredns-d76bd69b-gk2vd" podUID=b1f2714d-382d-4638-83f4-2b2055748db3 Jul 24 08:51:31 bluth k3s[11030]: I0724 08:51:31.652728 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"shared\" (UniqueName: \"kubernetes.io/empty-dir/35ac75bf-b85f-4286-a228-ea8459dde305-shared\") pod \"sonarr-867d7dd548-4bn8s\" (UID: \"35ac75bf-b85f-4286-a228-ea8459dde305\") " pod="ix-sonarr/sonarr-867d7dd548-4bn8s" Jul 24 08:51:31 bluth k3s[11030]: I0724 08:51:31.652782 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v58k\" (UniqueName: \"kubernetes.io/projected/35ac75bf-b85f-4286-a228-ea8459dde305-kube-api-access-2v58k\") pod \"sonarr-867d7dd548-4bn8s\" (UID: \"35ac75bf-b85f-4286-a228-ea8459dde305\") " pod="ix-sonarr/sonarr-867d7dd548-4bn8s" Jul 24 08:51:31 bluth k3s[11030]: I0724 08:51:31.652798 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67x4q\" (UniqueName: \"kubernetes.io/projected/b1f2714d-382d-4638-83f4-2b2055748db3-kube-api-access-67x4q\") pod \"coredns-d76bd69b-gk2vd\" (UID: \"b1f2714d-382d-4638-83f4-2b2055748db3\") " pod="kube-system/coredns-d76bd69b-gk2vd" Jul 24 08:51:31 bluth k3s[11030]: I0724 08:51:31.652835 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/empty-dir/508c7225-82e1-4311-9b86-6a6ad8bf3740-socket-dir\") pod \"openebs-zfs-controller-0\" (UID: \"508c7225-82e1-4311-9b86-6a6ad8bf3740\") " pod="kube-system/openebs-zfs-controller-0" Jul 24 08:51:31 bluth k3s[11030]: I0724 08:51:31.652859 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4\") pod \"sonarr-867d7dd548-4bn8s\" (UID: \"35ac75bf-b85f-4286-a228-ea8459dde305\") " pod="ix-sonarr/sonarr-867d7dd548-4bn8s" Jul 24 08:51:31 bluth k3s[11030]: I0724 08:51:31.652874 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b1f2714d-382d-4638-83f4-2b2055748db3-config-volume\") pod \"coredns-d76bd69b-gk2vd\" (UID: \"b1f2714d-382d-4638-83f4-2b2055748db3\") " pod="kube-system/coredns-d76bd69b-gk2vd" Jul 24 08:51:31 bluth k3s[11030]: I0724 08:51:31.652887 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"custom-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1f2714d-382d-4638-83f4-2b2055748db3-custom-config-volume\") pod \"coredns-d76bd69b-gk2vd\" (UID: \"b1f2714d-382d-4638-83f4-2b2055748db3\") " pod="kube-system/coredns-d76bd69b-gk2vd" Jul 24 08:51:31 bluth k3s[11030]: I0724 08:51:31.652904 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"list-0\" (UniqueName: \"kubernetes.io/host-path/35ac75bf-b85f-4286-a228-ea8459dde305-list-0\") pod \"sonarr-867d7dd548-4bn8s\" (UID: \"35ac75bf-b85f-4286-a228-ea8459dde305\") " pod="ix-sonarr/sonarr-867d7dd548-4bn8s" Jul 24 08:51:31 bluth k3s[11030]: I0724 08:51:31.652917 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp\" (UniqueName: \"kubernetes.io/empty-dir/35ac75bf-b85f-4286-a228-ea8459dde305-temp\") pod \"sonarr-867d7dd548-4bn8s\" (UID: \"35ac75bf-b85f-4286-a228-ea8459dde305\") " pod="ix-sonarr/sonarr-867d7dd548-4bn8s" Jul 24 08:51:31 bluth k3s[11030]: I0724 08:51:31.652930 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"varlogs\" (UniqueName: \"kubernetes.io/empty-dir/35ac75bf-b85f-4286-a228-ea8459dde305-varlogs\") pod \"sonarr-867d7dd548-4bn8s\" (UID: \"35ac75bf-b85f-4286-a228-ea8459dde305\") " pod="ix-sonarr/sonarr-867d7dd548-4bn8s" Jul 24 08:51:31 bluth k3s[11030]: I0724 08:51:31.652998 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr6l5\" (UniqueName: \"kubernetes.io/projected/508c7225-82e1-4311-9b86-6a6ad8bf3740-kube-api-access-cr6l5\") pod \"openebs-zfs-controller-0\" (UID: \"508c7225-82e1-4311-9b86-6a6ad8bf3740\") " pod="kube-system/openebs-zfs-controller-0" Jul 24 08:51:31 bluth k3s[11030]: E0724 08:51:31.742717 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-prowlarr/svclb-prowlarr-qhcm8" podUID=01a5e89a-9794-4261-ada8-82a5e21ccb7d Jul 24 08:51:31 bluth k3s[11030]: E0724 08:51:31.742786 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-mz95k" podUID=3f1f74fe-1d74-4b61-9bd2-083782908a35 Jul 24 08:51:31 bluth k3s[11030]: E0724 08:51:31.742844 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-tdarr/svclb-tdarr-44c97" podUID=e0e2f67d-a94d-4a52-a604-fb75e8d30555 Jul 24 08:51:31 bluth k3s[11030]: E0724 08:51:31.742910 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="kube-system/intel-gpu-plugin-fxs6q" podUID=bb5993ab-6276-43d2-8017-a6fc01efb9a3 Jul 24 08:51:31 bluth k3s[11030]: E0724 08:51:31.742950 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-radarr/svclb-radarr-nwf7v" podUID=51776aeb-f372-4d10-aa50-df5855b638e3 Jul 24 08:51:31 bluth k3s[11030]: E0724 08:51:31.742985 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-cloudflared/cloudflared-ix-chart-687fff6d55-z78xq" podUID=1f5bcb97-e684-4d8a-904e-1fdbd19a42f9 Jul 24 08:51:31 bluth k3s[11030]: E0724 08:51:31.747072 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=1 Jul 24 08:51:31 bluth k3s[11030]: E0724 08:51:31.782617 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-radarr/radarr-7486f78d7f-q4pn4" podUID=be152b2b-fd1c-4a36-9081-ac6be2fb7901 Jul 24 08:51:31 bluth k3s[11030]: E0724 08:51:31.782672 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-tdarr/tdarr-746d65b466-b7whb" podUID=6e3e9791-9bb3-487f-ae50-048fefbef8be Jul 24 08:51:31 bluth k3s[11030]: E0724 08:51:31.782742 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-qbittorrent/qbittorrent-6d7cf9f7cd-7n6c6" podUID=2fac2fcc-a503-4887-810c-c2ff11c07852 Jul 24 08:51:31 bluth k3s[11030]: E0724 08:51:31.796168 11030 pod_workers.go:949] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="ix-prowlarr/prowlarr-5bcc99cc47-rgrsl" podUID=afd760a7-8b6b-470e-aa87-8ed16adb34bb Jul 24 08:51:32 bluth k3s[11030]: E0724 08:51:32.054838 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-195f1a32-9c93-4737-983a-8dacee32e453 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:32.554719594 +0100 IST m=+19.042576474 (durationBeforeRetry 500ms). Error: MountVolume.MountDevice failed for volume "pvc-195f1a32-9c93-4737-983a-8dacee32e453" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-195f1a32-9c93-4737-983a-8dacee32e453") pod "radarr-7486f78d7f-q4pn4" (UID: "be152b2b-fd1c-4a36-9081-ac6be2fb7901") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:32 bluth k3s[11030]: E0724 08:51:32.054879 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:32.554871085 +0100 IST m=+19.042727965 (durationBeforeRetry 500ms). Error: MountVolume.MountDevice failed for volume "pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274") pod "prowlarr-5bcc99cc47-rgrsl" (UID: "afd760a7-8b6b-470e-aa87-8ed16adb34bb") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:32 bluth k3s[11030]: E0724 08:51:32.056771 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:32.556760228 +0100 IST m=+19.044617109 (durationBeforeRetry 500ms). Error: MountVolume.MountDevice failed for volume "pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4") pod "sonarr-867d7dd548-4bn8s" (UID: "35ac75bf-b85f-4286-a228-ea8459dde305") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:32 bluth k3s[11030]: E0724 08:51:32.582507 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:33.582495488 +0100 IST m=+20.070352373 (durationBeforeRetry 1s). Error: MountVolume.MountDevice failed for volume "pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274") pod "prowlarr-5bcc99cc47-rgrsl" (UID: "afd760a7-8b6b-470e-aa87-8ed16adb34bb") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:32 bluth k3s[11030]: E0724 08:51:32.582659 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-195f1a32-9c93-4737-983a-8dacee32e453 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:33.582652354 +0100 IST m=+20.070509235 (durationBeforeRetry 1s). Error: MountVolume.MountDevice failed for volume "pvc-195f1a32-9c93-4737-983a-8dacee32e453" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-195f1a32-9c93-4737-983a-8dacee32e453") pod "radarr-7486f78d7f-q4pn4" (UID: "be152b2b-fd1c-4a36-9081-ac6be2fb7901") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:32 bluth k3s[11030]: E0724 08:51:32.583283 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:33.583273698 +0100 IST m=+20.071130579 (durationBeforeRetry 1s). Error: MountVolume.MountDevice failed for volume "pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4") pod "sonarr-867d7dd548-4bn8s" (UID: "35ac75bf-b85f-4286-a228-ea8459dde305") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:32 bluth k3s[11030]: I0724 08:51:32.772299 11030 cni.go:334] "CNI failed to retrieve network namespace path" err="cannot find network namespace for the terminated container \"f1414372babc0c30df738691410c98290a02013f8804c21d718fbe797fbb3dec\"" Jul 24 08:51:32 bluth k3s[11030]: I0724 08:51:32.774414 11030 cni.go:334] "CNI failed to retrieve network namespace path" err="cannot find network namespace for the terminated container \"b6d43670deb9d67684a6743dad54c0dd6c4e7c295084b0ac48903cf764b79c20\"" Jul 24 08:51:32 bluth k3s[11030]: I0724 08:51:32.774760 11030 cni.go:334] "CNI failed to retrieve network namespace path" err="cannot find network namespace for the terminated container \"b6e3949c474adcffbc57d4c83242acb2528215cdb92ccaf02d777d568775734d\"" Jul 24 08:51:32 bluth k3s[11030]: I0724 08:51:32.775185 11030 cni.go:334] "CNI failed to retrieve network namespace path" err="cannot find network namespace for the terminated container \"69a8e880a3b2d17ecdaca01f72544c5de9832603ac3c68a4a4b668f13be89212\"" Jul 24 08:51:32 bluth k3s[11030]: I0724 08:51:32.776062 11030 cni.go:334] "CNI failed to retrieve network namespace path" err="cannot find network namespace for the terminated container \"9646db8ba80a34c2bef8a8af2f400a5605123c0d09435b55db342b1e27cfb9fa\"" Jul 24 08:51:32 bluth k3s[11030]: I0724 08:51:32.776212 11030 cni.go:334] "CNI failed to retrieve network namespace path" err="cannot find network namespace for the terminated container \"554e89bab665d26a32b8de48179d718626c945bef4e0914bdb72348cb4b0424c\"" Jul 24 08:51:32 bluth k3s[11030]: I0724 08:51:32.777384 11030 cni.go:334] "CNI failed to retrieve network namespace path" err="cannot find network namespace for the terminated container \"bf4c9c440473cecdcd094a4ebeea14778d786c81fa9f21f1b69c814963677f8f\"" Jul 24 08:51:32 bluth k3s[11030]: I0724 08:51:32.777648 11030 cni.go:334] "CNI failed to retrieve network namespace path" err="cannot find network namespace for the terminated container \"1c512175054884715ca92eb45b8a2a3ec64eb47b4580394d1ea00fff9a6c0ec5\"" Jul 24 08:51:32 bluth k3s[11030]: I0724 08:51:32.854723 11030 cni.go:334] "CNI failed to retrieve network namespace path" err="cannot find network namespace for the terminated container \"ae53aab9115f1640d86df436ceacb7f96dae2fa96c88a8e0d57d08aa26cc25b1\"" Jul 24 08:51:32 bluth k3s[11030]: I0724 08:51:32.860810 11030 cni.go:334] "CNI failed to retrieve network namespace path" err="cannot find network namespace for the terminated container \"5364fb99bc658579081df87e1915ed88fda1de41488a83ba9c0154f4e691952d\"" Jul 24 08:51:32 bluth k3s[11030]: I0724 08:51:32.863803 11030 cni.go:334] "CNI failed to retrieve network namespace path" err="cannot find network namespace for the terminated container \"14a15e5a147c12eddc4317a17434a059afa99b54bda399bd2d82ff6a5e07e9be\"" Jul 24 08:51:32 bluth k3s[11030]: I0724 08:51:32.868237 11030 cni.go:334] "CNI failed to retrieve network namespace path" err="cannot find network namespace for the terminated container \"1fc67adce9f4e30d9b6b1eb0f3a6b4aaf5fec9ab91f920aa6866764e09b0fdcf\"" Jul 24 08:51:32 bluth k3s[11030]: I0724 08:51:32.876834 11030 cni.go:334] "CNI failed to retrieve network namespace path" err="cannot find network namespace for the terminated container \"dce4f4b3e8ca8548d50565bb42384f5836c7fc83b991f548646aab0519c46a9d\"" Jul 24 08:51:32 bluth k3s[11030]: I0724 08:51:32.877334 11030 cni.go:334] "CNI failed to retrieve network namespace path" err="cannot find network namespace for the terminated container \"005e09ca79f65fcd916e189fde8a5b8aa1e12a236dcf8fa6e3bde9422eeb86ec\"" Jul 24 08:51:32 bluth k3s[11030]: I0724 08:51:32.889735 11030 cni.go:334] "CNI failed to retrieve network namespace path" err="cannot find network namespace for the terminated container \"717448f14cab88165abca913942f4948c690475046d08b5e08a8ce20182b5157\"" Jul 24 08:51:33 bluth k3s[11030]: I0724 08:51:33.060641 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-028dc34a-8c2e-4a0a-b507-c33a02541407\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-028dc34a-8c2e-4a0a-b507-c33a02541407\") pod \"tdarr-746d65b466-b7whb\" (UID: \"6e3e9791-9bb3-487f-ae50-048fefbef8be\") " pod="ix-tdarr/tdarr-746d65b466-b7whb" Jul 24 08:51:33 bluth k3s[11030]: I0724 08:51:33.060669 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"list-0\" (UniqueName: \"kubernetes.io/host-path/6e3e9791-9bb3-487f-ae50-048fefbef8be-list-0\") pod \"tdarr-746d65b466-b7whb\" (UID: \"6e3e9791-9bb3-487f-ae50-048fefbef8be\") " pod="ix-tdarr/tdarr-746d65b466-b7whb" Jul 24 08:51:33 bluth k3s[11030]: E0724 08:51:33.500327 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-028dc34a-8c2e-4a0a-b507-c33a02541407 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:34.000315642 +0100 IST m=+20.488172525 (durationBeforeRetry 500ms). Error: MountVolume.MountDevice failed for volume "pvc-028dc34a-8c2e-4a0a-b507-c33a02541407" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-028dc34a-8c2e-4a0a-b507-c33a02541407") pod "tdarr-746d65b466-b7whb" (UID: "6e3e9791-9bb3-487f-ae50-048fefbef8be") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:33 bluth k3s[11030]: I0724 08:51:33.745227 11030 cni.go:334] "CNI failed to retrieve network namespace path" err="cannot find network namespace for the terminated container \"2e4eeae2d2bc8561a0c65ea655e2d96c6388021de8a81b475bdaf0c3c2a7df72\"" Jul 24 08:51:33 bluth k3s[11030]: E0724 08:51:33.760152 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=1 Jul 24 08:51:33 bluth k3s[11030]: I0724 08:51:33.767091 11030 cni.go:334] "CNI failed to retrieve network namespace path" err="cannot find network namespace for the terminated container \"568c38abb65c3616d7624a6a1f8160bf3393ce5cce49c54dae9d81d75dbdf823\"" Jul 24 08:51:33 bluth k3s[11030]: I0724 08:51:33.767351 11030 cni.go:334] "CNI failed to retrieve network namespace path" err="cannot find network namespace for the terminated container \"e435127cb9f2eeb6d7d95a6f0ce58ca5db660af02044757de0c70f759204cb63\"" Jul 24 08:51:33 bluth k3s[11030]: I0724 08:51:33.769641 11030 cni.go:334] "CNI failed to retrieve network namespace path" err="cannot find network namespace for the terminated container \"f3a735687a4e40b168169d3ff90f0743e4298fc2345713b0805896bbf025be6f\"" Jul 24 08:51:33 bluth k3s[11030]: I0724 08:51:33.790487 11030 cni.go:334] "CNI failed to retrieve network namespace path" err="cannot find network namespace for the terminated container \"18435955b67859313eb3b3436e5eacf2203618b8fce6d2a313d84bb2239758d7\"" Jul 24 08:51:33 bluth k3s[11030]: I0724 08:51:33.865719 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-2e6b3d10-435c-4ce5-806d-61d551dcef05\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-2e6b3d10-435c-4ce5-806d-61d551dcef05\") pod \"tdarr-746d65b466-b7whb\" (UID: \"6e3e9791-9bb3-487f-ae50-048fefbef8be\") " pod="ix-tdarr/tdarr-746d65b466-b7whb" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.115940 11030 range_allocator.go:83] Sending events to api server. Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.116132 11030 range_allocator.go:111] No Service CIDR provided. Skipping filtering out service addresses. Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.116139 11030 range_allocator.go:117] No Secondary Service CIDR provided. Skipping filtering out secondary service addresses. Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.116171 11030 controllermanager.go:605] Started "nodeipam" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.116336 11030 node_ipam_controller.go:154] Starting ipam controller Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.116346 11030 shared_informer.go:240] Waiting for caches to sync for node Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.117636 11030 controllermanager.go:605] Started "ttl-after-finished" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.117708 11030 ttlafterfinished_controller.go:109] Starting TTL after finished controller Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.117716 11030 shared_informer.go:240] Waiting for caches to sync for TTL after finished Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.128130 11030 resource_quota_monitor.go:233] QuotaMonitor created object count evaluator for rolebindings.rbac.authorization.k8s.io Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.128158 11030 resource_quota_monitor.go:233] QuotaMonitor created object count evaluator for endpointslices.discovery.k8s.io Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.128200 11030 resource_quota_monitor.go:233] QuotaMonitor created object count evaluator for serviceaccounts Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.128310 11030 resource_quota_monitor.go:233] QuotaMonitor created object count evaluator for horizontalpodautoscalers.autoscaling Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.128343 11030 resource_quota_monitor.go:233] QuotaMonitor created object count evaluator for csistoragecapacities.storage.k8s.io Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.128468 11030 resource_quota_monitor.go:233] QuotaMonitor created object count evaluator for helmcharts.helm.cattle.io Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.128483 11030 resource_quota_monitor.go:233] QuotaMonitor created object count evaluator for zfsnodes.zfs.openebs.io Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.128494 11030 resource_quota_monitor.go:233] QuotaMonitor created object count evaluator for controllerrevisions.apps Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.128503 11030 resource_quota_monitor.go:233] QuotaMonitor created object count evaluator for daemonsets.apps Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.128515 11030 resource_quota_monitor.go:233] QuotaMonitor created object count evaluator for helmchartconfigs.helm.cattle.io Jul 24 08:51:34 bluth k3s[11030]: W0724 08:51:34.128522 11030 shared_informer.go:504] resyncPeriod 19h5m47.023014192s is smaller than resyncCheckPeriod 21h18m22.034471446s and the informer has already started. Changing it to 21h18m22.034471446s Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.128567 11030 resource_quota_monitor.go:233] QuotaMonitor created object count evaluator for replicasets.apps Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.128577 11030 resource_quota_monitor.go:233] QuotaMonitor created object count evaluator for limitranges Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.128595 11030 resource_quota_monitor.go:233] QuotaMonitor created object count evaluator for cronjobs.batch Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.128610 11030 resource_quota_monitor.go:233] QuotaMonitor created object count evaluator for poddisruptionbudgets.policy Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.128622 11030 resource_quota_monitor.go:233] QuotaMonitor created object count evaluator for podtemplates Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.128631 11030 resource_quota_monitor.go:233] QuotaMonitor created object count evaluator for endpoints Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.128651 11030 resource_quota_monitor.go:233] QuotaMonitor created object count evaluator for statefulsets.apps Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.128663 11030 resource_quota_monitor.go:233] QuotaMonitor created object count evaluator for zfsrestores.zfs.openebs.io Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.128672 11030 resource_quota_monitor.go:233] QuotaMonitor created object count evaluator for zfssnapshots.zfs.openebs.io Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.128709 11030 resource_quota_monitor.go:233] QuotaMonitor created object count evaluator for zfsvolumes.zfs.openebs.io Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.128722 11030 resource_quota_monitor.go:233] QuotaMonitor created object count evaluator for roles.rbac.authorization.k8s.io Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.128732 11030 resource_quota_monitor.go:233] QuotaMonitor created object count evaluator for network-attachment-definitions.k8s.cni.cncf.io Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.128743 11030 resource_quota_monitor.go:233] QuotaMonitor created object count evaluator for zfsbackups.zfs.openebs.io Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.128757 11030 resource_quota_monitor.go:233] QuotaMonitor created object count evaluator for jobs.batch Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.128768 11030 resource_quota_monitor.go:233] QuotaMonitor created object count evaluator for ingresses.networking.k8s.io Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.128866 11030 resource_quota_monitor.go:233] QuotaMonitor created object count evaluator for events.events.k8s.io Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.128880 11030 resource_quota_monitor.go:233] QuotaMonitor created object count evaluator for networkpolicies.networking.k8s.io Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.128889 11030 resource_quota_monitor.go:233] QuotaMonitor created object count evaluator for leases.coordination.k8s.io Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.128900 11030 resource_quota_monitor.go:233] QuotaMonitor created object count evaluator for addons.k3s.cattle.io Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.128910 11030 resource_quota_monitor.go:233] QuotaMonitor created object count evaluator for volumesnapshots.snapshot.storage.k8s.io Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.128919 11030 resource_quota_monitor.go:233] QuotaMonitor created object count evaluator for deployments.apps Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.128929 11030 controllermanager.go:605] Started "resourcequota" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.129047 11030 resource_quota_controller.go:273] Starting resource quota controller Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.129059 11030 shared_informer.go:240] Waiting for caches to sync for resource quota Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.129078 11030 resource_quota_monitor.go:308] QuotaMonitor running Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.130485 11030 controllermanager.go:605] Started "podgc" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.130593 11030 gc_controller.go:89] Starting GC controller Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.130607 11030 shared_informer.go:240] Waiting for caches to sync for GC Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.132822 11030 controllermanager.go:605] Started "daemonset" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.133122 11030 daemon_controller.go:284] Starting daemon sets controller Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.133131 11030 shared_informer.go:240] Waiting for caches to sync for daemon sets Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.138067 11030 controllermanager.go:605] Started "horizontalpodautoscaling" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.138357 11030 horizontal.go:168] Starting HPA controller Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.138367 11030 shared_informer.go:240] Waiting for caches to sync for HPA Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.139407 11030 controllermanager.go:605] Started "ttl" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.139490 11030 ttl_controller.go:121] Starting TTL controller Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.139497 11030 shared_informer.go:240] Waiting for caches to sync for TTL Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.141082 11030 node_lifecycle_controller.go:77] Sending events to api server Jul 24 08:51:34 bluth k3s[11030]: E0724 08:51:34.141108 11030 core.go:212] failed to start cloud node lifecycle controller: no cloud provider provided Jul 24 08:51:34 bluth k3s[11030]: W0724 08:51:34.141116 11030 controllermanager.go:583] Skipping "cloud-node-lifecycle" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.142745 11030 controllermanager.go:605] Started "endpointslicemirroring" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.143059 11030 endpointslicemirroring_controller.go:212] Starting EndpointSliceMirroring controller Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.143071 11030 shared_informer.go:240] Waiting for caches to sync for endpoint_slice_mirroring Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.144310 11030 controllermanager.go:605] Started "serviceaccount" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.144605 11030 serviceaccounts_controller.go:117] Starting service account controller Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.144618 11030 shared_informer.go:240] Waiting for caches to sync for service account Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.146922 11030 controllermanager.go:605] Started "disruption" Jul 24 08:51:34 bluth k3s[11030]: W0724 08:51:34.146933 11030 controllermanager.go:570] "bootstrapsigner" is disabled Jul 24 08:51:34 bluth k3s[11030]: W0724 08:51:34.147147 11030 core.go:226] configure-cloud-routes is set, but no cloud provider specified. Will not configure cloud provider routes. Jul 24 08:51:34 bluth k3s[11030]: W0724 08:51:34.147156 11030 controllermanager.go:583] Skipping "route" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.147222 11030 disruption.go:363] Starting disruption controller Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.147232 11030 shared_informer.go:240] Waiting for caches to sync for disruption Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.148844 11030 controllermanager.go:605] Started "attachdetach" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.149051 11030 attach_detach_controller.go:328] Starting attach detach controller Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.149061 11030 shared_informer.go:240] Waiting for caches to sync for attach detach Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.150480 11030 controllermanager.go:605] Started "replicationcontroller" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.150654 11030 replica_set.go:186] Starting replicationcontroller controller Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.150668 11030 shared_informer.go:240] Waiting for caches to sync for ReplicationController Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.152070 11030 controllermanager.go:605] Started "replicaset" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.152156 11030 replica_set.go:186] Starting replicaset controller Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.152162 11030 shared_informer.go:240] Waiting for caches to sync for ReplicaSet Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.153556 11030 controllermanager.go:605] Started "statefulset" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.153595 11030 stateful_set.go:147] Starting stateful set controller Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.153602 11030 shared_informer.go:240] Waiting for caches to sync for stateful set Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.154981 11030 controllermanager.go:605] Started "csrapproving" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.155059 11030 certificate_controller.go:118] Starting certificate controller "csrapproving" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.155067 11030 shared_informer.go:240] Waiting for caches to sync for certificate-csrapproving Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.156604 11030 controllermanager.go:605] Started "persistentvolume-binder" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.156705 11030 pv_controller_base.go:310] Starting persistent volume controller Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.156712 11030 shared_informer.go:240] Waiting for caches to sync for persistent volume Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.158030 11030 controllermanager.go:605] Started "persistentvolume-expander" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.158065 11030 expand_controller.go:342] Starting expand controller Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.158079 11030 shared_informer.go:240] Waiting for caches to sync for expand Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.159387 11030 controllermanager.go:605] Started "pvc-protection" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.159481 11030 pvc_protection_controller.go:103] "Starting PVC protection controller" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.159490 11030 shared_informer.go:240] Waiting for caches to sync for PVC protection Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.163413 11030 controllermanager.go:605] Started "garbagecollector" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.165360 11030 shared_informer.go:240] Waiting for caches to sync for resource quota Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.192714 11030 garbagecollector.go:146] Starting garbage collector controller Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.192778 11030 shared_informer.go:240] Waiting for caches to sync for garbage collector Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.192918 11030 graph_builder.go:289] GraphBuilder running Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.207349 11030 shared_informer.go:247] Caches are synced for cronjob Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.208702 11030 shared_informer.go:247] Caches are synced for certificate-csrsigning-kube-apiserver-client Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.209159 11030 shared_informer.go:247] Caches are synced for certificate-csrsigning-kubelet-client Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.209352 11030 shared_informer.go:247] Caches are synced for certificate-csrsigning-kubelet-serving Jul 24 08:51:34 bluth k3s[11030]: W0724 08:51:34.209960 11030 actual_state_of_world.go:539] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true, because nodeName="ix-truenas" does not exist Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.211627 11030 shared_informer.go:247] Caches are synced for certificate-csrsigning-legacy-unknown Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.211887 11030 shared_informer.go:247] Caches are synced for crt configmap Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.216365 11030 shared_informer.go:247] Caches are synced for node Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.216384 11030 range_allocator.go:173] Starting range CIDR allocator Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.216387 11030 shared_informer.go:240] Waiting for caches to sync for cidrallocator Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.216392 11030 shared_informer.go:247] Caches are synced for cidrallocator Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.217729 11030 shared_informer.go:247] Caches are synced for TTL after finished Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.230625 11030 shared_informer.go:247] Caches are synced for GC Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.232847 11030 shared_informer.go:240] Waiting for caches to sync for garbage collector Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.233305 11030 shared_informer.go:247] Caches are synced for daemon sets Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.238547 11030 shared_informer.go:247] Caches are synced for HPA Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.240117 11030 shared_informer.go:247] Caches are synced for TTL Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.245430 11030 shared_informer.go:247] Caches are synced for service account Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.249173 11030 shared_informer.go:247] Caches are synced for attach detach Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.251293 11030 shared_informer.go:247] Caches are synced for ReplicationController Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.253215 11030 shared_informer.go:247] Caches are synced for ReplicaSet Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.253669 11030 shared_informer.go:247] Caches are synced for stateful set Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.255204 11030 shared_informer.go:247] Caches are synced for certificate-csrapproving Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.257443 11030 shared_informer.go:247] Caches are synced for persistent volume Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.258176 11030 shared_informer.go:247] Caches are synced for expand Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.259515 11030 shared_informer.go:247] Caches are synced for PVC protection Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.275166 11030 shared_informer.go:247] Caches are synced for endpoint_slice Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.295455 11030 shared_informer.go:247] Caches are synced for namespace Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.296555 11030 shared_informer.go:247] Caches are synced for taint Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.296648 11030 node_lifecycle_controller.go:1397] Initializing eviction metric for zone: Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.296649 11030 taint_manager.go:187] "Starting NoExecuteTaintManager" Jul 24 08:51:34 bluth k3s[11030]: W0724 08:51:34.296799 11030 node_lifecycle_controller.go:1012] Missing timestamp for Node ix-truenas. Assuming now as a timestamp. Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.296826 11030 node_lifecycle_controller.go:1163] Controller detected that all Nodes are not-Ready. Entering master disruption mode. Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.296883 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="ix-qbittorrent/svclb-qbittorrent-ldv82" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.296925 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="kube-system/intel-gpu-plugin-fxs6q" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.296976 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="ix-esphome/svclb-esphome-q4lrj" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.297044 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="kube-system/coredns-d76bd69b-gk2vd" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.297078 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="ix-plex/plex-bd956645c-95gj8" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.297115 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="ix-tdarr/tdarr-746d65b466-b7whb" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.297151 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="ix-status-html/status-html-ix-chart-765b966694-9clbp" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.297180 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="ix-prowlarr/svclb-prowlarr-qhcm8" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.297218 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="ix-cloudflared/cloudflared-ix-chart-687fff6d55-t247w" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.297244 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="ix-radarr/svclb-radarr-nwf7v" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.297283 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="kube-system/intel-gpu-plugin-fxs6q" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.297304 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="ix-sonarr/svclb-sonarr-69zgj" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.297343 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-mz95k" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.297367 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="ix-qbittorrent/qbittorrent-6d7cf9f7cd-7n6c6" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.297405 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="ix-cloudflared/cloudflared-ix-chart-687fff6d55-z78xq" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.297429 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="ix-tdarr/svclb-tdarr-comm-npvsl" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.297467 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="ix-qbittorrent/svclb-qbittorrent-torrentudp-62ksx" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.297485 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="ix-sonarr/sonarr-867d7dd548-4bn8s" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.297527 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="ix-prowlarr/prowlarr-5bcc99cc47-rgrsl" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.297550 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="ix-qbittorrent/svclb-qbittorrent-torrent-8qgf8" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.297586 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="ix-radarr/radarr-7486f78d7f-q4pn4" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.297630 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="ix-esphome/esphome-658957fddc-bbcxl" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.297649 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="ix-esphome/esphome-658957fddc-bbcxl" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.297678 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="kube-system/openebs-zfs-node-bf7kk" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.297698 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="ix-prowlarr/svclb-prowlarr-qhcm8" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.297705 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="ix-esphome/esphome-658957fddc-sb8c8" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.297727 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="ix-radarr/svclb-radarr-nwf7v" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.297732 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="ix-tdarr/svclb-tdarr-44c97" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.297750 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="ix-sonarr/sonarr-867d7dd548-q6zsc" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.296884 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="kube-system/openebs-zfs-controller-0" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.304502 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="ix-cloudflared/cloudflared-ix-chart-687fff6d55-z78xq" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.304569 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="kube-system/openebs-zfs-node-bf7kk" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.304602 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="ix-esphome/esphome-658957fddc-sb8c8" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.304615 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="ix-tdarr/svclb-tdarr-comm-npvsl" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.304627 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="ix-qbittorrent/svclb-qbittorrent-torrentudp-62ksx" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.304658 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="ix-tdarr/svclb-tdarr-44c97" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.304670 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="ix-sonarr/sonarr-867d7dd548-4bn8s" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.304681 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="kube-system/openebs-zfs-controller-0" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.304709 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="ix-tdarr/tdarr-746d65b466-b7whb" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.304730 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="ix-prowlarr/prowlarr-5bcc99cc47-rgrsl" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.304749 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="ix-esphome/svclb-esphome-q4lrj" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.304782 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="ix-cloudflared/cloudflared-ix-chart-687fff6d55-t247w" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.304799 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="kube-system/coredns-d76bd69b-gk2vd" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.304813 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="ix-plex/plex-bd956645c-95gj8" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.304844 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="ix-radarr/radarr-7486f78d7f-q4pn4" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.304843 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="ix-qbittorrent/qbittorrent-6d7cf9f7cd-7n6c6" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.304859 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="ix-status-html/status-html-ix-chart-765b966694-9clbp" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.304873 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-mz95k" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.304890 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="ix-qbittorrent/svclb-qbittorrent-torrent-8qgf8" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.304906 11030 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="ix-sonarr/svclb-sonarr-69zgj" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307540 11030 event.go:294] "Event occurred" object="ix-truenas" kind="Node" apiVersion="v1" type="Normal" reason="RegisteredNode" message="Node ix-truenas event: Registered Node ix-truenas in Controller" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307544 11030 event.go:294] "Event occurred" object="ix-sonarr/sonarr-867d7dd548-q6zsc" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Cancelling deletion of Pod ix-sonarr/sonarr-867d7dd548-q6zsc" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307567 11030 shared_informer.go:247] Caches are synced for ephemeral Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307572 11030 event.go:294] "Event occurred" object="ix-qbittorrent/svclb-qbittorrent-ldv82" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Cancelling deletion of Pod ix-qbittorrent/svclb-qbittorrent-ldv82" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307579 11030 shared_informer.go:247] Caches are synced for ClusterRoleAggregator Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307580 11030 event.go:294] "Event occurred" object="ix-qbittorrent/svclb-qbittorrent-ldv82" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod ix-qbittorrent/svclb-qbittorrent-ldv82" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307588 11030 event.go:294] "Event occurred" object="ix-prowlarr/svclb-prowlarr-qhcm8" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Cancelling deletion of Pod ix-prowlarr/svclb-prowlarr-qhcm8" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307596 11030 event.go:294] "Event occurred" object="ix-radarr/svclb-radarr-nwf7v" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Cancelling deletion of Pod ix-radarr/svclb-radarr-nwf7v" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307604 11030 event.go:294] "Event occurred" object="kube-system/intel-gpu-plugin-fxs6q" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Cancelling deletion of Pod kube-system/intel-gpu-plugin-fxs6q" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307609 11030 event.go:294] "Event occurred" object="kube-system/intel-gpu-plugin-fxs6q" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod kube-system/intel-gpu-plugin-fxs6q" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307615 11030 event.go:294] "Event occurred" object="ix-esphome/svclb-esphome-q4lrj" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod ix-esphome/svclb-esphome-q4lrj" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307622 11030 event.go:294] "Event occurred" object="kube-system/coredns-d76bd69b-gk2vd" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod kube-system/coredns-d76bd69b-gk2vd" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307627 11030 event.go:294] "Event occurred" object="ix-plex/plex-bd956645c-95gj8" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod ix-plex/plex-bd956645c-95gj8" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307633 11030 event.go:294] "Event occurred" object="ix-tdarr/tdarr-746d65b466-b7whb" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod ix-tdarr/tdarr-746d65b466-b7whb" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307640 11030 event.go:294] "Event occurred" object="ix-status-html/status-html-ix-chart-765b966694-9clbp" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod ix-status-html/status-html-ix-chart-765b966694-9clbp" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307646 11030 event.go:294] "Event occurred" object="ix-prowlarr/svclb-prowlarr-qhcm8" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod ix-prowlarr/svclb-prowlarr-qhcm8" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307652 11030 event.go:294] "Event occurred" object="ix-cloudflared/cloudflared-ix-chart-687fff6d55-t247w" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod ix-cloudflared/cloudflared-ix-chart-687fff6d55-t247w" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307658 11030 event.go:294] "Event occurred" object="ix-radarr/svclb-radarr-nwf7v" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod ix-radarr/svclb-radarr-nwf7v" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307664 11030 event.go:294] "Event occurred" object="kube-system/intel-gpu-plugin-fxs6q" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod kube-system/intel-gpu-plugin-fxs6q" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307670 11030 event.go:294] "Event occurred" object="ix-sonarr/svclb-sonarr-69zgj" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod ix-sonarr/svclb-sonarr-69zgj" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307677 11030 event.go:294] "Event occurred" object="ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-mz95k" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-mz95k" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307685 11030 event.go:294] "Event occurred" object="ix-qbittorrent/qbittorrent-6d7cf9f7cd-7n6c6" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod ix-qbittorrent/qbittorrent-6d7cf9f7cd-7n6c6" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307691 11030 event.go:294] "Event occurred" object="ix-cloudflared/cloudflared-ix-chart-687fff6d55-z78xq" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod ix-cloudflared/cloudflared-ix-chart-687fff6d55-z78xq" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307697 11030 event.go:294] "Event occurred" object="ix-tdarr/svclb-tdarr-comm-npvsl" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod ix-tdarr/svclb-tdarr-comm-npvsl" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307703 11030 event.go:294] "Event occurred" object="ix-qbittorrent/svclb-qbittorrent-torrentudp-62ksx" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod ix-qbittorrent/svclb-qbittorrent-torrentudp-62ksx" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307709 11030 event.go:294] "Event occurred" object="ix-sonarr/sonarr-867d7dd548-4bn8s" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod ix-sonarr/sonarr-867d7dd548-4bn8s" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307715 11030 event.go:294] "Event occurred" object="ix-prowlarr/prowlarr-5bcc99cc47-rgrsl" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod ix-prowlarr/prowlarr-5bcc99cc47-rgrsl" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307721 11030 event.go:294] "Event occurred" object="ix-qbittorrent/svclb-qbittorrent-torrent-8qgf8" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod ix-qbittorrent/svclb-qbittorrent-torrent-8qgf8" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307728 11030 event.go:294] "Event occurred" object="ix-radarr/radarr-7486f78d7f-q4pn4" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod ix-radarr/radarr-7486f78d7f-q4pn4" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307734 11030 event.go:294] "Event occurred" object="ix-cloudflared/cloudflared-ix-chart-687fff6d55-z78xq" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Cancelling deletion of Pod ix-cloudflared/cloudflared-ix-chart-687fff6d55-z78xq" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307741 11030 event.go:294] "Event occurred" object="ix-qbittorrent/qbittorrent-6d7cf9f7cd-7n6c6" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Cancelling deletion of Pod ix-qbittorrent/qbittorrent-6d7cf9f7cd-7n6c6" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307748 11030 event.go:294] "Event occurred" object="ix-esphome/esphome-658957fddc-bbcxl" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Cancelling deletion of Pod ix-esphome/esphome-658957fddc-bbcxl" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307754 11030 event.go:294] "Event occurred" object="ix-esphome/esphome-658957fddc-bbcxl" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod ix-esphome/esphome-658957fddc-bbcxl" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307760 11030 event.go:294] "Event occurred" object="ix-esphome/esphome-658957fddc-bbcxl" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod ix-esphome/esphome-658957fddc-bbcxl" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307766 11030 event.go:294] "Event occurred" object="kube-system/openebs-zfs-node-bf7kk" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod kube-system/openebs-zfs-node-bf7kk" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307771 11030 event.go:294] "Event occurred" object="ix-prowlarr/svclb-prowlarr-qhcm8" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod ix-prowlarr/svclb-prowlarr-qhcm8" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307777 11030 event.go:294] "Event occurred" object="ix-esphome/esphome-658957fddc-sb8c8" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod ix-esphome/esphome-658957fddc-sb8c8" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307782 11030 event.go:294] "Event occurred" object="ix-radarr/svclb-radarr-nwf7v" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod ix-radarr/svclb-radarr-nwf7v" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307788 11030 event.go:294] "Event occurred" object="ix-tdarr/svclb-tdarr-44c97" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod ix-tdarr/svclb-tdarr-44c97" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307794 11030 event.go:294] "Event occurred" object="ix-sonarr/sonarr-867d7dd548-q6zsc" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod ix-sonarr/sonarr-867d7dd548-q6zsc" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307799 11030 event.go:294] "Event occurred" object="kube-system/openebs-zfs-controller-0" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod kube-system/openebs-zfs-controller-0" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307806 11030 event.go:294] "Event occurred" object="ix-cloudflared/cloudflared-ix-chart-687fff6d55-z78xq" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod ix-cloudflared/cloudflared-ix-chart-687fff6d55-z78xq" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307811 11030 event.go:294] "Event occurred" object="ix-tdarr/svclb-tdarr-comm-npvsl" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Cancelling deletion of Pod ix-tdarr/svclb-tdarr-comm-npvsl" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307817 11030 event.go:294] "Event occurred" object="ix-qbittorrent/svclb-qbittorrent-torrentudp-62ksx" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Cancelling deletion of Pod ix-qbittorrent/svclb-qbittorrent-torrentudp-62ksx" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307822 11030 event.go:294] "Event occurred" object="kube-system/openebs-zfs-node-bf7kk" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Cancelling deletion of Pod kube-system/openebs-zfs-node-bf7kk" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307828 11030 event.go:294] "Event occurred" object="kube-system/openebs-zfs-node-bf7kk" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod kube-system/openebs-zfs-node-bf7kk" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307834 11030 event.go:294] "Event occurred" object="ix-sonarr/sonarr-867d7dd548-4bn8s" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Cancelling deletion of Pod ix-sonarr/sonarr-867d7dd548-4bn8s" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307840 11030 event.go:294] "Event occurred" object="kube-system/openebs-zfs-controller-0" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Cancelling deletion of Pod kube-system/openebs-zfs-controller-0" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307845 11030 event.go:294] "Event occurred" object="ix-esphome/esphome-658957fddc-sb8c8" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Cancelling deletion of Pod ix-esphome/esphome-658957fddc-sb8c8" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307851 11030 event.go:294] "Event occurred" object="ix-esphome/esphome-658957fddc-sb8c8" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod ix-esphome/esphome-658957fddc-sb8c8" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307856 11030 event.go:294] "Event occurred" object="ix-tdarr/svclb-tdarr-comm-npvsl" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod ix-tdarr/svclb-tdarr-comm-npvsl" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307864 11030 event.go:294] "Event occurred" object="ix-qbittorrent/svclb-qbittorrent-torrentudp-62ksx" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod ix-qbittorrent/svclb-qbittorrent-torrentudp-62ksx" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307870 11030 event.go:294] "Event occurred" object="ix-prowlarr/prowlarr-5bcc99cc47-rgrsl" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Cancelling deletion of Pod ix-prowlarr/prowlarr-5bcc99cc47-rgrsl" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307875 11030 event.go:294] "Event occurred" object="ix-esphome/svclb-esphome-q4lrj" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Cancelling deletion of Pod ix-esphome/svclb-esphome-q4lrj" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307881 11030 event.go:294] "Event occurred" object="ix-tdarr/svclb-tdarr-44c97" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Cancelling deletion of Pod ix-tdarr/svclb-tdarr-44c97" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307886 11030 event.go:294] "Event occurred" object="ix-tdarr/svclb-tdarr-44c97" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod ix-tdarr/svclb-tdarr-44c97" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307892 11030 event.go:294] "Event occurred" object="ix-sonarr/sonarr-867d7dd548-4bn8s" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod ix-sonarr/sonarr-867d7dd548-4bn8s" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307898 11030 event.go:294] "Event occurred" object="kube-system/openebs-zfs-controller-0" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod kube-system/openebs-zfs-controller-0" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307904 11030 event.go:294] "Event occurred" object="kube-system/coredns-d76bd69b-gk2vd" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Cancelling deletion of Pod kube-system/coredns-d76bd69b-gk2vd" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307909 11030 event.go:294] "Event occurred" object="ix-plex/plex-bd956645c-95gj8" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Cancelling deletion of Pod ix-plex/plex-bd956645c-95gj8" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307914 11030 event.go:294] "Event occurred" object="ix-tdarr/tdarr-746d65b466-b7whb" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Cancelling deletion of Pod ix-tdarr/tdarr-746d65b466-b7whb" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307920 11030 event.go:294] "Event occurred" object="ix-tdarr/tdarr-746d65b466-b7whb" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod ix-tdarr/tdarr-746d65b466-b7whb" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307925 11030 event.go:294] "Event occurred" object="ix-prowlarr/prowlarr-5bcc99cc47-rgrsl" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod ix-prowlarr/prowlarr-5bcc99cc47-rgrsl" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307931 11030 event.go:294] "Event occurred" object="ix-esphome/svclb-esphome-q4lrj" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod ix-esphome/svclb-esphome-q4lrj" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307937 11030 event.go:294] "Event occurred" object="ix-status-html/status-html-ix-chart-765b966694-9clbp" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Cancelling deletion of Pod ix-status-html/status-html-ix-chart-765b966694-9clbp" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307943 11030 event.go:294] "Event occurred" object="ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-mz95k" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Cancelling deletion of Pod ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-mz95k" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307949 11030 event.go:294] "Event occurred" object="ix-cloudflared/cloudflared-ix-chart-687fff6d55-t247w" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Cancelling deletion of Pod ix-cloudflared/cloudflared-ix-chart-687fff6d55-t247w" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307956 11030 event.go:294] "Event occurred" object="ix-cloudflared/cloudflared-ix-chart-687fff6d55-t247w" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod ix-cloudflared/cloudflared-ix-chart-687fff6d55-t247w" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307962 11030 event.go:294] "Event occurred" object="kube-system/coredns-d76bd69b-gk2vd" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod kube-system/coredns-d76bd69b-gk2vd" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307971 11030 event.go:294] "Event occurred" object="ix-plex/plex-bd956645c-95gj8" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod ix-plex/plex-bd956645c-95gj8" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307976 11030 event.go:294] "Event occurred" object="ix-qbittorrent/svclb-qbittorrent-torrent-8qgf8" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Cancelling deletion of Pod ix-qbittorrent/svclb-qbittorrent-torrent-8qgf8" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307982 11030 event.go:294] "Event occurred" object="ix-sonarr/svclb-sonarr-69zgj" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Cancelling deletion of Pod ix-sonarr/svclb-sonarr-69zgj" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307988 11030 event.go:294] "Event occurred" object="ix-radarr/radarr-7486f78d7f-q4pn4" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Cancelling deletion of Pod ix-radarr/radarr-7486f78d7f-q4pn4" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307993 11030 event.go:294] "Event occurred" object="ix-radarr/radarr-7486f78d7f-q4pn4" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod ix-radarr/radarr-7486f78d7f-q4pn4" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.307999 11030 event.go:294] "Event occurred" object="ix-qbittorrent/qbittorrent-6d7cf9f7cd-7n6c6" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod ix-qbittorrent/qbittorrent-6d7cf9f7cd-7n6c6" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.308005 11030 event.go:294] "Event occurred" object="ix-status-html/status-html-ix-chart-765b966694-9clbp" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod ix-status-html/status-html-ix-chart-765b966694-9clbp" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.308011 11030 event.go:294] "Event occurred" object="ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-mz95k" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-mz95k" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.308017 11030 event.go:294] "Event occurred" object="ix-qbittorrent/svclb-qbittorrent-torrent-8qgf8" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod ix-qbittorrent/svclb-qbittorrent-torrent-8qgf8" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.308022 11030 event.go:294] "Event occurred" object="ix-sonarr/svclb-sonarr-69zgj" kind="Pod" apiVersion="" type="Normal" reason="TaintManagerEviction" message="Marking for deletion Pod ix-sonarr/svclb-sonarr-69zgj" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.308035 11030 shared_informer.go:247] Caches are synced for job Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.319115 11030 shared_informer.go:247] Caches are synced for PV protection Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.321166 11030 cni.go:334] "CNI failed to retrieve network namespace path" err="cannot find network namespace for the terminated container \"d2f132034b94577c4f38f3822a64e2609c355bbe34a9181ea5e259031c0ac943\"" Jul 24 08:51:34 bluth k3s[11030]: time="2022-07-24T08:51:34+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-sonarr\", Name:\"sonarr\", UID:\"b24571a2-a018-4716-ab7d-d6ab31e66e44\", APIVersion:\"v1\", ResourceVersion:\"2334145\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-sonarr/svclb-sonarr" Jul 24 08:51:34 bluth k3s[11030]: time="2022-07-24T08:51:34+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-qbittorrent\", Name:\"qbittorrent-torrentudp\", UID:\"1d92e74f-b912-46f1-b780-c740f80bb9ca\", APIVersion:\"v1\", ResourceVersion:\"2504584\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-qbittorrent/svclb-qbittorrent-torrentudp" Jul 24 08:51:34 bluth k3s[11030]: time="2022-07-24T08:51:34+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-tdarr\", Name:\"tdarr-comm\", UID:\"c3b33312-b11b-4e16-915e-205202cc0a2d\", APIVersion:\"v1\", ResourceVersion:\"2504455\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-tdarr/svclb-tdarr-comm" Jul 24 08:51:34 bluth k3s[11030]: time="2022-07-24T08:51:34+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-prowlarr\", Name:\"prowlarr\", UID:\"bd43faf9-73db-4bcd-a678-c673a7b54098\", APIVersion:\"v1\", ResourceVersion:\"2334148\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-prowlarr/svclb-prowlarr" Jul 24 08:51:34 bluth k3s[11030]: time="2022-07-24T08:51:34+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-qbittorrent\", Name:\"qbittorrent-torrent\", UID:\"29b172f8-d799-4e0c-aac7-78ec5af5c01b\", APIVersion:\"v1\", ResourceVersion:\"2334164\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-qbittorrent/svclb-qbittorrent-torrent" Jul 24 08:51:34 bluth k3s[11030]: time="2022-07-24T08:51:34+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-radarr\", Name:\"radarr\", UID:\"2ceba47d-fa27-462f-9816-af4352781795\", APIVersion:\"v1\", ResourceVersion:\"2504670\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-radarr/svclb-radarr" Jul 24 08:51:34 bluth k3s[11030]: time="2022-07-24T08:51:34+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-esphome\", Name:\"esphome\", UID:\"9d7f07bb-0ed3-4a9b-b9b3-ce707e119669\", APIVersion:\"v1\", ResourceVersion:\"2334230\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-esphome/svclb-esphome" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.343232 11030 shared_informer.go:247] Caches are synced for endpoint_slice_mirroring Jul 24 08:51:34 bluth k3s[11030]: time="2022-07-24T08:51:34+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-tdarr\", Name:\"tdarr\", UID:\"90709eb9-3d84-4b91-b6ab-73ece8ffcb86\", APIVersion:\"v1\", ResourceVersion:\"2504489\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-tdarr/svclb-tdarr" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.397784 11030 shared_informer.go:247] Caches are synced for endpoint Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.429319 11030 shared_informer.go:247] Caches are synced for resource quota Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.447292 11030 shared_informer.go:247] Caches are synced for disruption Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.447334 11030 disruption.go:371] Sending events to api server. Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.457878 11030 shared_informer.go:247] Caches are synced for deployment Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.465548 11030 shared_informer.go:247] Caches are synced for resource quota Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.477544 11030 event.go:294] "Event occurred" object="ix-radarr/svclb-radarr" kind="DaemonSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulDelete" message="Deleted pod: svclb-radarr-nwf7v" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.477592 11030 event.go:294] "Event occurred" object="kube-system/intel-gpu-plugin" kind="DaemonSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulDelete" message="Deleted pod: intel-gpu-plugin-fxs6q" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.571722 11030 event.go:294] "Event occurred" object="ix-cloudflared/cloudflared-ix-chart-687fff6d55" kind="ReplicaSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulCreate" message="Created pod: cloudflared-ix-chart-687fff6d55-vzff2" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.571740 11030 event.go:294] "Event occurred" object="ix-status-html/status-html-ix-chart-765b966694" kind="ReplicaSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulCreate" message="Created pod: status-html-ix-chart-765b966694-c4nwz" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.573110 11030 controller.go:611] quota admission added evaluator for: events.events.k8s.io Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.573278 11030 event.go:294] "Event occurred" object="ix-plex/plex-bd956645c" kind="ReplicaSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulCreate" message="Created pod: plex-bd956645c-wxsbd" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.574028 11030 event.go:294] "Event occurred" object="ix-radarr/radarr-7486f78d7f" kind="ReplicaSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulCreate" message="Created pod: radarr-7486f78d7f-ntt44" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.576378 11030 event.go:294] "Event occurred" object="kube-system/coredns-d76bd69b" kind="ReplicaSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulCreate" message="Created pod: coredns-d76bd69b-mhb5r" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.586821 11030 event.go:294] "Event occurred" object="ix-esphome/esphome-658957fddc" kind="ReplicaSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulCreate" message="Created pod: esphome-658957fddc-6z2jm" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.590878 11030 event.go:294] "Event occurred" object="ix-home-assistant/home-assistant-ix-chart-54bf5d89c8" kind="ReplicaSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulCreate" message="Created pod: home-assistant-ix-chart-54bf5d89c8-5rpjh" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.594523 11030 event.go:294] "Event occurred" object="ix-sonarr/sonarr-867d7dd548" kind="ReplicaSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulCreate" message="Created pod: sonarr-867d7dd548-fw6z4" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.595471 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-3706f211-9694-4ec9-88b9-55f66b73b802\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-3706f211-9694-4ec9-88b9-55f66b73b802\") pod \"tdarr-746d65b466-b7whb\" (UID: \"6e3e9791-9bb3-487f-ae50-048fefbef8be\") " pod="ix-tdarr/tdarr-746d65b466-b7whb" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.600732 11030 event.go:294] "Event occurred" object="ix-qbittorrent/qbittorrent-6d7cf9f7cd" kind="ReplicaSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulCreate" message="Created pod: qbittorrent-6d7cf9f7cd-xrqwj" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.605180 11030 event.go:294] "Event occurred" object="ix-tdarr/tdarr-746d65b466" kind="ReplicaSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulCreate" message="Created pod: tdarr-746d65b466-vdnf9" Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.816317 11030 controller.go:611] quota admission added evaluator for: endpointslices.discovery.k8s.io Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.893305 11030 shared_informer.go:247] Caches are synced for garbage collector Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.893346 11030 garbagecollector.go:155] Garbage collector: all resource monitors have synced. Proceeding to collect garbage Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.933195 11030 shared_informer.go:247] Caches are synced for garbage collector Jul 24 08:51:34 bluth k3s[11030]: I0724 08:51:34.942658 11030 controller.go:611] quota admission added evaluator for: endpoints Jul 24 08:51:34 bluth k3s[11030]: time="2022-07-24T08:51:34+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-tdarr\", Name:\"tdarr\", UID:\"90709eb9-3d84-4b91-b6ab-73ece8ffcb86\", APIVersion:\"v1\", ResourceVersion:\"2504489\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-tdarr/svclb-tdarr" Jul 24 08:51:34 bluth k3s[11030]: time="2022-07-24T08:51:34+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-sonarr\", Name:\"sonarr\", UID:\"b24571a2-a018-4716-ab7d-d6ab31e66e44\", APIVersion:\"v1\", ResourceVersion:\"2334145\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-sonarr/svclb-sonarr" Jul 24 08:51:34 bluth k3s[11030]: time="2022-07-24T08:51:34+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-qbittorrent\", Name:\"qbittorrent-torrent\", UID:\"29b172f8-d799-4e0c-aac7-78ec5af5c01b\", APIVersion:\"v1\", ResourceVersion:\"2334164\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-qbittorrent/svclb-qbittorrent-torrent" Jul 24 08:51:34 bluth k3s[11030]: time="2022-07-24T08:51:34+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-radarr\", Name:\"radarr\", UID:\"2ceba47d-fa27-462f-9816-af4352781795\", APIVersion:\"v1\", ResourceVersion:\"2504670\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-radarr/svclb-radarr" Jul 24 08:51:34 bluth k3s[11030]: time="2022-07-24T08:51:34+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-qbittorrent\", Name:\"qbittorrent-torrentudp\", UID:\"1d92e74f-b912-46f1-b780-c740f80bb9ca\", APIVersion:\"v1\", ResourceVersion:\"2504584\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-qbittorrent/svclb-qbittorrent-torrentudp" Jul 24 08:51:34 bluth k3s[11030]: time="2022-07-24T08:51:34+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-esphome\", Name:\"esphome\", UID:\"9d7f07bb-0ed3-4a9b-b9b3-ce707e119669\", APIVersion:\"v1\", ResourceVersion:\"2334230\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-esphome/svclb-esphome" Jul 24 08:51:34 bluth k3s[11030]: time="2022-07-24T08:51:34+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-qbittorrent\", Name:\"qbittorrent\", UID:\"728a4977-c72e-46a0-96f2-a9279b009171\", APIVersion:\"v1\", ResourceVersion:\"2334264\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-qbittorrent/svclb-qbittorrent" Jul 24 08:51:34 bluth k3s[11030]: time="2022-07-24T08:51:34+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-tdarr\", Name:\"tdarr-comm\", UID:\"c3b33312-b11b-4e16-915e-205202cc0a2d\", APIVersion:\"v1\", ResourceVersion:\"2504455\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-tdarr/svclb-tdarr-comm" Jul 24 08:51:34 bluth k3s[11030]: time="2022-07-24T08:51:34+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-qbittorrent\", Name:\"qbittorrent\", UID:\"728a4977-c72e-46a0-96f2-a9279b009171\", APIVersion:\"v1\", ResourceVersion:\"2334264\", FieldPath:\"\"}): type: 'Normal' reason: 'UpdatedIngressIP' LoadBalancer Ingress IP addresses updated: " Jul 24 08:51:34 bluth k3s[11030]: time="2022-07-24T08:51:34+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-prowlarr\", Name:\"prowlarr\", UID:\"bd43faf9-73db-4bcd-a678-c673a7b54098\", APIVersion:\"v1\", ResourceVersion:\"2334148\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-prowlarr/svclb-prowlarr" Jul 24 08:51:34 bluth k3s[11030]: time="2022-07-24T08:51:34+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-qbittorrent\", Name:\"qbittorrent\", UID:\"728a4977-c72e-46a0-96f2-a9279b009171\", APIVersion:\"v1\", ResourceVersion:\"3027187\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-qbittorrent/svclb-qbittorrent" Jul 24 08:51:35 bluth k3s[11030]: I0724 08:51:35.399191 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp\" (UniqueName: \"kubernetes.io/empty-dir/6e3e9791-9bb3-487f-ae50-048fefbef8be-temp\") pod \"tdarr-746d65b466-b7whb\" (UID: \"6e3e9791-9bb3-487f-ae50-048fefbef8be\") " pod="ix-tdarr/tdarr-746d65b466-b7whb" Jul 24 08:51:35 bluth k3s[11030]: I0724 08:51:35.399390 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"shared\" (UniqueName: \"kubernetes.io/empty-dir/6e3e9791-9bb3-487f-ae50-048fefbef8be-shared\") pod \"tdarr-746d65b466-b7whb\" (UID: \"6e3e9791-9bb3-487f-ae50-048fefbef8be\") " pod="ix-tdarr/tdarr-746d65b466-b7whb" Jul 24 08:51:35 bluth k3s[11030]: I0724 08:51:35.399496 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-4e6fcc6b-6890-4751-a09a-75bb6cde146f\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-4e6fcc6b-6890-4751-a09a-75bb6cde146f\") pod \"tdarr-746d65b466-b7whb\" (UID: \"6e3e9791-9bb3-487f-ae50-048fefbef8be\") " pod="ix-tdarr/tdarr-746d65b466-b7whb" Jul 24 08:51:35 bluth k3s[11030]: I0724 08:51:35.760466 11030 kubelet_volumes.go:133] "Cleaned up orphaned volume from pod" podUID=1f5bcb97-e684-4d8a-904e-1fdbd19a42f9 path="/var/lib/kubelet/pods/1f5bcb97-e684-4d8a-904e-1fdbd19a42f9/volumes/kubernetes.io~projected/kube-api-access-rdnmb" Jul 24 08:51:35 bluth k3s[11030]: I0724 08:51:35.761000 11030 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=1f5bcb97-e684-4d8a-904e-1fdbd19a42f9 path="/var/lib/kubelet/pods/1f5bcb97-e684-4d8a-904e-1fdbd19a42f9/volumes" Jul 24 08:51:35 bluth k3s[11030]: I0724 08:51:35.927376 11030 trace.go:205] Trace[185751053]: "Create" url:/api/v1/namespaces/ix-prowlarr/pods,user-agent:k3s/v1.23.5+k3s (linux/amd64) kubernetes/fbfa51e/system:serviceaccount:kube-system:replicaset-controller,audit-id:e305dd1c-e1b6-4216-a505-573e99468a8d,client:127.0.0.1,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (24-Jul-2022 08:51:34.580) (total time: 1346ms): Jul 24 08:51:35 bluth k3s[11030]: Trace[185751053]: ---"Object stored in database" 1345ms (08:51:35.926) Jul 24 08:51:35 bluth k3s[11030]: Trace[185751053]: [1.346132573s] [1.346132573s] END Jul 24 08:51:35 bluth k3s[11030]: I0724 08:51:35.928195 11030 event.go:294] "Event occurred" object="ix-prowlarr/prowlarr-5bcc99cc47" kind="ReplicaSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulCreate" message="Created pod: prowlarr-5bcc99cc47-v86hm" Jul 24 08:51:36 bluth k3s[11030]: I0724 08:51:36.207649 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"varrun\" (UniqueName: \"kubernetes.io/empty-dir/6e3e9791-9bb3-487f-ae50-048fefbef8be-varrun\") pod \"tdarr-746d65b466-b7whb\" (UID: \"6e3e9791-9bb3-487f-ae50-048fefbef8be\") " pod="ix-tdarr/tdarr-746d65b466-b7whb" Jul 24 08:51:36 bluth k3s[11030]: I0724 08:51:36.207772 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56dxr\" (UniqueName: \"kubernetes.io/projected/6e3e9791-9bb3-487f-ae50-048fefbef8be-kube-api-access-56dxr\") pod \"tdarr-746d65b466-b7whb\" (UID: \"6e3e9791-9bb3-487f-ae50-048fefbef8be\") " pod="ix-tdarr/tdarr-746d65b466-b7whb" Jul 24 08:51:36 bluth k3s[11030]: I0724 08:51:36.207837 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"varlogs\" (UniqueName: \"kubernetes.io/empty-dir/6e3e9791-9bb3-487f-ae50-048fefbef8be-varlogs\") pod \"tdarr-746d65b466-b7whb\" (UID: \"6e3e9791-9bb3-487f-ae50-048fefbef8be\") " pod="ix-tdarr/tdarr-746d65b466-b7whb" Jul 24 08:51:36 bluth k3s[11030]: I0724 08:51:36.208070 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-c41ca0c8-d532-4ca2-b6ca-fe3e9ddf87d2\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c41ca0c8-d532-4ca2-b6ca-fe3e9ddf87d2\") pod \"tdarr-746d65b466-b7whb\" (UID: \"6e3e9791-9bb3-487f-ae50-048fefbef8be\") " pod="ix-tdarr/tdarr-746d65b466-b7whb" Jul 24 08:51:36 bluth k3s[11030]: I0724 08:51:36.978985 11030 trace.go:205] Trace[963380169]: "Create" url:/api/v1/namespaces/ix-prowlarr/events,user-agent:k3s/v1.23.5+k3s (linux/amd64) kubernetes/fbfa51e/system:serviceaccount:kube-system:replicaset-controller,audit-id:988d5602-80d6-42de-bc44-3aa22c245a60,client:127.0.0.1,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (24-Jul-2022 08:51:35.929) (total time: 1049ms): Jul 24 08:51:36 bluth k3s[11030]: Trace[963380169]: ---"Object stored in database" 1049ms (08:51:36.978) Jul 24 08:51:36 bluth k3s[11030]: Trace[963380169]: [1.049622153s] [1.049622153s] END Jul 24 08:51:37 bluth k3s[11030]: I0724 08:51:37.053859 11030 scope.go:110] "RemoveContainer" containerID="83de9bb2e016582dcd1febf8fe41eaaf734e62ad8e27a5befa1c8287cd8cc1a2" Jul 24 08:51:37 bluth k3s[11030]: E0724 08:51:37.064308 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:39.064271653 +0100 IST m=+25.552128579 (durationBeforeRetry 2s). Error: MountVolume.MountDevice failed for volume "pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274") pod "prowlarr-5bcc99cc47-rgrsl" (UID: "afd760a7-8b6b-470e-aa87-8ed16adb34bb") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:37 bluth k3s[11030]: E0724 08:51:37.064524 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-2e6b3d10-435c-4ce5-806d-61d551dcef05 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:37.5644916 +0100 IST m=+24.052348523 (durationBeforeRetry 500ms). Error: MountVolume.MountDevice failed for volume "pvc-2e6b3d10-435c-4ce5-806d-61d551dcef05" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-2e6b3d10-435c-4ce5-806d-61d551dcef05") pod "tdarr-746d65b466-b7whb" (UID: "6e3e9791-9bb3-487f-ae50-048fefbef8be") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:37 bluth k3s[11030]: E0724 08:51:37.064773 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-028dc34a-8c2e-4a0a-b507-c33a02541407 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:38.064658747 +0100 IST m=+24.552515674 (durationBeforeRetry 1s). Error: MountVolume.MountDevice failed for volume "pvc-028dc34a-8c2e-4a0a-b507-c33a02541407" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-028dc34a-8c2e-4a0a-b507-c33a02541407") pod "tdarr-746d65b466-b7whb" (UID: "6e3e9791-9bb3-487f-ae50-048fefbef8be") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:37 bluth k3s[11030]: E0724 08:51:37.065111 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-4e6fcc6b-6890-4751-a09a-75bb6cde146f podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:37.565079488 +0100 IST m=+24.052936413 (durationBeforeRetry 500ms). Error: MountVolume.MountDevice failed for volume "pvc-4e6fcc6b-6890-4751-a09a-75bb6cde146f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-4e6fcc6b-6890-4751-a09a-75bb6cde146f") pod "tdarr-746d65b466-b7whb" (UID: "6e3e9791-9bb3-487f-ae50-048fefbef8be") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:37 bluth k3s[11030]: E0724 08:51:37.065166 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-3706f211-9694-4ec9-88b9-55f66b73b802 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:37.565144533 +0100 IST m=+24.053001457 (durationBeforeRetry 500ms). Error: MountVolume.MountDevice failed for volume "pvc-3706f211-9694-4ec9-88b9-55f66b73b802" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-3706f211-9694-4ec9-88b9-55f66b73b802") pod "tdarr-746d65b466-b7whb" (UID: "6e3e9791-9bb3-487f-ae50-048fefbef8be") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:37 bluth k3s[11030]: E0724 08:51:37.065201 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-195f1a32-9c93-4737-983a-8dacee32e453 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:39.065182218 +0100 IST m=+25.553039144 (durationBeforeRetry 2s). Error: MountVolume.MountDevice failed for volume "pvc-195f1a32-9c93-4737-983a-8dacee32e453" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-195f1a32-9c93-4737-983a-8dacee32e453") pod "radarr-7486f78d7f-q4pn4" (UID: "be152b2b-fd1c-4a36-9081-ac6be2fb7901") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:37 bluth k3s[11030]: E0724 08:51:37.065249 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c41ca0c8-d532-4ca2-b6ca-fe3e9ddf87d2 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:37.565222338 +0100 IST m=+24.053079265 (durationBeforeRetry 500ms). Error: MountVolume.MountDevice failed for volume "pvc-c41ca0c8-d532-4ca2-b6ca-fe3e9ddf87d2" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c41ca0c8-d532-4ca2-b6ca-fe3e9ddf87d2") pod "tdarr-746d65b466-b7whb" (UID: "6e3e9791-9bb3-487f-ae50-048fefbef8be") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:37 bluth k3s[11030]: E0724 08:51:37.066809 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:39.066778841 +0100 IST m=+25.554635767 (durationBeforeRetry 2s). Error: MountVolume.MountDevice failed for volume "pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4") pod "sonarr-867d7dd548-4bn8s" (UID: "35ac75bf-b85f-4286-a228-ea8459dde305") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:37 bluth k3s[11030]: I0724 08:51:37.072935 11030 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=6e3059a0-723f-4d0b-8c8d-014da4503ed2 path="/var/lib/kubelet/pods/6e3059a0-723f-4d0b-8c8d-014da4503ed2/volumes" Jul 24 08:51:37 bluth k3s[11030]: I0724 08:51:37.076949 11030 kubelet_volumes.go:133] "Cleaned up orphaned volume from pod" podUID=8af245a0-51e1-46d0-bfce-c0c41307e43f path="/var/lib/kubelet/pods/8af245a0-51e1-46d0-bfce-c0c41307e43f/volumes/kubernetes.io~csi/pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4/mount" Jul 24 08:51:37 bluth k3s[11030]: I0724 08:51:37.078979 11030 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=a8783016-2cef-4391-bf18-fe88e39bc01e path="/var/lib/kubelet/pods/a8783016-2cef-4391-bf18-fe88e39bc01e/volumes" Jul 24 08:51:37 bluth k3s[11030]: E0724 08:51:37.080377 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:51:37 bluth k3s[11030]: I0724 08:51:37.080583 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:51:37 bluth k3s[11030]: I0724 08:51:37.215218 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e\") pod \"esphome-658957fddc-sb8c8\" (UID: \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\") " pod="ix-esphome/esphome-658957fddc-sb8c8" Jul 24 08:51:37 bluth k3s[11030]: I0724 08:51:37.762713 11030 kubelet_volumes.go:133] "Cleaned up orphaned volume from pod" podUID=e6cd7c21-4755-4e8e-93fc-f4fab0fbd811 path="/var/lib/kubelet/pods/e6cd7c21-4755-4e8e-93fc-f4fab0fbd811/volumes/kubernetes.io~projected/kube-api-access-74hhc" Jul 24 08:51:37 bluth k3s[11030]: I0724 08:51:37.763380 11030 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=e6cd7c21-4755-4e8e-93fc-f4fab0fbd811 path="/var/lib/kubelet/pods/e6cd7c21-4755-4e8e-93fc-f4fab0fbd811/volumes" Jul 24 08:51:38 bluth k3s[11030]: I0724 08:51:38.023889 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"shared\" (UniqueName: \"kubernetes.io/empty-dir/05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b-shared\") pod \"esphome-658957fddc-sb8c8\" (UID: \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\") " pod="ix-esphome/esphome-658957fddc-sb8c8" Jul 24 08:51:38 bluth k3s[11030]: I0724 08:51:38.023981 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp\" (UniqueName: \"kubernetes.io/empty-dir/05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b-temp\") pod \"esphome-658957fddc-sb8c8\" (UID: \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\") " pod="ix-esphome/esphome-658957fddc-sb8c8" Jul 24 08:51:38 bluth k3s[11030]: I0724 08:51:38.024191 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-06521873-c179-499e-9079-194ec6618fdd\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-06521873-c179-499e-9079-194ec6618fdd\") pod \"esphome-658957fddc-sb8c8\" (UID: \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\") " pod="ix-esphome/esphome-658957fddc-sb8c8" Jul 24 08:51:38 bluth k3s[11030]: I0724 08:51:38.024411 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"varlogs\" (UniqueName: \"kubernetes.io/empty-dir/05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b-varlogs\") pod \"esphome-658957fddc-sb8c8\" (UID: \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\") " pod="ix-esphome/esphome-658957fddc-sb8c8" Jul 24 08:51:38 bluth k3s[11030]: I0724 08:51:38.024524 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr6bj\" (UniqueName: \"kubernetes.io/projected/05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b-kube-api-access-xr6bj\") pod \"esphome-658957fddc-sb8c8\" (UID: \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\") " pod="ix-esphome/esphome-658957fddc-sb8c8" Jul 24 08:51:38 bluth k3s[11030]: E0724 08:51:38.270971 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:38.770946097 +0100 IST m=+25.258803006 (durationBeforeRetry 500ms). Error: MountVolume.MountDevice failed for volume "pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e") pod "esphome-658957fddc-sb8c8" (UID: "05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:38 bluth k3s[11030]: E0724 08:51:38.271023 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c41ca0c8-d532-4ca2-b6ca-fe3e9ddf87d2 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:39.271007657 +0100 IST m=+25.758864565 (durationBeforeRetry 1s). Error: MountVolume.MountDevice failed for volume "pvc-c41ca0c8-d532-4ca2-b6ca-fe3e9ddf87d2" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c41ca0c8-d532-4ca2-b6ca-fe3e9ddf87d2") pod "tdarr-746d65b466-b7whb" (UID: "6e3e9791-9bb3-487f-ae50-048fefbef8be") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:38 bluth k3s[11030]: E0724 08:51:38.271369 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-06521873-c179-499e-9079-194ec6618fdd podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:38.771350111 +0100 IST m=+25.259207018 (durationBeforeRetry 500ms). Error: MountVolume.MountDevice failed for volume "pvc-06521873-c179-499e-9079-194ec6618fdd" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-06521873-c179-499e-9079-194ec6618fdd") pod "esphome-658957fddc-sb8c8" (UID: "05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:38 bluth k3s[11030]: E0724 08:51:38.275215 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-3706f211-9694-4ec9-88b9-55f66b73b802 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:39.275182434 +0100 IST m=+25.763039340 (durationBeforeRetry 1s). Error: MountVolume.MountDevice failed for volume "pvc-3706f211-9694-4ec9-88b9-55f66b73b802" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-3706f211-9694-4ec9-88b9-55f66b73b802") pod "tdarr-746d65b466-b7whb" (UID: "6e3e9791-9bb3-487f-ae50-048fefbef8be") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:38 bluth k3s[11030]: E0724 08:51:38.275373 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-4e6fcc6b-6890-4751-a09a-75bb6cde146f podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:39.275353276 +0100 IST m=+25.763210182 (durationBeforeRetry 1s). Error: MountVolume.MountDevice failed for volume "pvc-4e6fcc6b-6890-4751-a09a-75bb6cde146f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-4e6fcc6b-6890-4751-a09a-75bb6cde146f") pod "tdarr-746d65b466-b7whb" (UID: "6e3e9791-9bb3-487f-ae50-048fefbef8be") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:38 bluth k3s[11030]: E0724 08:51:38.275322 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:51:38 bluth k3s[11030]: I0724 08:51:38.275646 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="701697deca9bf7b8e1f4e7db07fd50948523f662557bda540a935e84cafcb74d" Jul 24 08:51:38 bluth k3s[11030]: E0724 08:51:38.275651 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-2e6b3d10-435c-4ce5-806d-61d551dcef05 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:39.275624092 +0100 IST m=+25.763480988 (durationBeforeRetry 1s). Error: MountVolume.MountDevice failed for volume "pvc-2e6b3d10-435c-4ce5-806d-61d551dcef05" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-2e6b3d10-435c-4ce5-806d-61d551dcef05") pod "tdarr-746d65b466-b7whb" (UID: "6e3e9791-9bb3-487f-ae50-048fefbef8be") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:38 bluth k3s[11030]: E0724 08:51:38.276093 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-028dc34a-8c2e-4a0a-b507-c33a02541407 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:40.276076268 +0100 IST m=+26.763933175 (durationBeforeRetry 2s). Error: MountVolume.MountDevice failed for volume "pvc-028dc34a-8c2e-4a0a-b507-c33a02541407" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-028dc34a-8c2e-4a0a-b507-c33a02541407") pod "tdarr-746d65b466-b7whb" (UID: "6e3e9791-9bb3-487f-ae50-048fefbef8be") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:38 bluth k3s[11030]: E0724 08:51:38.568821 11030 projected.go:293] Couldn't get configMap ix-esphome/kube-root-ca.crt: object "ix-esphome"/"kube-root-ca.crt" not registered Jul 24 08:51:38 bluth k3s[11030]: E0724 08:51:38.568844 11030 projected.go:199] Error preparing data for projected volume kube-api-access-xr6bj for pod ix-esphome/esphome-658957fddc-sb8c8: object "ix-esphome"/"kube-root-ca.crt" not registered Jul 24 08:51:38 bluth k3s[11030]: E0724 08:51:38.568992 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/projected/05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b-kube-api-access-xr6bj podName:05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b nodeName:}" failed. No retries permitted until 2022-07-24 08:51:39.068979241 +0100 IST m=+25.556836125 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xr6bj" (UniqueName: "kubernetes.io/projected/05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b-kube-api-access-xr6bj") pod "esphome-658957fddc-sb8c8" (UID: "05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b") : object "ix-esphome"/"kube-root-ca.crt" not registered Jul 24 08:51:38 bluth k3s[11030]: E0724 08:51:38.833734 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:39.833700126 +0100 IST m=+26.321557055 (durationBeforeRetry 1s). Error: MountVolume.MountDevice failed for volume "pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e") pod "esphome-658957fddc-sb8c8" (UID: "05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:38 bluth k3s[11030]: E0724 08:51:38.833810 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-06521873-c179-499e-9079-194ec6618fdd podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:39.833785062 +0100 IST m=+26.321641990 (durationBeforeRetry 1s). Error: MountVolume.MountDevice failed for volume "pvc-06521873-c179-499e-9079-194ec6618fdd" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-06521873-c179-499e-9079-194ec6618fdd") pod "esphome-658957fddc-sb8c8" (UID: "05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:39 bluth k3s[11030]: I0724 08:51:39.034187 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcltd\" (UniqueName: \"kubernetes.io/projected/3f1f74fe-1d74-4b61-9bd2-083782908a35-kube-api-access-lcltd\") pod \"home-assistant-ix-chart-54bf5d89c8-mz95k\" (UID: \"3f1f74fe-1d74-4b61-9bd2-083782908a35\") " pod="ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-mz95k" Jul 24 08:51:39 bluth k3s[11030]: I0724 08:51:39.034288 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"devfs\" (UniqueName: \"kubernetes.io/host-path/bb5993ab-6276-43d2-8017-a6fc01efb9a3-devfs\") pod \"intel-gpu-plugin-fxs6q\" (UID: \"bb5993ab-6276-43d2-8017-a6fc01efb9a3\") " pod="kube-system/intel-gpu-plugin-fxs6q" Jul 24 08:51:39 bluth k3s[11030]: I0724 08:51:39.034373 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sysfs\" (UniqueName: \"kubernetes.io/host-path/bb5993ab-6276-43d2-8017-a6fc01efb9a3-sysfs\") pod \"intel-gpu-plugin-fxs6q\" (UID: \"bb5993ab-6276-43d2-8017-a6fc01efb9a3\") " pod="kube-system/intel-gpu-plugin-fxs6q" Jul 24 08:51:39 bluth k3s[11030]: I0724 08:51:39.034440 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"varlogs\" (UniqueName: \"kubernetes.io/empty-dir/2fac2fcc-a503-4887-810c-c2ff11c07852-varlogs\") pod \"qbittorrent-6d7cf9f7cd-7n6c6\" (UID: \"2fac2fcc-a503-4887-810c-c2ff11c07852\") " pod="ix-qbittorrent/qbittorrent-6d7cf9f7cd-7n6c6" Jul 24 08:51:39 bluth k3s[11030]: I0724 08:51:39.034499 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubeletsockets\" (UniqueName: \"kubernetes.io/host-path/bb5993ab-6276-43d2-8017-a6fc01efb9a3-kubeletsockets\") pod \"intel-gpu-plugin-fxs6q\" (UID: \"bb5993ab-6276-43d2-8017-a6fc01efb9a3\") " pod="kube-system/intel-gpu-plugin-fxs6q" Jul 24 08:51:39 bluth k3s[11030]: I0724 08:51:39.034559 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdlrl\" (UniqueName: \"kubernetes.io/projected/495a78dd-7609-44be-bb7e-7036e65c860b-kube-api-access-xdlrl\") pod \"plex-bd956645c-95gj8\" (UID: \"495a78dd-7609-44be-bb7e-7036e65c860b\") " pod="ix-plex/plex-bd956645c-95gj8" Jul 24 08:51:39 bluth k3s[11030]: I0724 08:51:39.034678 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qgd6\" (UniqueName: \"kubernetes.io/projected/2fac2fcc-a503-4887-810c-c2ff11c07852-kube-api-access-2qgd6\") pod \"qbittorrent-6d7cf9f7cd-7n6c6\" (UID: \"2fac2fcc-a503-4887-810c-c2ff11c07852\") " pod="ix-qbittorrent/qbittorrent-6d7cf9f7cd-7n6c6" Jul 24 08:51:39 bluth k3s[11030]: I0724 08:51:39.034759 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smtsz\" (UniqueName: \"kubernetes.io/projected/bb5993ab-6276-43d2-8017-a6fc01efb9a3-kube-api-access-smtsz\") pod \"intel-gpu-plugin-fxs6q\" (UID: \"bb5993ab-6276-43d2-8017-a6fc01efb9a3\") " pod="kube-system/intel-gpu-plugin-fxs6q" Jul 24 08:51:39 bluth k3s[11030]: I0724 08:51:39.034817 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"transcode\" (UniqueName: \"kubernetes.io/host-path/495a78dd-7609-44be-bb7e-7036e65c860b-transcode\") pod \"plex-bd956645c-95gj8\" (UID: \"495a78dd-7609-44be-bb7e-7036e65c860b\") " pod="ix-plex/plex-bd956645c-95gj8" Jul 24 08:51:39 bluth k3s[11030]: I0724 08:51:39.034879 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plex-probe-check\" (UniqueName: \"kubernetes.io/configmap/495a78dd-7609-44be-bb7e-7036e65c860b-plex-probe-check\") pod \"plex-bd956645c-95gj8\" (UID: \"495a78dd-7609-44be-bb7e-7036e65c860b\") " pod="ix-plex/plex-bd956645c-95gj8" Jul 24 08:51:39 bluth k3s[11030]: I0724 08:51:39.034959 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"list-0\" (UniqueName: \"kubernetes.io/host-path/2fac2fcc-a503-4887-810c-c2ff11c07852-list-0\") pod \"qbittorrent-6d7cf9f7cd-7n6c6\" (UID: \"2fac2fcc-a503-4887-810c-c2ff11c07852\") " pod="ix-qbittorrent/qbittorrent-6d7cf9f7cd-7n6c6" Jul 24 08:51:39 bluth k3s[11030]: I0724 08:51:39.035017 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nfd-source-hooks\" (UniqueName: \"kubernetes.io/host-path/bb5993ab-6276-43d2-8017-a6fc01efb9a3-nfd-source-hooks\") pod \"intel-gpu-plugin-fxs6q\" (UID: \"bb5993ab-6276-43d2-8017-a6fc01efb9a3\") " pod="kube-system/intel-gpu-plugin-fxs6q" Jul 24 08:51:39 bluth k3s[11030]: I0724 08:51:39.035099 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp\" (UniqueName: \"kubernetes.io/empty-dir/2fac2fcc-a503-4887-810c-c2ff11c07852-temp\") pod \"qbittorrent-6d7cf9f7cd-7n6c6\" (UID: \"2fac2fcc-a503-4887-810c-c2ff11c07852\") " pod="ix-qbittorrent/qbittorrent-6d7cf9f7cd-7n6c6" Jul 24 08:51:39 bluth k3s[11030]: I0724 08:51:39.035220 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ix-host-path-home-assistant-0\" (UniqueName: \"kubernetes.io/host-path/3f1f74fe-1d74-4b61-9bd2-083782908a35-ix-host-path-home-assistant-0\") pod \"home-assistant-ix-chart-54bf5d89c8-mz95k\" (UID: \"3f1f74fe-1d74-4b61-9bd2-083782908a35\") " pod="ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-mz95k" Jul 24 08:51:39 bluth k3s[11030]: I0724 08:51:39.035276 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/495a78dd-7609-44be-bb7e-7036e65c860b-config\") pod \"plex-bd956645c-95gj8\" (UID: \"495a78dd-7609-44be-bb7e-7036e65c860b\") " pod="ix-plex/plex-bd956645c-95gj8" Jul 24 08:51:39 bluth k3s[11030]: I0724 08:51:39.035360 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-58562e12-4adc-4e83-8131-848f1b39bf00\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-58562e12-4adc-4e83-8131-848f1b39bf00\") pod \"qbittorrent-6d7cf9f7cd-7n6c6\" (UID: \"2fac2fcc-a503-4887-810c-c2ff11c07852\") " pod="ix-qbittorrent/qbittorrent-6d7cf9f7cd-7n6c6" Jul 24 08:51:39 bluth k3s[11030]: I0724 08:51:39.035415 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"shared-logs\" (UniqueName: \"kubernetes.io/empty-dir/495a78dd-7609-44be-bb7e-7036e65c860b-shared-logs\") pod \"plex-bd956645c-95gj8\" (UID: \"495a78dd-7609-44be-bb7e-7036e65c860b\") " pod="ix-plex/plex-bd956645c-95gj8" Jul 24 08:51:39 bluth k3s[11030]: I0724 08:51:39.035522 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ix-host-path-home-assistant-1\" (UniqueName: \"kubernetes.io/host-path/3f1f74fe-1d74-4b61-9bd2-083782908a35-ix-host-path-home-assistant-1\") pod \"home-assistant-ix-chart-54bf5d89c8-mz95k\" (UID: \"3f1f74fe-1d74-4b61-9bd2-083782908a35\") " pod="ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-mz95k" Jul 24 08:51:39 bluth k3s[11030]: I0724 08:51:39.035578 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/host-path/495a78dd-7609-44be-bb7e-7036e65c860b-data\") pod \"plex-bd956645c-95gj8\" (UID: \"495a78dd-7609-44be-bb7e-7036e65c860b\") " pod="ix-plex/plex-bd956645c-95gj8" Jul 24 08:51:39 bluth k3s[11030]: I0724 08:51:39.035641 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"shared\" (UniqueName: \"kubernetes.io/empty-dir/2fac2fcc-a503-4887-810c-c2ff11c07852-shared\") pod \"qbittorrent-6d7cf9f7cd-7n6c6\" (UID: \"2fac2fcc-a503-4887-810c-c2ff11c07852\") " pod="ix-qbittorrent/qbittorrent-6d7cf9f7cd-7n6c6" Jul 24 08:51:39 bluth k3s[11030]: I0724 08:51:39.035720 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"shared\" (UniqueName: \"kubernetes.io/empty-dir/495a78dd-7609-44be-bb7e-7036e65c860b-shared\") pod \"plex-bd956645c-95gj8\" (UID: \"495a78dd-7609-44be-bb7e-7036e65c860b\") " pod="ix-plex/plex-bd956645c-95gj8" Jul 24 08:51:39 bluth k3s[11030]: I0724 08:51:39.036243 11030 reconciler.go:157] "Reconciler: start to sync state" Jul 24 08:51:39 bluth k3s[11030]: I0724 08:51:39.224732 11030 reconciler.go:388] "Could not construct volume information, cleaning up mounts" podName=71936d7f-4447-4ac2-aab8-7586f615b53e volumeSpecName="pvc-121e6513-05f0-4032-8a90-26ab50c8cb97" err="could not check whether the volume \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-121e6513-05f0-4032-8a90-26ab50c8cb97\" (spec.Name: \"pvc-121e6513-05f0-4032-8a90-26ab50c8cb97\") pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" (UID: \"71936d7f-4447-4ac2-aab8-7586f615b53e\") is mounted with: stat /var/lib/kubelet/pods/71936d7f-4447-4ac2-aab8-7586f615b53e/volumes/kubernetes.io~csi/pvc-121e6513-05f0-4032-8a90-26ab50c8cb97/mount: no such file or directory" Jul 24 08:51:39 bluth k3s[11030]: E0724 08:51:39.225271 11030 operation_generator.go:902] UnmountVolume.MarkVolumeMountAsUncertain failed for volume "" (UniqueName: "pvc-121e6513-05f0-4032-8a90-26ab50c8cb97") pod "71936d7f-4447-4ac2-aab8-7586f615b53e" (UID: "71936d7f-4447-4ac2-aab8-7586f615b53e") : no volume with the name "pvc-121e6513-05f0-4032-8a90-26ab50c8cb97" exists in the list of attached volumes Jul 24 08:51:39 bluth k3s[11030]: E0724 08:51:39.225305 11030 nestedpendingoperations.go:335] Operation for "{volumeName:pvc-121e6513-05f0-4032-8a90-26ab50c8cb97 podName:71936d7f-4447-4ac2-aab8-7586f615b53e nodeName:}" failed. No retries permitted until 2022-07-24 08:51:39.725294114 +0100 IST m=+26.213150995 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "" (UniqueName: "pvc-121e6513-05f0-4032-8a90-26ab50c8cb97") pod "71936d7f-4447-4ac2-aab8-7586f615b53e" (UID: "71936d7f-4447-4ac2-aab8-7586f615b53e") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:39 bluth k3s[11030]: I0724 08:51:39.290535 11030 reconciler.go:388] "Could not construct volume information, cleaning up mounts" podName=8af245a0-51e1-46d0-bfce-c0c41307e43f volumeSpecName="pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4" err="could not check whether the volume \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4\" (spec.Name: \"pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4\") pod \"8af245a0-51e1-46d0-bfce-c0c41307e43f\" (UID: \"8af245a0-51e1-46d0-bfce-c0c41307e43f\") is mounted with: stat /var/lib/kubelet/pods/8af245a0-51e1-46d0-bfce-c0c41307e43f/volumes/kubernetes.io~csi/pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4/mount: no such file or directory" Jul 24 08:51:39 bluth k3s[11030]: E0724 08:51:39.290714 11030 operation_generator.go:902] UnmountVolume.MarkVolumeMountAsUncertain failed for volume "" (UniqueName: "pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4") pod "8af245a0-51e1-46d0-bfce-c0c41307e43f" (UID: "8af245a0-51e1-46d0-bfce-c0c41307e43f") : no volume with the name "pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4" exists in the list of attached volumes Jul 24 08:51:39 bluth k3s[11030]: E0724 08:51:39.290747 11030 nestedpendingoperations.go:335] Operation for "{volumeName:pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4 podName:8af245a0-51e1-46d0-bfce-c0c41307e43f nodeName:}" failed. No retries permitted until 2022-07-24 08:51:39.790733073 +0100 IST m=+26.278589953 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "" (UniqueName: "pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4") pod "8af245a0-51e1-46d0-bfce-c0c41307e43f" (UID: "8af245a0-51e1-46d0-bfce-c0c41307e43f") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:39 bluth k3s[11030]: I0724 08:51:39.297294 11030 node_lifecycle_controller.go:1190] Controller detected that some Nodes are Ready. Exiting master disruption mode. Jul 24 08:51:39 bluth k3s[11030]: E0724 08:51:39.391831 11030 projected.go:293] Couldn't get configMap ix-esphome/kube-root-ca.crt: object "ix-esphome"/"kube-root-ca.crt" not registered Jul 24 08:51:39 bluth k3s[11030]: E0724 08:51:39.391853 11030 projected.go:199] Error preparing data for projected volume kube-api-access-xr6bj for pod ix-esphome/esphome-658957fddc-sb8c8: object "ix-esphome"/"kube-root-ca.crt" not registered Jul 24 08:51:39 bluth k3s[11030]: E0724 08:51:39.391879 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/projected/05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b-kube-api-access-xr6bj podName:05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b nodeName:}" failed. No retries permitted until 2022-07-24 08:51:40.391869636 +0100 IST m=+26.879726517 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xr6bj" (UniqueName: "kubernetes.io/projected/05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b-kube-api-access-xr6bj") pod "esphome-658957fddc-sb8c8" (UID: "05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b") : object "ix-esphome"/"kube-root-ca.crt" not registered Jul 24 08:51:39 bluth k3s[11030]: E0724 08:51:39.392026 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:43.392019132 +0100 IST m=+29.879876012 (durationBeforeRetry 4s). Error: MountVolume.MountDevice failed for volume "pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274") pod "prowlarr-5bcc99cc47-rgrsl" (UID: "afd760a7-8b6b-470e-aa87-8ed16adb34bb") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:39 bluth k3s[11030]: E0724 08:51:39.392042 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-195f1a32-9c93-4737-983a-8dacee32e453 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:43.392036944 +0100 IST m=+29.879893824 (durationBeforeRetry 4s). Error: MountVolume.MountDevice failed for volume "pvc-195f1a32-9c93-4737-983a-8dacee32e453" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-195f1a32-9c93-4737-983a-8dacee32e453") pod "radarr-7486f78d7f-q4pn4" (UID: "be152b2b-fd1c-4a36-9081-ac6be2fb7901") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:39 bluth k3s[11030]: E0724 08:51:39.392082 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-2e6b3d10-435c-4ce5-806d-61d551dcef05 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:41.392074901 +0100 IST m=+27.879931781 (durationBeforeRetry 2s). Error: MountVolume.MountDevice failed for volume "pvc-2e6b3d10-435c-4ce5-806d-61d551dcef05" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-2e6b3d10-435c-4ce5-806d-61d551dcef05") pod "tdarr-746d65b466-b7whb" (UID: "6e3e9791-9bb3-487f-ae50-048fefbef8be") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:39 bluth k3s[11030]: E0724 08:51:39.392161 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c41ca0c8-d532-4ca2-b6ca-fe3e9ddf87d2 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:41.392155477 +0100 IST m=+27.880012357 (durationBeforeRetry 2s). Error: MountVolume.MountDevice failed for volume "pvc-c41ca0c8-d532-4ca2-b6ca-fe3e9ddf87d2" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c41ca0c8-d532-4ca2-b6ca-fe3e9ddf87d2") pod "tdarr-746d65b466-b7whb" (UID: "6e3e9791-9bb3-487f-ae50-048fefbef8be") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:39 bluth k3s[11030]: E0724 08:51:39.392176 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-4e6fcc6b-6890-4751-a09a-75bb6cde146f podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:41.39217 +0100 IST m=+27.880026880 (durationBeforeRetry 2s). Error: MountVolume.MountDevice failed for volume "pvc-4e6fcc6b-6890-4751-a09a-75bb6cde146f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-4e6fcc6b-6890-4751-a09a-75bb6cde146f") pod "tdarr-746d65b466-b7whb" (UID: "6e3e9791-9bb3-487f-ae50-048fefbef8be") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:39 bluth k3s[11030]: E0724 08:51:39.405579 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:43.405569583 +0100 IST m=+29.893426463 (durationBeforeRetry 4s). Error: MountVolume.MountDevice failed for volume "pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4") pod "sonarr-867d7dd548-4bn8s" (UID: "35ac75bf-b85f-4286-a228-ea8459dde305") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:39 bluth k3s[11030]: E0724 08:51:39.405596 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-3706f211-9694-4ec9-88b9-55f66b73b802 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:41.405592051 +0100 IST m=+27.893448931 (durationBeforeRetry 2s). Error: MountVolume.MountDevice failed for volume "pvc-3706f211-9694-4ec9-88b9-55f66b73b802" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-3706f211-9694-4ec9-88b9-55f66b73b802") pod "tdarr-746d65b466-b7whb" (UID: "6e3e9791-9bb3-487f-ae50-048fefbef8be") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:39 bluth k3s[11030]: E0724 08:51:39.405605 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-58562e12-4adc-4e83-8131-848f1b39bf00 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:39.905600634 +0100 IST m=+26.393457514 (durationBeforeRetry 500ms). Error: MountVolume.MountDevice failed for volume "pvc-58562e12-4adc-4e83-8131-848f1b39bf00" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-58562e12-4adc-4e83-8131-848f1b39bf00") pod "qbittorrent-6d7cf9f7cd-7n6c6" (UID: "2fac2fcc-a503-4887-810c-c2ff11c07852") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:39 bluth k3s[11030]: I0724 08:51:39.457920 11030 scope.go:110] "RemoveContainer" containerID="0cc34af722b67d4e3567a59268d634aa0b99635c2e9bbaaa83c0ee28b84d7458" Jul 24 08:51:39 bluth k3s[11030]: E0724 08:51:39.458808 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 0cc34af722b67d4e3567a59268d634aa0b99635c2e9bbaaa83c0ee28b84d7458" containerID="0cc34af722b67d4e3567a59268d634aa0b99635c2e9bbaaa83c0ee28b84d7458" Jul 24 08:51:39 bluth k3s[11030]: I0724 08:51:39.458837 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:0cc34af722b67d4e3567a59268d634aa0b99635c2e9bbaaa83c0ee28b84d7458} err="failed to get container status \"0cc34af722b67d4e3567a59268d634aa0b99635c2e9bbaaa83c0ee28b84d7458\": rpc error: code = Unknown desc = Error: No such container: 0cc34af722b67d4e3567a59268d634aa0b99635c2e9bbaaa83c0ee28b84d7458" Jul 24 08:51:39 bluth k3s[11030]: I0724 08:51:39.458848 11030 scope.go:110] "RemoveContainer" containerID="701697deca9bf7b8e1f4e7db07fd50948523f662557bda540a935e84cafcb74d" Jul 24 08:51:39 bluth k3s[11030]: E0724 08:51:39.459082 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 701697deca9bf7b8e1f4e7db07fd50948523f662557bda540a935e84cafcb74d" containerID="701697deca9bf7b8e1f4e7db07fd50948523f662557bda540a935e84cafcb74d" Jul 24 08:51:39 bluth k3s[11030]: I0724 08:51:39.459102 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:701697deca9bf7b8e1f4e7db07fd50948523f662557bda540a935e84cafcb74d} err="failed to get container status \"701697deca9bf7b8e1f4e7db07fd50948523f662557bda540a935e84cafcb74d\": rpc error: code = Unknown desc = Error: No such container: 701697deca9bf7b8e1f4e7db07fd50948523f662557bda540a935e84cafcb74d" Jul 24 08:51:39 bluth k3s[11030]: I0724 08:51:39.459109 11030 scope.go:110] "RemoveContainer" containerID="701697deca9bf7b8e1f4e7db07fd50948523f662557bda540a935e84cafcb74d" Jul 24 08:51:39 bluth k3s[11030]: I0724 08:51:39.459316 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:701697deca9bf7b8e1f4e7db07fd50948523f662557bda540a935e84cafcb74d} err="failed to get container status \"701697deca9bf7b8e1f4e7db07fd50948523f662557bda540a935e84cafcb74d\": rpc error: code = Unknown desc = Error: No such container: 701697deca9bf7b8e1f4e7db07fd50948523f662557bda540a935e84cafcb74d" Jul 24 08:51:39 bluth k3s[11030]: I0724 08:51:39.459327 11030 scope.go:110] "RemoveContainer" containerID="42f6521e6785b9192e442c8f8b5075d1a19fe2ff86adb83a34c2ee72c820662f" Jul 24 08:51:39 bluth k3s[11030]: E0724 08:51:39.746583 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:51:39 bluth k3s[11030]: I0724 08:51:39.794353 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3f09401b-76fa-4bff-8736-00b8e6a1a41f-config-volume\") pod \"coredns-d76bd69b-mhb5r\" (UID: \"3f09401b-76fa-4bff-8736-00b8e6a1a41f\") " pod="kube-system/coredns-d76bd69b-mhb5r" Jul 24 08:51:39 bluth k3s[11030]: I0724 08:51:39.794405 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"custom-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f09401b-76fa-4bff-8736-00b8e6a1a41f-custom-config-volume\") pod \"coredns-d76bd69b-mhb5r\" (UID: \"3f09401b-76fa-4bff-8736-00b8e6a1a41f\") " pod="kube-system/coredns-d76bd69b-mhb5r" Jul 24 08:51:39 bluth k3s[11030]: I0724 08:51:39.794426 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjstp\" (UniqueName: \"kubernetes.io/projected/3f09401b-76fa-4bff-8736-00b8e6a1a41f-kube-api-access-fjstp\") pod \"coredns-d76bd69b-mhb5r\" (UID: \"3f09401b-76fa-4bff-8736-00b8e6a1a41f\") " pod="kube-system/coredns-d76bd69b-mhb5r" Jul 24 08:51:39 bluth k3s[11030]: E0724 08:51:39.794480 11030 projected.go:293] Couldn't get configMap ix-sonarr/kube-root-ca.crt: object "ix-sonarr"/"kube-root-ca.crt" not registered Jul 24 08:51:39 bluth k3s[11030]: E0724 08:51:39.794492 11030 projected.go:199] Error preparing data for projected volume kube-api-access-2v58k for pod ix-sonarr/sonarr-867d7dd548-4bn8s: object "ix-sonarr"/"kube-root-ca.crt" not registered Jul 24 08:51:39 bluth k3s[11030]: E0724 08:51:39.794523 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/projected/35ac75bf-b85f-4286-a228-ea8459dde305-kube-api-access-2v58k podName:35ac75bf-b85f-4286-a228-ea8459dde305 nodeName:}" failed. No retries permitted until 2022-07-24 08:51:40.294512522 +0100 IST m=+26.782369402 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2v58k" (UniqueName: "kubernetes.io/projected/35ac75bf-b85f-4286-a228-ea8459dde305-kube-api-access-2v58k") pod "sonarr-867d7dd548-4bn8s" (UID: "35ac75bf-b85f-4286-a228-ea8459dde305") : object "ix-sonarr"/"kube-root-ca.crt" not registered Jul 24 08:51:39 bluth k3s[11030]: E0724 08:51:39.895686 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:41.895676201 +0100 IST m=+28.383533084 (durationBeforeRetry 2s). Error: MountVolume.MountDevice failed for volume "pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e") pod "esphome-658957fddc-sb8c8" (UID: "05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:39 bluth k3s[11030]: E0724 08:51:39.895705 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-06521873-c179-499e-9079-194ec6618fdd podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:41.895698819 +0100 IST m=+28.383555705 (durationBeforeRetry 2s). Error: MountVolume.MountDevice failed for volume "pvc-06521873-c179-499e-9079-194ec6618fdd" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-06521873-c179-499e-9079-194ec6618fdd") pod "esphome-658957fddc-sb8c8" (UID: "05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:39 bluth k3s[11030]: I0724 08:51:39.988810 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="0cc34af722b67d4e3567a59268d634aa0b99635c2e9bbaaa83c0ee28b84d7458" Jul 24 08:51:39 bluth k3s[11030]: E0724 08:51:39.996109 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-58562e12-4adc-4e83-8131-848f1b39bf00 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:40.996097298 +0100 IST m=+27.483954179 (durationBeforeRetry 1s). Error: MountVolume.MountDevice failed for volume "pvc-58562e12-4adc-4e83-8131-848f1b39bf00" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-58562e12-4adc-4e83-8131-848f1b39bf00") pod "qbittorrent-6d7cf9f7cd-7n6c6" (UID: "2fac2fcc-a503-4887-810c-c2ff11c07852") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:40 bluth k3s[11030]: E0724 08:51:40.169076 11030 projected.go:293] Couldn't get configMap ix-home-assistant/kube-root-ca.crt: object "ix-home-assistant"/"kube-root-ca.crt" not registered Jul 24 08:51:40 bluth k3s[11030]: E0724 08:51:40.169104 11030 projected.go:199] Error preparing data for projected volume kube-api-access-lcltd for pod ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-mz95k: object "ix-home-assistant"/"kube-root-ca.crt" not registered Jul 24 08:51:40 bluth k3s[11030]: E0724 08:51:40.169146 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/projected/3f1f74fe-1d74-4b61-9bd2-083782908a35-kube-api-access-lcltd podName:3f1f74fe-1d74-4b61-9bd2-083782908a35 nodeName:}" failed. No retries permitted until 2022-07-24 08:51:40.669130194 +0100 IST m=+27.156987074 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lcltd" (UniqueName: "kubernetes.io/projected/3f1f74fe-1d74-4b61-9bd2-083782908a35-kube-api-access-lcltd") pod "home-assistant-ix-chart-54bf5d89c8-mz95k" (UID: "3f1f74fe-1d74-4b61-9bd2-083782908a35") : object "ix-home-assistant"/"kube-root-ca.crt" not registered Jul 24 08:51:40 bluth k3s[11030]: E0724 08:51:40.298990 11030 projected.go:293] Couldn't get configMap ix-sonarr/kube-root-ca.crt: object "ix-sonarr"/"kube-root-ca.crt" not registered Jul 24 08:51:40 bluth k3s[11030]: E0724 08:51:40.299015 11030 projected.go:199] Error preparing data for projected volume kube-api-access-2v58k for pod ix-sonarr/sonarr-867d7dd548-4bn8s: object "ix-sonarr"/"kube-root-ca.crt" not registered Jul 24 08:51:40 bluth k3s[11030]: E0724 08:51:40.299047 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/projected/35ac75bf-b85f-4286-a228-ea8459dde305-kube-api-access-2v58k podName:35ac75bf-b85f-4286-a228-ea8459dde305 nodeName:}" failed. No retries permitted until 2022-07-24 08:51:41.299036969 +0100 IST m=+27.786893849 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2v58k" (UniqueName: "kubernetes.io/projected/35ac75bf-b85f-4286-a228-ea8459dde305-kube-api-access-2v58k") pod "sonarr-867d7dd548-4bn8s" (UID: "35ac75bf-b85f-4286-a228-ea8459dde305") : object "ix-sonarr"/"kube-root-ca.crt" not registered Jul 24 08:51:40 bluth k3s[11030]: E0724 08:51:40.299280 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-028dc34a-8c2e-4a0a-b507-c33a02541407 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:44.299273712 +0100 IST m=+30.787130593 (durationBeforeRetry 4s). Error: MountVolume.MountDevice failed for volume "pvc-028dc34a-8c2e-4a0a-b507-c33a02541407" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-028dc34a-8c2e-4a0a-b507-c33a02541407") pod "tdarr-746d65b466-b7whb" (UID: "6e3e9791-9bb3-487f-ae50-048fefbef8be") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:40 bluth k3s[11030]: E0724 08:51:40.399200 11030 projected.go:293] Couldn't get configMap ix-esphome/kube-root-ca.crt: object "ix-esphome"/"kube-root-ca.crt" not registered Jul 24 08:51:40 bluth k3s[11030]: E0724 08:51:40.399224 11030 projected.go:199] Error preparing data for projected volume kube-api-access-xr6bj for pod ix-esphome/esphome-658957fddc-sb8c8: object "ix-esphome"/"kube-root-ca.crt" not registered Jul 24 08:51:40 bluth k3s[11030]: E0724 08:51:40.399392 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/projected/05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b-kube-api-access-xr6bj podName:05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b nodeName:}" failed. No retries permitted until 2022-07-24 08:51:42.399379568 +0100 IST m=+28.887236451 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xr6bj" (UniqueName: "kubernetes.io/projected/05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b-kube-api-access-xr6bj") pod "esphome-658957fddc-sb8c8" (UID: "05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b") : object "ix-esphome"/"kube-root-ca.crt" not registered Jul 24 08:51:40 bluth k3s[11030]: I0724 08:51:40.564755 11030 request.go:665] Waited for 1.172315192s due to client-side throttling, not priority and fairness, request: POST:https://127.0.0.1:6443/api/v1/namespaces/kube-system/serviceaccounts/default/token Jul 24 08:51:40 bluth k3s[11030]: I0724 08:51:40.666985 11030 cni.go:334] "CNI failed to retrieve network namespace path" err="cannot find network namespace for the terminated container \"08296103f6b3137ee4836a8a22882c87ac8204632cd2fb900918cde27fc9f00d\"" Jul 24 08:51:40 bluth k3s[11030]: E0724 08:51:40.702847 11030 projected.go:293] Couldn't get configMap ix-home-assistant/kube-root-ca.crt: object "ix-home-assistant"/"kube-root-ca.crt" not registered Jul 24 08:51:40 bluth k3s[11030]: E0724 08:51:40.702866 11030 projected.go:199] Error preparing data for projected volume kube-api-access-lcltd for pod ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-mz95k: object "ix-home-assistant"/"kube-root-ca.crt" not registered Jul 24 08:51:40 bluth k3s[11030]: E0724 08:51:40.702896 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/projected/3f1f74fe-1d74-4b61-9bd2-083782908a35-kube-api-access-lcltd podName:3f1f74fe-1d74-4b61-9bd2-083782908a35 nodeName:}" failed. No retries permitted until 2022-07-24 08:51:41.702885717 +0100 IST m=+28.190742601 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lcltd" (UniqueName: "kubernetes.io/projected/3f1f74fe-1d74-4b61-9bd2-083782908a35-kube-api-access-lcltd") pod "home-assistant-ix-chart-54bf5d89c8-mz95k" (UID: "3f1f74fe-1d74-4b61-9bd2-083782908a35") : object "ix-home-assistant"/"kube-root-ca.crt" not registered Jul 24 08:51:40 bluth k3s[11030]: I0724 08:51:40.767495 11030 cni.go:334] "CNI failed to retrieve network namespace path" err="cannot find network namespace for the terminated container \"1f9702e48cd10a6ed8d91715e373237e2c86c037931231de35857ee760f1237a\"" Jul 24 08:51:40 bluth k3s[11030]: I0724 08:51:40.866583 11030 scope.go:110] "RemoveContainer" containerID="0b9afc6eaa2accf484d333b3f1cf4b621ed551bbc2bf8b1198218f3bd645c664" Jul 24 08:51:41 bluth k3s[11030]: E0724 08:51:41.005978 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-58562e12-4adc-4e83-8131-848f1b39bf00 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:43.005968302 +0100 IST m=+29.493825182 (durationBeforeRetry 2s). Error: MountVolume.MountDevice failed for volume "pvc-58562e12-4adc-4e83-8131-848f1b39bf00" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-58562e12-4adc-4e83-8131-848f1b39bf00") pod "qbittorrent-6d7cf9f7cd-7n6c6" (UID: "2fac2fcc-a503-4887-810c-c2ff11c07852") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:41 bluth k3s[11030]: E0724 08:51:41.307452 11030 projected.go:293] Couldn't get configMap ix-sonarr/kube-root-ca.crt: object "ix-sonarr"/"kube-root-ca.crt" not registered Jul 24 08:51:41 bluth k3s[11030]: E0724 08:51:41.307476 11030 projected.go:199] Error preparing data for projected volume kube-api-access-2v58k for pod ix-sonarr/sonarr-867d7dd548-4bn8s: object "ix-sonarr"/"kube-root-ca.crt" not registered Jul 24 08:51:41 bluth k3s[11030]: E0724 08:51:41.307508 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/projected/35ac75bf-b85f-4286-a228-ea8459dde305-kube-api-access-2v58k podName:35ac75bf-b85f-4286-a228-ea8459dde305 nodeName:}" failed. No retries permitted until 2022-07-24 08:51:43.307497897 +0100 IST m=+29.795354781 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2v58k" (UniqueName: "kubernetes.io/projected/35ac75bf-b85f-4286-a228-ea8459dde305-kube-api-access-2v58k") pod "sonarr-867d7dd548-4bn8s" (UID: "35ac75bf-b85f-4286-a228-ea8459dde305") : object "ix-sonarr"/"kube-root-ca.crt" not registered Jul 24 08:51:41 bluth k3s[11030]: E0724 08:51:41.408455 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-3706f211-9694-4ec9-88b9-55f66b73b802 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:45.408445867 +0100 IST m=+31.896302748 (durationBeforeRetry 4s). Error: MountVolume.MountDevice failed for volume "pvc-3706f211-9694-4ec9-88b9-55f66b73b802" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-3706f211-9694-4ec9-88b9-55f66b73b802") pod "tdarr-746d65b466-b7whb" (UID: "6e3e9791-9bb3-487f-ae50-048fefbef8be") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:41 bluth k3s[11030]: E0724 08:51:41.408622 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-2e6b3d10-435c-4ce5-806d-61d551dcef05 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:45.40861577 +0100 IST m=+31.896472650 (durationBeforeRetry 4s). Error: MountVolume.MountDevice failed for volume "pvc-2e6b3d10-435c-4ce5-806d-61d551dcef05" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-2e6b3d10-435c-4ce5-806d-61d551dcef05") pod "tdarr-746d65b466-b7whb" (UID: "6e3e9791-9bb3-487f-ae50-048fefbef8be") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:41 bluth k3s[11030]: E0724 08:51:41.408764 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-4e6fcc6b-6890-4751-a09a-75bb6cde146f podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:45.408758795 +0100 IST m=+31.896615676 (durationBeforeRetry 4s). Error: MountVolume.MountDevice failed for volume "pvc-4e6fcc6b-6890-4751-a09a-75bb6cde146f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-4e6fcc6b-6890-4751-a09a-75bb6cde146f") pod "tdarr-746d65b466-b7whb" (UID: "6e3e9791-9bb3-487f-ae50-048fefbef8be") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:41 bluth k3s[11030]: E0724 08:51:41.408906 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c41ca0c8-d532-4ca2-b6ca-fe3e9ddf87d2 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:45.408901021 +0100 IST m=+31.896757901 (durationBeforeRetry 4s). Error: MountVolume.MountDevice failed for volume "pvc-c41ca0c8-d532-4ca2-b6ca-fe3e9ddf87d2" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c41ca0c8-d532-4ca2-b6ca-fe3e9ddf87d2") pod "tdarr-746d65b466-b7whb" (UID: "6e3e9791-9bb3-487f-ae50-048fefbef8be") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:41 bluth k3s[11030]: E0724 08:51:41.709849 11030 projected.go:293] Couldn't get configMap ix-home-assistant/kube-root-ca.crt: object "ix-home-assistant"/"kube-root-ca.crt" not registered Jul 24 08:51:41 bluth k3s[11030]: E0724 08:51:41.709872 11030 projected.go:199] Error preparing data for projected volume kube-api-access-lcltd for pod ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-mz95k: object "ix-home-assistant"/"kube-root-ca.crt" not registered Jul 24 08:51:41 bluth k3s[11030]: E0724 08:51:41.709909 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/projected/3f1f74fe-1d74-4b61-9bd2-083782908a35-kube-api-access-lcltd podName:3f1f74fe-1d74-4b61-9bd2-083782908a35 nodeName:}" failed. No retries permitted until 2022-07-24 08:51:43.70989871 +0100 IST m=+30.197755593 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lcltd" (UniqueName: "kubernetes.io/projected/3f1f74fe-1d74-4b61-9bd2-083782908a35-kube-api-access-lcltd") pod "home-assistant-ix-chart-54bf5d89c8-mz95k" (UID: "3f1f74fe-1d74-4b61-9bd2-083782908a35") : object "ix-home-assistant"/"kube-root-ca.crt" not registered Jul 24 08:51:41 bluth k3s[11030]: E0724 08:51:41.747678 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:51:41 bluth k3s[11030]: E0724 08:51:41.911564 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-06521873-c179-499e-9079-194ec6618fdd podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:45.911554494 +0100 IST m=+32.399411374 (durationBeforeRetry 4s). Error: MountVolume.MountDevice failed for volume "pvc-06521873-c179-499e-9079-194ec6618fdd" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-06521873-c179-499e-9079-194ec6618fdd") pod "esphome-658957fddc-sb8c8" (UID: "05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:41 bluth k3s[11030]: E0724 08:51:41.911730 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:45.911724057 +0100 IST m=+32.399580937 (durationBeforeRetry 4s). Error: MountVolume.MountDevice failed for volume "pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e") pod "esphome-658957fddc-sb8c8" (UID: "05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:42 bluth k3s[11030]: E0724 08:51:42.418030 11030 projected.go:293] Couldn't get configMap ix-esphome/kube-root-ca.crt: object "ix-esphome"/"kube-root-ca.crt" not registered Jul 24 08:51:42 bluth k3s[11030]: E0724 08:51:42.418348 11030 projected.go:199] Error preparing data for projected volume kube-api-access-xr6bj for pod ix-esphome/esphome-658957fddc-sb8c8: object "ix-esphome"/"kube-root-ca.crt" not registered Jul 24 08:51:42 bluth k3s[11030]: E0724 08:51:42.418425 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/projected/05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b-kube-api-access-xr6bj podName:05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b nodeName:}" failed. No retries permitted until 2022-07-24 08:51:46.418415101 +0100 IST m=+32.906271981 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xr6bj" (UniqueName: "kubernetes.io/projected/05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b-kube-api-access-xr6bj") pod "esphome-658957fddc-sb8c8" (UID: "05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b") : object "ix-esphome"/"kube-root-ca.crt" not registered Jul 24 08:51:43 bluth k3s[11030]: E0724 08:51:43.021711 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-58562e12-4adc-4e83-8131-848f1b39bf00 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:47.021701746 +0100 IST m=+33.509558626 (durationBeforeRetry 4s). Error: MountVolume.MountDevice failed for volume "pvc-58562e12-4adc-4e83-8131-848f1b39bf00" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-58562e12-4adc-4e83-8131-848f1b39bf00") pod "qbittorrent-6d7cf9f7cd-7n6c6" (UID: "2fac2fcc-a503-4887-810c-c2ff11c07852") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:43 bluth k3s[11030]: E0724 08:51:43.322709 11030 projected.go:293] Couldn't get configMap ix-sonarr/kube-root-ca.crt: object "ix-sonarr"/"kube-root-ca.crt" not registered Jul 24 08:51:43 bluth k3s[11030]: E0724 08:51:43.323011 11030 projected.go:199] Error preparing data for projected volume kube-api-access-2v58k for pod ix-sonarr/sonarr-867d7dd548-4bn8s: object "ix-sonarr"/"kube-root-ca.crt" not registered Jul 24 08:51:43 bluth k3s[11030]: E0724 08:51:43.323095 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/projected/35ac75bf-b85f-4286-a228-ea8459dde305-kube-api-access-2v58k podName:35ac75bf-b85f-4286-a228-ea8459dde305 nodeName:}" failed. No retries permitted until 2022-07-24 08:51:47.323084214 +0100 IST m=+33.810941095 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2v58k" (UniqueName: "kubernetes.io/projected/35ac75bf-b85f-4286-a228-ea8459dde305-kube-api-access-2v58k") pod "sonarr-867d7dd548-4bn8s" (UID: "35ac75bf-b85f-4286-a228-ea8459dde305") : object "ix-sonarr"/"kube-root-ca.crt" not registered Jul 24 08:51:43 bluth k3s[11030]: E0724 08:51:43.423326 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:51.423314978 +0100 IST m=+37.911171858 (durationBeforeRetry 8s). Error: MountVolume.MountDevice failed for volume "pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274") pod "prowlarr-5bcc99cc47-rgrsl" (UID: "afd760a7-8b6b-470e-aa87-8ed16adb34bb") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:43 bluth k3s[11030]: E0724 08:51:43.423503 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-195f1a32-9c93-4737-983a-8dacee32e453 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:51.423494775 +0100 IST m=+37.911351656 (durationBeforeRetry 8s). Error: MountVolume.MountDevice failed for volume "pvc-195f1a32-9c93-4737-983a-8dacee32e453" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-195f1a32-9c93-4737-983a-8dacee32e453") pod "radarr-7486f78d7f-q4pn4" (UID: "be152b2b-fd1c-4a36-9081-ac6be2fb7901") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:43 bluth k3s[11030]: E0724 08:51:43.423650 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:51.42364285 +0100 IST m=+37.911499730 (durationBeforeRetry 8s). Error: MountVolume.MountDevice failed for volume "pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4") pod "sonarr-867d7dd548-4bn8s" (UID: "35ac75bf-b85f-4286-a228-ea8459dde305") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:43 bluth k3s[11030]: E0724 08:51:43.726226 11030 projected.go:293] Couldn't get configMap ix-home-assistant/kube-root-ca.crt: object "ix-home-assistant"/"kube-root-ca.crt" not registered Jul 24 08:51:43 bluth k3s[11030]: E0724 08:51:43.726366 11030 projected.go:199] Error preparing data for projected volume kube-api-access-lcltd for pod ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-mz95k: object "ix-home-assistant"/"kube-root-ca.crt" not registered Jul 24 08:51:43 bluth k3s[11030]: E0724 08:51:43.726444 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/projected/3f1f74fe-1d74-4b61-9bd2-083782908a35-kube-api-access-lcltd podName:3f1f74fe-1d74-4b61-9bd2-083782908a35 nodeName:}" failed. No retries permitted until 2022-07-24 08:51:47.726432241 +0100 IST m=+34.214289121 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lcltd" (UniqueName: "kubernetes.io/projected/3f1f74fe-1d74-4b61-9bd2-083782908a35-kube-api-access-lcltd") pod "home-assistant-ix-chart-54bf5d89c8-mz95k" (UID: "3f1f74fe-1d74-4b61-9bd2-083782908a35") : object "ix-home-assistant"/"kube-root-ca.crt" not registered Jul 24 08:51:43 bluth k3s[11030]: E0724 08:51:43.748009 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:51:44 bluth k3s[11030]: E0724 08:51:44.331028 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-028dc34a-8c2e-4a0a-b507-c33a02541407 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:52.331018786 +0100 IST m=+38.818875667 (durationBeforeRetry 8s). Error: MountVolume.MountDevice failed for volume "pvc-028dc34a-8c2e-4a0a-b507-c33a02541407" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-028dc34a-8c2e-4a0a-b507-c33a02541407") pod "tdarr-746d65b466-b7whb" (UID: "6e3e9791-9bb3-487f-ae50-048fefbef8be") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:45 bluth k3s[11030]: E0724 08:51:45.438080 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-3706f211-9694-4ec9-88b9-55f66b73b802 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:53.438070105 +0100 IST m=+39.925926986 (durationBeforeRetry 8s). Error: MountVolume.MountDevice failed for volume "pvc-3706f211-9694-4ec9-88b9-55f66b73b802" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-3706f211-9694-4ec9-88b9-55f66b73b802") pod "tdarr-746d65b466-b7whb" (UID: "6e3e9791-9bb3-487f-ae50-048fefbef8be") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:45 bluth k3s[11030]: E0724 08:51:45.438103 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-4e6fcc6b-6890-4751-a09a-75bb6cde146f podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:53.438097985 +0100 IST m=+39.925954866 (durationBeforeRetry 8s). Error: MountVolume.MountDevice failed for volume "pvc-4e6fcc6b-6890-4751-a09a-75bb6cde146f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-4e6fcc6b-6890-4751-a09a-75bb6cde146f") pod "tdarr-746d65b466-b7whb" (UID: "6e3e9791-9bb3-487f-ae50-048fefbef8be") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:45 bluth k3s[11030]: E0724 08:51:45.438111 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-2e6b3d10-435c-4ce5-806d-61d551dcef05 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:53.438107034 +0100 IST m=+39.925963915 (durationBeforeRetry 8s). Error: MountVolume.MountDevice failed for volume "pvc-2e6b3d10-435c-4ce5-806d-61d551dcef05" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-2e6b3d10-435c-4ce5-806d-61d551dcef05") pod "tdarr-746d65b466-b7whb" (UID: "6e3e9791-9bb3-487f-ae50-048fefbef8be") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:45 bluth k3s[11030]: E0724 08:51:45.438214 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c41ca0c8-d532-4ca2-b6ca-fe3e9ddf87d2 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:53.438206578 +0100 IST m=+39.926063457 (durationBeforeRetry 8s). Error: MountVolume.MountDevice failed for volume "pvc-c41ca0c8-d532-4ca2-b6ca-fe3e9ddf87d2" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c41ca0c8-d532-4ca2-b6ca-fe3e9ddf87d2") pod "tdarr-746d65b466-b7whb" (UID: "6e3e9791-9bb3-487f-ae50-048fefbef8be") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:45 bluth k3s[11030]: E0724 08:51:45.746408 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:51:45 bluth k3s[11030]: E0724 08:51:45.941370 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-06521873-c179-499e-9079-194ec6618fdd podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:53.94136093 +0100 IST m=+40.429217814 (durationBeforeRetry 8s). Error: MountVolume.MountDevice failed for volume "pvc-06521873-c179-499e-9079-194ec6618fdd" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-06521873-c179-499e-9079-194ec6618fdd") pod "esphome-658957fddc-sb8c8" (UID: "05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:45 bluth k3s[11030]: E0724 08:51:45.941389 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:53.941383509 +0100 IST m=+40.429240389 (durationBeforeRetry 8s). Error: MountVolume.MountDevice failed for volume "pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e") pod "esphome-658957fddc-sb8c8" (UID: "05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:46 bluth k3s[11030]: E0724 08:51:46.444054 11030 projected.go:293] Couldn't get configMap ix-esphome/kube-root-ca.crt: object "ix-esphome"/"kube-root-ca.crt" not registered Jul 24 08:51:46 bluth k3s[11030]: E0724 08:51:46.444102 11030 projected.go:199] Error preparing data for projected volume kube-api-access-xr6bj for pod ix-esphome/esphome-658957fddc-sb8c8: object "ix-esphome"/"kube-root-ca.crt" not registered Jul 24 08:51:46 bluth k3s[11030]: E0724 08:51:46.444141 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/projected/05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b-kube-api-access-xr6bj podName:05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b nodeName:}" failed. No retries permitted until 2022-07-24 08:51:54.444124584 +0100 IST m=+40.931981464 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xr6bj" (UniqueName: "kubernetes.io/projected/05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b-kube-api-access-xr6bj") pod "esphome-658957fddc-sb8c8" (UID: "05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b") : object "ix-esphome"/"kube-root-ca.crt" not registered Jul 24 08:51:46 bluth k3s[11030]: I0724 08:51:46.981940 11030 trace.go:205] Trace[245985465]: "GuaranteedUpdate etcd3" type:*core.Event (24-Jul-2022 08:51:45.453) (total time: 1528ms): Jul 24 08:51:46 bluth k3s[11030]: Trace[245985465]: ---"Transaction committed" 1526ms (08:51:46.981) Jul 24 08:51:46 bluth k3s[11030]: Trace[245985465]: [1.528155427s] [1.528155427s] END Jul 24 08:51:46 bluth k3s[11030]: I0724 08:51:46.982202 11030 trace.go:205] Trace[988772168]: "Patch" url:/api/v1/namespaces/ix-tdarr/events/tdarr-746d65b466-b7whb.1704b490dd9822e8,user-agent:k3s/v1.23.5+k3s (linux/amd64) kubernetes/fbfa51e,audit-id:8826204f-a327-4135-9b2d-9d25a64971b2,client:127.0.0.1,accept:application/vnd.kubernetes.protobuf,application/json,protocol:HTTP/2.0 (24-Jul-2022 08:51:45.453) (total time: 1528ms): Jul 24 08:51:46 bluth k3s[11030]: Trace[988772168]: ---"Object stored in database" 1527ms (08:51:46.982) Jul 24 08:51:46 bluth k3s[11030]: Trace[988772168]: [1.528468181s] [1.528468181s] END Jul 24 08:51:47 bluth k3s[11030]: E0724 08:51:47.047856 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-58562e12-4adc-4e83-8131-848f1b39bf00 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:51:55.047846628 +0100 IST m=+41.535703508 (durationBeforeRetry 8s). Error: MountVolume.MountDevice failed for volume "pvc-58562e12-4adc-4e83-8131-848f1b39bf00" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-58562e12-4adc-4e83-8131-848f1b39bf00") pod "qbittorrent-6d7cf9f7cd-7n6c6" (UID: "2fac2fcc-a503-4887-810c-c2ff11c07852") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:47 bluth k3s[11030]: E0724 08:51:47.349630 11030 projected.go:293] Couldn't get configMap ix-sonarr/kube-root-ca.crt: object "ix-sonarr"/"kube-root-ca.crt" not registered Jul 24 08:51:47 bluth k3s[11030]: E0724 08:51:47.349654 11030 projected.go:199] Error preparing data for projected volume kube-api-access-2v58k for pod ix-sonarr/sonarr-867d7dd548-4bn8s: object "ix-sonarr"/"kube-root-ca.crt" not registered Jul 24 08:51:47 bluth k3s[11030]: E0724 08:51:47.349683 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/projected/35ac75bf-b85f-4286-a228-ea8459dde305-kube-api-access-2v58k podName:35ac75bf-b85f-4286-a228-ea8459dde305 nodeName:}" failed. No retries permitted until 2022-07-24 08:51:55.349673628 +0100 IST m=+41.837530508 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2v58k" (UniqueName: "kubernetes.io/projected/35ac75bf-b85f-4286-a228-ea8459dde305-kube-api-access-2v58k") pod "sonarr-867d7dd548-4bn8s" (UID: "35ac75bf-b85f-4286-a228-ea8459dde305") : object "ix-sonarr"/"kube-root-ca.crt" not registered Jul 24 08:51:47 bluth k3s[11030]: E0724 08:51:47.748577 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:51:47 bluth k3s[11030]: E0724 08:51:47.754017 11030 projected.go:293] Couldn't get configMap ix-home-assistant/kube-root-ca.crt: object "ix-home-assistant"/"kube-root-ca.crt" not registered Jul 24 08:51:47 bluth k3s[11030]: E0724 08:51:47.754135 11030 projected.go:199] Error preparing data for projected volume kube-api-access-lcltd for pod ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-mz95k: object "ix-home-assistant"/"kube-root-ca.crt" not registered Jul 24 08:51:47 bluth k3s[11030]: E0724 08:51:47.754218 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/projected/3f1f74fe-1d74-4b61-9bd2-083782908a35-kube-api-access-lcltd podName:3f1f74fe-1d74-4b61-9bd2-083782908a35 nodeName:}" failed. No retries permitted until 2022-07-24 08:51:55.754206515 +0100 IST m=+42.242063395 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lcltd" (UniqueName: "kubernetes.io/projected/3f1f74fe-1d74-4b61-9bd2-083782908a35-kube-api-access-lcltd") pod "home-assistant-ix-chart-54bf5d89c8-mz95k" (UID: "3f1f74fe-1d74-4b61-9bd2-083782908a35") : object "ix-home-assistant"/"kube-root-ca.crt" not registered Jul 24 08:51:47 bluth k3s[11030]: I0724 08:51:47.909556 11030 trace.go:205] Trace[1581213309]: "Create" url:/api/v1/namespaces/ix-esphome/events,user-agent:k3s/v1.23.5+k3s (linux/amd64) kubernetes/fbfa51e,audit-id:fbe3af02-94a9-4c54-acce-7d2be034b167,client:127.0.0.1,accept:application/vnd.kubernetes.protobuf,application/json,protocol:HTTP/2.0 (24-Jul-2022 08:51:46.983) (total time: 926ms): Jul 24 08:51:47 bluth k3s[11030]: Trace[1581213309]: ---"Object stored in database" 926ms (08:51:47.909) Jul 24 08:51:47 bluth k3s[11030]: Trace[1581213309]: [926.201679ms] [926.201679ms] END Jul 24 08:51:48 bluth k3s[11030]: I0724 08:51:48.820803 11030 trace.go:205] Trace[275890512]: "Create" url:/api/v1/namespaces/ix-esphome/events,user-agent:k3s/v1.23.5+k3s (linux/amd64) kubernetes/fbfa51e,audit-id:b314b595-2af3-4e2d-82db-6d09b0407306,client:127.0.0.1,accept:application/vnd.kubernetes.protobuf,application/json,protocol:HTTP/2.0 (24-Jul-2022 08:51:47.910) (total time: 909ms): Jul 24 08:51:48 bluth k3s[11030]: Trace[275890512]: ---"Object stored in database" 909ms (08:51:48.820) Jul 24 08:51:48 bluth k3s[11030]: Trace[275890512]: [909.78279ms] [909.78279ms] END Jul 24 08:51:49 bluth k3s[11030]: E0724 08:51:49.746812 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:51:50 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T08:51:50.774+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 08:51:51 bluth k3s[11030]: E0724 08:51:51.481639 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-195f1a32-9c93-4737-983a-8dacee32e453 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:52:07.48162815 +0100 IST m=+53.969485033 (durationBeforeRetry 16s). Error: MountVolume.MountDevice failed for volume "pvc-195f1a32-9c93-4737-983a-8dacee32e453" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-195f1a32-9c93-4737-983a-8dacee32e453") pod "radarr-7486f78d7f-q4pn4" (UID: "be152b2b-fd1c-4a36-9081-ac6be2fb7901") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:51 bluth k3s[11030]: E0724 08:51:51.481658 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:52:07.481652837 +0100 IST m=+53.969509717 (durationBeforeRetry 16s). Error: MountVolume.MountDevice failed for volume "pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4") pod "sonarr-867d7dd548-4bn8s" (UID: "35ac75bf-b85f-4286-a228-ea8459dde305") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:51 bluth k3s[11030]: E0724 08:51:51.481826 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:52:07.481819578 +0100 IST m=+53.969676459 (durationBeforeRetry 16s). Error: MountVolume.MountDevice failed for volume "pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274") pod "prowlarr-5bcc99cc47-rgrsl" (UID: "afd760a7-8b6b-470e-aa87-8ed16adb34bb") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:51 bluth k3s[11030]: E0724 08:51:51.750420 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:51:51 bluth k3s[11030]: I0724 08:51:51.755218 11030 trace.go:205] Trace[1912533983]: "GuaranteedUpdate etcd3" type:*core.Event (24-Jul-2022 08:51:48.821) (total time: 2933ms): Jul 24 08:51:51 bluth k3s[11030]: Trace[1912533983]: ---"Transaction committed" 2932ms (08:51:51.755) Jul 24 08:51:51 bluth k3s[11030]: Trace[1912533983]: [2.933260627s] [2.933260627s] END Jul 24 08:51:51 bluth k3s[11030]: I0724 08:51:51.755314 11030 trace.go:205] Trace[1411933472]: "Patch" url:/api/v1/namespaces/ix-tdarr/events/tdarr-746d65b466-b7whb.1704b490dd96916f,user-agent:k3s/v1.23.5+k3s (linux/amd64) kubernetes/fbfa51e,audit-id:f70d1fab-0c65-4590-907e-37e303b03248,client:127.0.0.1,accept:application/vnd.kubernetes.protobuf,application/json,protocol:HTTP/2.0 (24-Jul-2022 08:51:48.821) (total time: 2933ms): Jul 24 08:51:51 bluth k3s[11030]: Trace[1411933472]: ---"Object stored in database" 2932ms (08:51:51.755) Jul 24 08:51:51 bluth k3s[11030]: Trace[1411933472]: [2.933398501s] [2.933398501s] END Jul 24 08:51:52 bluth k3s[11030]: E0724 08:51:52.389247 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-028dc34a-8c2e-4a0a-b507-c33a02541407 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:52:08.389235789 +0100 IST m=+54.877092669 (durationBeforeRetry 16s). Error: MountVolume.MountDevice failed for volume "pvc-028dc34a-8c2e-4a0a-b507-c33a02541407" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-028dc34a-8c2e-4a0a-b507-c33a02541407") pod "tdarr-746d65b466-b7whb" (UID: "6e3e9791-9bb3-487f-ae50-048fefbef8be") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:53 bluth k3s[11030]: I0724 08:51:53.232406 11030 trace.go:205] Trace[1375640030]: "GuaranteedUpdate etcd3" type:*core.Event (24-Jul-2022 08:51:51.757) (total time: 1474ms): Jul 24 08:51:53 bluth k3s[11030]: Trace[1375640030]: ---"Transaction committed" 1473ms (08:51:53.232) Jul 24 08:51:53 bluth k3s[11030]: Trace[1375640030]: [1.474745375s] [1.474745375s] END Jul 24 08:51:53 bluth k3s[11030]: I0724 08:51:53.232509 11030 trace.go:205] Trace[1773390478]: "Patch" url:/api/v1/namespaces/ix-tdarr/events/tdarr-746d65b466-b7whb.1704b490dd95ef33,user-agent:k3s/v1.23.5+k3s (linux/amd64) kubernetes/fbfa51e,audit-id:5b7713db-b8cd-4c42-b05b-017145bf9f14,client:127.0.0.1,accept:application/vnd.kubernetes.protobuf,application/json,protocol:HTTP/2.0 (24-Jul-2022 08:51:51.757) (total time: 1474ms): Jul 24 08:51:53 bluth k3s[11030]: Trace[1773390478]: ---"Object stored in database" 1473ms (08:51:53.232) Jul 24 08:51:53 bluth k3s[11030]: Trace[1773390478]: [1.474891096s] [1.474891096s] END Jul 24 08:51:53 bluth k3s[11030]: I0724 08:51:53.270289 11030 trace.go:205] Trace[1068176062]: "GuaranteedUpdate etcd3" type:*v1.Endpoints (24-Jul-2022 08:51:52.139) (total time: 1130ms): Jul 24 08:51:53 bluth k3s[11030]: Trace[1068176062]: ---"Transaction committed" 1130ms (08:51:53.270) Jul 24 08:51:53 bluth k3s[11030]: Trace[1068176062]: [1.130802189s] [1.130802189s] END Jul 24 08:51:53 bluth k3s[11030]: E0724 08:51:53.497357 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-3706f211-9694-4ec9-88b9-55f66b73b802 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:52:09.49734474 +0100 IST m=+55.985201625 (durationBeforeRetry 16s). Error: MountVolume.MountDevice failed for volume "pvc-3706f211-9694-4ec9-88b9-55f66b73b802" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-3706f211-9694-4ec9-88b9-55f66b73b802") pod "tdarr-746d65b466-b7whb" (UID: "6e3e9791-9bb3-487f-ae50-048fefbef8be") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:53 bluth k3s[11030]: E0724 08:51:53.497379 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-2e6b3d10-435c-4ce5-806d-61d551dcef05 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:52:09.497372645 +0100 IST m=+55.985229525 (durationBeforeRetry 16s). Error: MountVolume.MountDevice failed for volume "pvc-2e6b3d10-435c-4ce5-806d-61d551dcef05" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-2e6b3d10-435c-4ce5-806d-61d551dcef05") pod "tdarr-746d65b466-b7whb" (UID: "6e3e9791-9bb3-487f-ae50-048fefbef8be") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:53 bluth k3s[11030]: E0724 08:51:53.497531 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c41ca0c8-d532-4ca2-b6ca-fe3e9ddf87d2 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:52:09.497521971 +0100 IST m=+55.985378855 (durationBeforeRetry 16s). Error: MountVolume.MountDevice failed for volume "pvc-c41ca0c8-d532-4ca2-b6ca-fe3e9ddf87d2" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c41ca0c8-d532-4ca2-b6ca-fe3e9ddf87d2") pod "tdarr-746d65b466-b7whb" (UID: "6e3e9791-9bb3-487f-ae50-048fefbef8be") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:53 bluth k3s[11030]: E0724 08:51:53.497634 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-4e6fcc6b-6890-4751-a09a-75bb6cde146f podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:52:09.497626828 +0100 IST m=+55.985483708 (durationBeforeRetry 16s). Error: MountVolume.MountDevice failed for volume "pvc-4e6fcc6b-6890-4751-a09a-75bb6cde146f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-4e6fcc6b-6890-4751-a09a-75bb6cde146f") pod "tdarr-746d65b466-b7whb" (UID: "6e3e9791-9bb3-487f-ae50-048fefbef8be") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:53 bluth k3s[11030]: E0724 08:51:53.748627 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:51:54 bluth k3s[11030]: E0724 08:51:54.000862 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-06521873-c179-499e-9079-194ec6618fdd podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:52:10.000853377 +0100 IST m=+56.488710266 (durationBeforeRetry 16s). Error: MountVolume.MountDevice failed for volume "pvc-06521873-c179-499e-9079-194ec6618fdd" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-06521873-c179-499e-9079-194ec6618fdd") pod "esphome-658957fddc-sb8c8" (UID: "05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:54 bluth k3s[11030]: E0724 08:51:54.001299 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:52:10.001291334 +0100 IST m=+56.489148219 (durationBeforeRetry 16s). Error: MountVolume.MountDevice failed for volume "pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e") pod "esphome-658957fddc-sb8c8" (UID: "05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:54 bluth k3s[11030]: E0724 08:51:54.503992 11030 projected.go:293] Couldn't get configMap ix-esphome/kube-root-ca.crt: object "ix-esphome"/"kube-root-ca.crt" not registered Jul 24 08:51:54 bluth k3s[11030]: E0724 08:51:54.504318 11030 projected.go:199] Error preparing data for projected volume kube-api-access-xr6bj for pod ix-esphome/esphome-658957fddc-sb8c8: object "ix-esphome"/"kube-root-ca.crt" not registered Jul 24 08:51:54 bluth k3s[11030]: E0724 08:51:54.504411 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/projected/05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b-kube-api-access-xr6bj podName:05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b nodeName:}" failed. No retries permitted until 2022-07-24 08:52:10.504400601 +0100 IST m=+56.992257482 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xr6bj" (UniqueName: "kubernetes.io/projected/05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b-kube-api-access-xr6bj") pod "esphome-658957fddc-sb8c8" (UID: "05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b") : object "ix-esphome"/"kube-root-ca.crt" not registered Jul 24 08:51:55 bluth k3s[11030]: E0724 08:51:55.108192 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-58562e12-4adc-4e83-8131-848f1b39bf00 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:52:11.108182703 +0100 IST m=+57.596039587 (durationBeforeRetry 16s). Error: MountVolume.MountDevice failed for volume "pvc-58562e12-4adc-4e83-8131-848f1b39bf00" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-58562e12-4adc-4e83-8131-848f1b39bf00") pod "qbittorrent-6d7cf9f7cd-7n6c6" (UID: "2fac2fcc-a503-4887-810c-c2ff11c07852") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:51:55 bluth k3s[11030]: E0724 08:51:55.410417 11030 projected.go:293] Couldn't get configMap ix-sonarr/kube-root-ca.crt: object "ix-sonarr"/"kube-root-ca.crt" not registered Jul 24 08:51:55 bluth k3s[11030]: E0724 08:51:55.410544 11030 projected.go:199] Error preparing data for projected volume kube-api-access-2v58k for pod ix-sonarr/sonarr-867d7dd548-4bn8s: object "ix-sonarr"/"kube-root-ca.crt" not registered Jul 24 08:51:55 bluth k3s[11030]: E0724 08:51:55.410623 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/projected/35ac75bf-b85f-4286-a228-ea8459dde305-kube-api-access-2v58k podName:35ac75bf-b85f-4286-a228-ea8459dde305 nodeName:}" failed. No retries permitted until 2022-07-24 08:52:11.410612219 +0100 IST m=+57.898469104 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2v58k" (UniqueName: "kubernetes.io/projected/35ac75bf-b85f-4286-a228-ea8459dde305-kube-api-access-2v58k") pod "sonarr-867d7dd548-4bn8s" (UID: "35ac75bf-b85f-4286-a228-ea8459dde305") : object "ix-sonarr"/"kube-root-ca.crt" not registered Jul 24 08:51:55 bluth k3s[11030]: E0724 08:51:55.747141 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:51:55 bluth k3s[11030]: E0724 08:51:55.813940 11030 projected.go:293] Couldn't get configMap ix-home-assistant/kube-root-ca.crt: object "ix-home-assistant"/"kube-root-ca.crt" not registered Jul 24 08:51:55 bluth k3s[11030]: E0724 08:51:55.814061 11030 projected.go:199] Error preparing data for projected volume kube-api-access-lcltd for pod ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-mz95k: object "ix-home-assistant"/"kube-root-ca.crt" not registered Jul 24 08:51:55 bluth k3s[11030]: E0724 08:51:55.814142 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/projected/3f1f74fe-1d74-4b61-9bd2-083782908a35-kube-api-access-lcltd podName:3f1f74fe-1d74-4b61-9bd2-083782908a35 nodeName:}" failed. No retries permitted until 2022-07-24 08:52:11.814130834 +0100 IST m=+58.301987718 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lcltd" (UniqueName: "kubernetes.io/projected/3f1f74fe-1d74-4b61-9bd2-083782908a35-kube-api-access-lcltd") pod "home-assistant-ix-chart-54bf5d89c8-mz95k" (UID: "3f1f74fe-1d74-4b61-9bd2-083782908a35") : object "ix-home-assistant"/"kube-root-ca.crt" not registered Jul 24 08:51:57 bluth k3s[11030]: E0724 08:51:57.748353 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:51:59 bluth k3s[11030]: E0724 08:51:59.746960 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:52:01 bluth k3s[11030]: E0724 08:52:01.748253 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:52:03 bluth k3s[11030]: E0724 08:52:03.747037 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:52:05 bluth k3s[11030]: E0724 08:52:05.751850 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:52:07 bluth k3s[11030]: E0724 08:52:07.494661 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:52:39.494651776 +0100 IST m=+85.982508657 (durationBeforeRetry 32s). Error: MountVolume.MountDevice failed for volume "pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4") pod "sonarr-867d7dd548-4bn8s" (UID: "35ac75bf-b85f-4286-a228-ea8459dde305") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:52:07 bluth k3s[11030]: E0724 08:52:07.495154 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:52:39.495147815 +0100 IST m=+85.983004699 (durationBeforeRetry 32s). Error: MountVolume.MountDevice failed for volume "pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274") pod "prowlarr-5bcc99cc47-rgrsl" (UID: "afd760a7-8b6b-470e-aa87-8ed16adb34bb") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:52:07 bluth k3s[11030]: E0724 08:52:07.495365 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-195f1a32-9c93-4737-983a-8dacee32e453 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:52:39.495358771 +0100 IST m=+85.983215655 (durationBeforeRetry 32s). Error: MountVolume.MountDevice failed for volume "pvc-195f1a32-9c93-4737-983a-8dacee32e453" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-195f1a32-9c93-4737-983a-8dacee32e453") pod "radarr-7486f78d7f-q4pn4" (UID: "be152b2b-fd1c-4a36-9081-ac6be2fb7901") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:52:07 bluth k3s[11030]: E0724 08:52:07.747760 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:52:08 bluth k3s[11030]: E0724 08:52:08.400999 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-028dc34a-8c2e-4a0a-b507-c33a02541407 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:52:40.400989676 +0100 IST m=+86.888846556 (durationBeforeRetry 32s). Error: MountVolume.MountDevice failed for volume "pvc-028dc34a-8c2e-4a0a-b507-c33a02541407" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-028dc34a-8c2e-4a0a-b507-c33a02541407") pod "tdarr-746d65b466-b7whb" (UID: "6e3e9791-9bb3-487f-ae50-048fefbef8be") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:52:09 bluth k3s[11030]: E0724 08:52:09.511113 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-3706f211-9694-4ec9-88b9-55f66b73b802 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:52:41.511103078 +0100 IST m=+87.998959959 (durationBeforeRetry 32s). Error: MountVolume.MountDevice failed for volume "pvc-3706f211-9694-4ec9-88b9-55f66b73b802" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-3706f211-9694-4ec9-88b9-55f66b73b802") pod "tdarr-746d65b466-b7whb" (UID: "6e3e9791-9bb3-487f-ae50-048fefbef8be") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:52:09 bluth k3s[11030]: E0724 08:52:09.511368 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-2e6b3d10-435c-4ce5-806d-61d551dcef05 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:52:41.511361803 +0100 IST m=+87.999218683 (durationBeforeRetry 32s). Error: MountVolume.MountDevice failed for volume "pvc-2e6b3d10-435c-4ce5-806d-61d551dcef05" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-2e6b3d10-435c-4ce5-806d-61d551dcef05") pod "tdarr-746d65b466-b7whb" (UID: "6e3e9791-9bb3-487f-ae50-048fefbef8be") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:52:09 bluth k3s[11030]: E0724 08:52:09.511563 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-4e6fcc6b-6890-4751-a09a-75bb6cde146f podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:52:41.511557301 +0100 IST m=+87.999414182 (durationBeforeRetry 32s). Error: MountVolume.MountDevice failed for volume "pvc-4e6fcc6b-6890-4751-a09a-75bb6cde146f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-4e6fcc6b-6890-4751-a09a-75bb6cde146f") pod "tdarr-746d65b466-b7whb" (UID: "6e3e9791-9bb3-487f-ae50-048fefbef8be") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:52:09 bluth k3s[11030]: E0724 08:52:09.511758 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c41ca0c8-d532-4ca2-b6ca-fe3e9ddf87d2 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:52:41.511752433 +0100 IST m=+87.999609313 (durationBeforeRetry 32s). Error: MountVolume.MountDevice failed for volume "pvc-c41ca0c8-d532-4ca2-b6ca-fe3e9ddf87d2" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c41ca0c8-d532-4ca2-b6ca-fe3e9ddf87d2") pod "tdarr-746d65b466-b7whb" (UID: "6e3e9791-9bb3-487f-ae50-048fefbef8be") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:52:09 bluth k3s[11030]: E0724 08:52:09.748505 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:52:10 bluth k3s[11030]: E0724 08:52:10.014547 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-06521873-c179-499e-9079-194ec6618fdd podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:52:42.014538037 +0100 IST m=+88.502394917 (durationBeforeRetry 32s). Error: MountVolume.MountDevice failed for volume "pvc-06521873-c179-499e-9079-194ec6618fdd" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-06521873-c179-499e-9079-194ec6618fdd") pod "esphome-658957fddc-sb8c8" (UID: "05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:52:10 bluth k3s[11030]: E0724 08:52:10.014914 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:52:42.014906838 +0100 IST m=+88.502763719 (durationBeforeRetry 32s). Error: MountVolume.MountDevice failed for volume "pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e") pod "esphome-658957fddc-sb8c8" (UID: "05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:52:10 bluth k3s[11030]: E0724 08:52:10.518264 11030 projected.go:293] Couldn't get configMap ix-esphome/kube-root-ca.crt: object "ix-esphome"/"kube-root-ca.crt" not registered Jul 24 08:52:10 bluth k3s[11030]: E0724 08:52:10.518589 11030 projected.go:199] Error preparing data for projected volume kube-api-access-xr6bj for pod ix-esphome/esphome-658957fddc-sb8c8: object "ix-esphome"/"kube-root-ca.crt" not registered Jul 24 08:52:10 bluth k3s[11030]: E0724 08:52:10.518673 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/projected/05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b-kube-api-access-xr6bj podName:05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b nodeName:}" failed. No retries permitted until 2022-07-24 08:52:42.518662206 +0100 IST m=+89.006519086 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-xr6bj" (UniqueName: "kubernetes.io/projected/05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b-kube-api-access-xr6bj") pod "esphome-658957fddc-sb8c8" (UID: "05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b") : object "ix-esphome"/"kube-root-ca.crt" not registered Jul 24 08:52:11 bluth k3s[11030]: I0724 08:52:11.063848 11030 scope.go:110] "RemoveContainer" containerID="42f6521e6785b9192e442c8f8b5075d1a19fe2ff86adb83a34c2ee72c820662f" Jul 24 08:52:11 bluth k3s[11030]: E0724 08:52:11.065739 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 42f6521e6785b9192e442c8f8b5075d1a19fe2ff86adb83a34c2ee72c820662f" containerID="42f6521e6785b9192e442c8f8b5075d1a19fe2ff86adb83a34c2ee72c820662f" Jul 24 08:52:11 bluth k3s[11030]: I0724 08:52:11.065767 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:42f6521e6785b9192e442c8f8b5075d1a19fe2ff86adb83a34c2ee72c820662f} err="failed to get container status \"42f6521e6785b9192e442c8f8b5075d1a19fe2ff86adb83a34c2ee72c820662f\": rpc error: code = Unknown desc = Error: No such container: 42f6521e6785b9192e442c8f8b5075d1a19fe2ff86adb83a34c2ee72c820662f" Jul 24 08:52:11 bluth k3s[11030]: I0724 08:52:11.065778 11030 scope.go:110] "RemoveContainer" containerID="51ae3a4c735e322dac9d2fdb80a850a3d38ba3375fb16e1609d34c9c5ae191a2" Jul 24 08:52:11 bluth k3s[11030]: I0724 08:52:11.068757 11030 scope.go:110] "RemoveContainer" containerID="3792d5854b3664bf23548595384863a12958ee8842cfca99582b3f61ff8be7a4" Jul 24 08:52:11 bluth k3s[11030]: E0724 08:52:11.121682 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-58562e12-4adc-4e83-8131-848f1b39bf00 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:52:43.121672733 +0100 IST m=+89.609529613 (durationBeforeRetry 32s). Error: MountVolume.MountDevice failed for volume "pvc-58562e12-4adc-4e83-8131-848f1b39bf00" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-58562e12-4adc-4e83-8131-848f1b39bf00") pod "qbittorrent-6d7cf9f7cd-7n6c6" (UID: "2fac2fcc-a503-4887-810c-c2ff11c07852") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:52:11 bluth k3s[11030]: I0724 08:52:11.350936 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="42f6521e6785b9192e442c8f8b5075d1a19fe2ff86adb83a34c2ee72c820662f" Jul 24 08:52:11 bluth k3s[11030]: I0724 08:52:11.355462 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="83de9bb2e016582dcd1febf8fe41eaaf734e62ad8e27a5befa1c8287cd8cc1a2" Jul 24 08:52:11 bluth k3s[11030]: E0724 08:52:11.423586 11030 projected.go:293] Couldn't get configMap ix-sonarr/kube-root-ca.crt: object "ix-sonarr"/"kube-root-ca.crt" not registered Jul 24 08:52:11 bluth k3s[11030]: E0724 08:52:11.423607 11030 projected.go:199] Error preparing data for projected volume kube-api-access-2v58k for pod ix-sonarr/sonarr-867d7dd548-4bn8s: object "ix-sonarr"/"kube-root-ca.crt" not registered Jul 24 08:52:11 bluth k3s[11030]: E0724 08:52:11.423635 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/projected/35ac75bf-b85f-4286-a228-ea8459dde305-kube-api-access-2v58k podName:35ac75bf-b85f-4286-a228-ea8459dde305 nodeName:}" failed. No retries permitted until 2022-07-24 08:52:43.423625588 +0100 IST m=+89.911482472 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2v58k" (UniqueName: "kubernetes.io/projected/35ac75bf-b85f-4286-a228-ea8459dde305-kube-api-access-2v58k") pod "sonarr-867d7dd548-4bn8s" (UID: "35ac75bf-b85f-4286-a228-ea8459dde305") : object "ix-sonarr"/"kube-root-ca.crt" not registered Jul 24 08:52:11 bluth k3s[11030]: E0724 08:52:11.746638 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:52:11 bluth k3s[11030]: E0724 08:52:11.826694 11030 projected.go:293] Couldn't get configMap ix-home-assistant/kube-root-ca.crt: object "ix-home-assistant"/"kube-root-ca.crt" not registered Jul 24 08:52:11 bluth k3s[11030]: E0724 08:52:11.826715 11030 projected.go:199] Error preparing data for projected volume kube-api-access-lcltd for pod ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-mz95k: object "ix-home-assistant"/"kube-root-ca.crt" not registered Jul 24 08:52:11 bluth k3s[11030]: E0724 08:52:11.826743 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/projected/3f1f74fe-1d74-4b61-9bd2-083782908a35-kube-api-access-lcltd podName:3f1f74fe-1d74-4b61-9bd2-083782908a35 nodeName:}" failed. No retries permitted until 2022-07-24 08:52:43.826733646 +0100 IST m=+90.314590526 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lcltd" (UniqueName: "kubernetes.io/projected/3f1f74fe-1d74-4b61-9bd2-083782908a35-kube-api-access-lcltd") pod "home-assistant-ix-chart-54bf5d89c8-mz95k" (UID: "3f1f74fe-1d74-4b61-9bd2-083782908a35") : object "ix-home-assistant"/"kube-root-ca.crt" not registered Jul 24 08:52:13 bluth k3s[11030]: E0724 08:52:13.747669 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:52:14 bluth k3s[11030]: I0724 08:52:14.026485 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="0b9afc6eaa2accf484d333b3f1cf4b621ed551bbc2bf8b1198218f3bd645c664" Jul 24 08:52:15 bluth k3s[11030]: E0724 08:52:15.748112 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:52:17 bluth k3s[11030]: E0724 08:52:17.746829 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:52:19 bluth k3s[11030]: E0724 08:52:19.756487 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:52:21 bluth k3s[11030]: E0724 08:52:21.746479 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:52:23 bluth k3s[11030]: E0724 08:52:23.749204 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:52:24 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T08:52:24.582+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 08:52:25 bluth k3s[11030]: E0724 08:52:25.747043 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:52:26 bluth k3s[11030]: I0724 08:52:26.708224 11030 scope.go:110] "RemoveContainer" containerID="a0136712c1af6ba5698a1f924b41cdf3819b6d1939998b2a7afa2d2e33da4f78" Jul 24 08:52:26 bluth k3s[11030]: I0724 08:52:26.725739 11030 scope.go:110] "RemoveContainer" containerID="a0136712c1af6ba5698a1f924b41cdf3819b6d1939998b2a7afa2d2e33da4f78" Jul 24 08:52:27 bluth k3s[11030]: E0724 08:52:27.747754 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:52:29 bluth k3s[11030]: E0724 08:52:29.516435 11030 remote_runtime.go:505] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to remove container \"a0136712c1af6ba5698a1f924b41cdf3819b6d1939998b2a7afa2d2e33da4f78\": Error response from daemon: removal of container a0136712c1af6ba5698a1f924b41cdf3819b6d1939998b2a7afa2d2e33da4f78 is already in progress" containerID="a0136712c1af6ba5698a1f924b41cdf3819b6d1939998b2a7afa2d2e33da4f78" Jul 24 08:52:29 bluth k3s[11030]: E0724 08:52:29.516570 11030 kuberuntime_gc.go:146] "Failed to remove container" err="rpc error: code = Unknown desc = failed to remove container \"a0136712c1af6ba5698a1f924b41cdf3819b6d1939998b2a7afa2d2e33da4f78\": Error response from daemon: removal of container a0136712c1af6ba5698a1f924b41cdf3819b6d1939998b2a7afa2d2e33da4f78 is already in progress" containerID="a0136712c1af6ba5698a1f924b41cdf3819b6d1939998b2a7afa2d2e33da4f78" Jul 24 08:52:29 bluth k3s[11030]: I0724 08:52:29.516584 11030 scope.go:110] "RemoveContainer" containerID="1739236b851a117da4b61c694e91f2518258a69b0c7751eb467912def4e79df3" Jul 24 08:52:29 bluth k3s[11030]: I0724 08:52:29.522675 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="51ae3a4c735e322dac9d2fdb80a850a3d38ba3375fb16e1609d34c9c5ae191a2" Jul 24 08:52:29 bluth k3s[11030]: I0724 08:52:29.526014 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="3792d5854b3664bf23548595384863a12958ee8842cfca99582b3f61ff8be7a4" Jul 24 08:52:29 bluth k3s[11030]: E0724 08:52:29.761170 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:52:31 bluth k3s[11030]: E0724 08:52:31.748359 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:52:33 bluth k3s[11030]: E0724 08:52:33.746698 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:52:35 bluth k3s[11030]: E0724 08:52:35.747471 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:52:36 bluth k3s[11030]: I0724 08:52:36.992740 11030 scope.go:110] "RemoveContainer" containerID="1739236b851a117da4b61c694e91f2518258a69b0c7751eb467912def4e79df3" Jul 24 08:52:36 bluth k3s[11030]: E0724 08:52:36.995836 11030 remote_runtime.go:505] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to remove container \"1739236b851a117da4b61c694e91f2518258a69b0c7751eb467912def4e79df3\": Error response from daemon: removal of container 1739236b851a117da4b61c694e91f2518258a69b0c7751eb467912def4e79df3 is already in progress" containerID="1739236b851a117da4b61c694e91f2518258a69b0c7751eb467912def4e79df3" Jul 24 08:52:36 bluth k3s[11030]: I0724 08:52:36.995862 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:1739236b851a117da4b61c694e91f2518258a69b0c7751eb467912def4e79df3} err="rpc error: code = Unknown desc = failed to remove container \"1739236b851a117da4b61c694e91f2518258a69b0c7751eb467912def4e79df3\": Error response from daemon: removal of container 1739236b851a117da4b61c694e91f2518258a69b0c7751eb467912def4e79df3 is already in progress" Jul 24 08:52:36 bluth k3s[11030]: I0724 08:52:36.995871 11030 scope.go:110] "RemoveContainer" containerID="a0136712c1af6ba5698a1f924b41cdf3819b6d1939998b2a7afa2d2e33da4f78" Jul 24 08:52:36 bluth k3s[11030]: E0724 08:52:36.996114 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: a0136712c1af6ba5698a1f924b41cdf3819b6d1939998b2a7afa2d2e33da4f78" containerID="a0136712c1af6ba5698a1f924b41cdf3819b6d1939998b2a7afa2d2e33da4f78" Jul 24 08:52:36 bluth k3s[11030]: I0724 08:52:36.996133 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:a0136712c1af6ba5698a1f924b41cdf3819b6d1939998b2a7afa2d2e33da4f78} err="failed to get container status \"a0136712c1af6ba5698a1f924b41cdf3819b6d1939998b2a7afa2d2e33da4f78\": rpc error: code = Unknown desc = Error: No such container: a0136712c1af6ba5698a1f924b41cdf3819b6d1939998b2a7afa2d2e33da4f78" Jul 24 08:52:36 bluth k3s[11030]: I0724 08:52:36.996141 11030 scope.go:110] "RemoveContainer" containerID="1739236b851a117da4b61c694e91f2518258a69b0c7751eb467912def4e79df3" Jul 24 08:52:36 bluth k3s[11030]: E0724 08:52:36.997502 11030 remote_runtime.go:505] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to remove container \"1739236b851a117da4b61c694e91f2518258a69b0c7751eb467912def4e79df3\": Error response from daemon: removal of container 1739236b851a117da4b61c694e91f2518258a69b0c7751eb467912def4e79df3 is already in progress" containerID="1739236b851a117da4b61c694e91f2518258a69b0c7751eb467912def4e79df3" Jul 24 08:52:36 bluth k3s[11030]: I0724 08:52:36.997522 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:1739236b851a117da4b61c694e91f2518258a69b0c7751eb467912def4e79df3} err="rpc error: code = Unknown desc = failed to remove container \"1739236b851a117da4b61c694e91f2518258a69b0c7751eb467912def4e79df3\": Error response from daemon: removal of container 1739236b851a117da4b61c694e91f2518258a69b0c7751eb467912def4e79df3 is already in progress" Jul 24 08:52:36 bluth k3s[11030]: I0724 08:52:36.997530 11030 scope.go:110] "RemoveContainer" containerID="51ae3a4c735e322dac9d2fdb80a850a3d38ba3375fb16e1609d34c9c5ae191a2" Jul 24 08:52:36 bluth k3s[11030]: E0724 08:52:36.997775 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 51ae3a4c735e322dac9d2fdb80a850a3d38ba3375fb16e1609d34c9c5ae191a2" containerID="51ae3a4c735e322dac9d2fdb80a850a3d38ba3375fb16e1609d34c9c5ae191a2" Jul 24 08:52:36 bluth k3s[11030]: I0724 08:52:36.997793 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:51ae3a4c735e322dac9d2fdb80a850a3d38ba3375fb16e1609d34c9c5ae191a2} err="failed to get container status \"51ae3a4c735e322dac9d2fdb80a850a3d38ba3375fb16e1609d34c9c5ae191a2\": rpc error: code = Unknown desc = Error: No such container: 51ae3a4c735e322dac9d2fdb80a850a3d38ba3375fb16e1609d34c9c5ae191a2" Jul 24 08:52:36 bluth k3s[11030]: I0724 08:52:36.997800 11030 scope.go:110] "RemoveContainer" containerID="54cfb20a79b2d5e5f6eaf64dadabfffd4916dde065da6801479b5a9956b68217" Jul 24 08:52:37 bluth k3s[11030]: I0724 08:52:37.671777 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="a0136712c1af6ba5698a1f924b41cdf3819b6d1939998b2a7afa2d2e33da4f78" Jul 24 08:52:37 bluth k3s[11030]: I0724 08:52:37.722732 11030 scope.go:110] "RemoveContainer" containerID="9a9d2fbb90507d877353427182ae2a0e0147652b13e44a236d6531e7a9fde18b" Jul 24 08:52:37 bluth k3s[11030]: E0724 08:52:37.747975 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:52:38 bluth k3s[11030]: I0724 08:52:38.687715 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="1739236b851a117da4b61c694e91f2518258a69b0c7751eb467912def4e79df3" Jul 24 08:52:39 bluth k3s[11030]: E0724 08:52:39.537967 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:53:43.537956473 +0100 IST m=+150.025813353 (durationBeforeRetry 1m4s). Error: MountVolume.MountDevice failed for volume "pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4") pod "sonarr-867d7dd548-4bn8s" (UID: "35ac75bf-b85f-4286-a228-ea8459dde305") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:52:39 bluth k3s[11030]: E0724 08:52:39.538479 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:53:43.538470878 +0100 IST m=+150.026327762 (durationBeforeRetry 1m4s). Error: MountVolume.MountDevice failed for volume "pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274") pod "prowlarr-5bcc99cc47-rgrsl" (UID: "afd760a7-8b6b-470e-aa87-8ed16adb34bb") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:52:39 bluth k3s[11030]: E0724 08:52:39.538637 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-195f1a32-9c93-4737-983a-8dacee32e453 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:53:43.538630548 +0100 IST m=+150.026487432 (durationBeforeRetry 1m4s). Error: MountVolume.MountDevice failed for volume "pvc-195f1a32-9c93-4737-983a-8dacee32e453" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-195f1a32-9c93-4737-983a-8dacee32e453") pod "radarr-7486f78d7f-q4pn4" (UID: "be152b2b-fd1c-4a36-9081-ac6be2fb7901") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:52:39 bluth k3s[11030]: E0724 08:52:39.746539 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:52:40 bluth k3s[11030]: E0724 08:52:40.444212 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-028dc34a-8c2e-4a0a-b507-c33a02541407 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:53:44.44420191 +0100 IST m=+150.932058790 (durationBeforeRetry 1m4s). Error: MountVolume.MountDevice failed for volume "pvc-028dc34a-8c2e-4a0a-b507-c33a02541407" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-028dc34a-8c2e-4a0a-b507-c33a02541407") pod "tdarr-746d65b466-b7whb" (UID: "6e3e9791-9bb3-487f-ae50-048fefbef8be") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:52:41 bluth k3s[11030]: I0724 08:52:41.194826 11030 scope.go:110] "RemoveContainer" containerID="54cfb20a79b2d5e5f6eaf64dadabfffd4916dde065da6801479b5a9956b68217" Jul 24 08:52:41 bluth k3s[11030]: E0724 08:52:41.195746 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 54cfb20a79b2d5e5f6eaf64dadabfffd4916dde065da6801479b5a9956b68217" containerID="54cfb20a79b2d5e5f6eaf64dadabfffd4916dde065da6801479b5a9956b68217" Jul 24 08:52:41 bluth k3s[11030]: I0724 08:52:41.195771 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:54cfb20a79b2d5e5f6eaf64dadabfffd4916dde065da6801479b5a9956b68217} err="failed to get container status \"54cfb20a79b2d5e5f6eaf64dadabfffd4916dde065da6801479b5a9956b68217\": rpc error: code = Unknown desc = Error: No such container: 54cfb20a79b2d5e5f6eaf64dadabfffd4916dde065da6801479b5a9956b68217" Jul 24 08:52:41 bluth k3s[11030]: I0724 08:52:41.195782 11030 scope.go:110] "RemoveContainer" containerID="097557fceb4d955ca22ba5e8c9129450fda31f9a02a736c640feff3f7f75d076" Jul 24 08:52:41 bluth k3s[11030]: E0724 08:52:41.551074 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-3706f211-9694-4ec9-88b9-55f66b73b802 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:53:45.55106418 +0100 IST m=+152.038921064 (durationBeforeRetry 1m4s). Error: MountVolume.MountDevice failed for volume "pvc-3706f211-9694-4ec9-88b9-55f66b73b802" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-3706f211-9694-4ec9-88b9-55f66b73b802") pod "tdarr-746d65b466-b7whb" (UID: "6e3e9791-9bb3-487f-ae50-048fefbef8be") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:52:41 bluth k3s[11030]: E0724 08:52:41.551318 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-2e6b3d10-435c-4ce5-806d-61d551dcef05 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:53:45.551309749 +0100 IST m=+152.039166630 (durationBeforeRetry 1m4s). Error: MountVolume.MountDevice failed for volume "pvc-2e6b3d10-435c-4ce5-806d-61d551dcef05" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-2e6b3d10-435c-4ce5-806d-61d551dcef05") pod "tdarr-746d65b466-b7whb" (UID: "6e3e9791-9bb3-487f-ae50-048fefbef8be") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:52:41 bluth k3s[11030]: E0724 08:52:41.551517 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-4e6fcc6b-6890-4751-a09a-75bb6cde146f podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:53:45.551510831 +0100 IST m=+152.039367712 (durationBeforeRetry 1m4s). Error: MountVolume.MountDevice failed for volume "pvc-4e6fcc6b-6890-4751-a09a-75bb6cde146f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-4e6fcc6b-6890-4751-a09a-75bb6cde146f") pod "tdarr-746d65b466-b7whb" (UID: "6e3e9791-9bb3-487f-ae50-048fefbef8be") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:52:41 bluth k3s[11030]: E0724 08:52:41.551713 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c41ca0c8-d532-4ca2-b6ca-fe3e9ddf87d2 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:53:45.551706089 +0100 IST m=+152.039562969 (durationBeforeRetry 1m4s). Error: MountVolume.MountDevice failed for volume "pvc-c41ca0c8-d532-4ca2-b6ca-fe3e9ddf87d2" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c41ca0c8-d532-4ca2-b6ca-fe3e9ddf87d2") pod "tdarr-746d65b466-b7whb" (UID: "6e3e9791-9bb3-487f-ae50-048fefbef8be") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:52:41 bluth k3s[11030]: E0724 08:52:41.722696 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: aae50ebfe6ac41f8d0dc1ad40636699bee4580fe3b3bda1d98c1565da66e94d0" podSandboxID="aae50ebfe6ac41f8d0dc1ad40636699bee4580fe3b3bda1d98c1565da66e94d0" pod="kube-system/openebs-zfs-node-bf7kk" Jul 24 08:52:41 bluth k3s[11030]: I0724 08:52:41.725416 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="54cfb20a79b2d5e5f6eaf64dadabfffd4916dde065da6801479b5a9956b68217" Jul 24 08:52:41 bluth k3s[11030]: E0724 08:52:41.747596 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:52:42 bluth k3s[11030]: E0724 08:52:42.054188 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-06521873-c179-499e-9079-194ec6618fdd podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:53:46.054178996 +0100 IST m=+152.542035876 (durationBeforeRetry 1m4s). Error: MountVolume.MountDevice failed for volume "pvc-06521873-c179-499e-9079-194ec6618fdd" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-06521873-c179-499e-9079-194ec6618fdd") pod "esphome-658957fddc-sb8c8" (UID: "05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:52:42 bluth k3s[11030]: E0724 08:52:42.054610 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:53:46.05460143 +0100 IST m=+152.542458315 (durationBeforeRetry 1m4s). Error: MountVolume.MountDevice failed for volume "pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e") pod "esphome-658957fddc-sb8c8" (UID: "05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:52:42 bluth k3s[11030]: E0724 08:52:42.556678 11030 projected.go:293] Couldn't get configMap ix-esphome/kube-root-ca.crt: object "ix-esphome"/"kube-root-ca.crt" not registered Jul 24 08:52:42 bluth k3s[11030]: E0724 08:52:42.557007 11030 projected.go:199] Error preparing data for projected volume kube-api-access-xr6bj for pod ix-esphome/esphome-658957fddc-sb8c8: object "ix-esphome"/"kube-root-ca.crt" not registered Jul 24 08:52:42 bluth k3s[11030]: E0724 08:52:42.557043 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/projected/05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b-kube-api-access-xr6bj podName:05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b nodeName:}" failed. No retries permitted until 2022-07-24 08:53:46.557032077 +0100 IST m=+153.044888957 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xr6bj" (UniqueName: "kubernetes.io/projected/05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b-kube-api-access-xr6bj") pod "esphome-658957fddc-sb8c8" (UID: "05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b") : object "ix-esphome"/"kube-root-ca.crt" not registered Jul 24 08:52:42 bluth k3s[11030]: E0724 08:52:42.735261 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: aae50ebfe6ac41f8d0dc1ad40636699bee4580fe3b3bda1d98c1565da66e94d0" podSandboxID="aae50ebfe6ac41f8d0dc1ad40636699bee4580fe3b3bda1d98c1565da66e94d0" pod="kube-system/openebs-zfs-node-bf7kk" Jul 24 08:52:42 bluth k3s[11030]: E0724 08:52:42.736538 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: aae50ebfe6ac41f8d0dc1ad40636699bee4580fe3b3bda1d98c1565da66e94d0" podSandboxID="aae50ebfe6ac41f8d0dc1ad40636699bee4580fe3b3bda1d98c1565da66e94d0" pod="kube-system/openebs-zfs-node-bf7kk" Jul 24 08:52:43 bluth k3s[11030]: E0724 08:52:43.160620 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-58562e12-4adc-4e83-8131-848f1b39bf00 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:53:47.160609825 +0100 IST m=+153.648466705 (durationBeforeRetry 1m4s). Error: MountVolume.MountDevice failed for volume "pvc-58562e12-4adc-4e83-8131-848f1b39bf00" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-58562e12-4adc-4e83-8131-848f1b39bf00") pod "qbittorrent-6d7cf9f7cd-7n6c6" (UID: "2fac2fcc-a503-4887-810c-c2ff11c07852") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:52:43 bluth k3s[11030]: E0724 08:52:43.461455 11030 projected.go:293] Couldn't get configMap ix-sonarr/kube-root-ca.crt: object "ix-sonarr"/"kube-root-ca.crt" not registered Jul 24 08:52:43 bluth k3s[11030]: E0724 08:52:43.461479 11030 projected.go:199] Error preparing data for projected volume kube-api-access-2v58k for pod ix-sonarr/sonarr-867d7dd548-4bn8s: object "ix-sonarr"/"kube-root-ca.crt" not registered Jul 24 08:52:43 bluth k3s[11030]: E0724 08:52:43.461699 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/projected/35ac75bf-b85f-4286-a228-ea8459dde305-kube-api-access-2v58k podName:35ac75bf-b85f-4286-a228-ea8459dde305 nodeName:}" failed. No retries permitted until 2022-07-24 08:53:47.461498727 +0100 IST m=+153.949355611 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2v58k" (UniqueName: "kubernetes.io/projected/35ac75bf-b85f-4286-a228-ea8459dde305-kube-api-access-2v58k") pod "sonarr-867d7dd548-4bn8s" (UID: "35ac75bf-b85f-4286-a228-ea8459dde305") : object "ix-sonarr"/"kube-root-ca.crt" not registered Jul 24 08:52:43 bluth k3s[11030]: E0724 08:52:43.747291 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: aae50ebfe6ac41f8d0dc1ad40636699bee4580fe3b3bda1d98c1565da66e94d0" podSandboxID="aae50ebfe6ac41f8d0dc1ad40636699bee4580fe3b3bda1d98c1565da66e94d0" pod="kube-system/openebs-zfs-node-bf7kk" Jul 24 08:52:43 bluth k3s[11030]: E0724 08:52:43.749182 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: aae50ebfe6ac41f8d0dc1ad40636699bee4580fe3b3bda1d98c1565da66e94d0" podSandboxID="aae50ebfe6ac41f8d0dc1ad40636699bee4580fe3b3bda1d98c1565da66e94d0" pod="kube-system/openebs-zfs-node-bf7kk" Jul 24 08:52:43 bluth k3s[11030]: E0724 08:52:43.749748 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:52:43 bluth k3s[11030]: E0724 08:52:43.863811 11030 projected.go:293] Couldn't get configMap ix-home-assistant/kube-root-ca.crt: object "ix-home-assistant"/"kube-root-ca.crt" not registered Jul 24 08:52:43 bluth k3s[11030]: E0724 08:52:43.863836 11030 projected.go:199] Error preparing data for projected volume kube-api-access-lcltd for pod ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-mz95k: object "ix-home-assistant"/"kube-root-ca.crt" not registered Jul 24 08:52:43 bluth k3s[11030]: E0724 08:52:43.863864 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/projected/3f1f74fe-1d74-4b61-9bd2-083782908a35-kube-api-access-lcltd podName:3f1f74fe-1d74-4b61-9bd2-083782908a35 nodeName:}" failed. No retries permitted until 2022-07-24 08:53:47.863854326 +0100 IST m=+154.351711206 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lcltd" (UniqueName: "kubernetes.io/projected/3f1f74fe-1d74-4b61-9bd2-083782908a35-kube-api-access-lcltd") pod "home-assistant-ix-chart-54bf5d89c8-mz95k" (UID: "3f1f74fe-1d74-4b61-9bd2-083782908a35") : object "ix-home-assistant"/"kube-root-ca.crt" not registered Jul 24 08:52:44 bluth k3s[11030]: E0724 08:52:44.764420 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: aae50ebfe6ac41f8d0dc1ad40636699bee4580fe3b3bda1d98c1565da66e94d0" podSandboxID="aae50ebfe6ac41f8d0dc1ad40636699bee4580fe3b3bda1d98c1565da66e94d0" pod="kube-system/openebs-zfs-node-bf7kk" Jul 24 08:52:44 bluth k3s[11030]: E0724 08:52:44.765981 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: aae50ebfe6ac41f8d0dc1ad40636699bee4580fe3b3bda1d98c1565da66e94d0" podSandboxID="aae50ebfe6ac41f8d0dc1ad40636699bee4580fe3b3bda1d98c1565da66e94d0" pod="kube-system/openebs-zfs-node-bf7kk" Jul 24 08:52:45 bluth k3s[11030]: E0724 08:52:45.747037 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:52:45 bluth k3s[11030]: E0724 08:52:45.782847 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: aae50ebfe6ac41f8d0dc1ad40636699bee4580fe3b3bda1d98c1565da66e94d0" podSandboxID="aae50ebfe6ac41f8d0dc1ad40636699bee4580fe3b3bda1d98c1565da66e94d0" pod="kube-system/openebs-zfs-node-bf7kk" Jul 24 08:52:45 bluth k3s[11030]: E0724 08:52:45.784120 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: aae50ebfe6ac41f8d0dc1ad40636699bee4580fe3b3bda1d98c1565da66e94d0" podSandboxID="aae50ebfe6ac41f8d0dc1ad40636699bee4580fe3b3bda1d98c1565da66e94d0" pod="kube-system/openebs-zfs-node-bf7kk" Jul 24 08:52:46 bluth k3s[11030]: E0724 08:52:46.794700 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: aae50ebfe6ac41f8d0dc1ad40636699bee4580fe3b3bda1d98c1565da66e94d0" podSandboxID="aae50ebfe6ac41f8d0dc1ad40636699bee4580fe3b3bda1d98c1565da66e94d0" pod="kube-system/openebs-zfs-node-bf7kk" Jul 24 08:52:46 bluth k3s[11030]: E0724 08:52:46.796436 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: aae50ebfe6ac41f8d0dc1ad40636699bee4580fe3b3bda1d98c1565da66e94d0" podSandboxID="aae50ebfe6ac41f8d0dc1ad40636699bee4580fe3b3bda1d98c1565da66e94d0" pod="kube-system/openebs-zfs-node-bf7kk" Jul 24 08:52:47 bluth k3s[11030]: I0724 08:52:47.566650 11030 scope.go:110] "RemoveContainer" containerID="51ae3a4c735e322dac9d2fdb80a850a3d38ba3375fb16e1609d34c9c5ae191a2" Jul 24 08:52:47 bluth k3s[11030]: E0724 08:52:47.567029 11030 kuberuntime_gc.go:146] "Failed to remove container" err="failed to get container status \"51ae3a4c735e322dac9d2fdb80a850a3d38ba3375fb16e1609d34c9c5ae191a2\": rpc error: code = Unknown desc = Error: No such container: 51ae3a4c735e322dac9d2fdb80a850a3d38ba3375fb16e1609d34c9c5ae191a2" containerID="51ae3a4c735e322dac9d2fdb80a850a3d38ba3375fb16e1609d34c9c5ae191a2" Jul 24 08:52:47 bluth k3s[11030]: I0724 08:52:47.567045 11030 scope.go:110] "RemoveContainer" containerID="9f74ec129cf06d3c31eca044f56ab1d5213ec9cb7ca364b4f672a59039447ea5" Jul 24 08:52:47 bluth k3s[11030]: E0724 08:52:47.747983 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:52:47 bluth k3s[11030]: E0724 08:52:47.806122 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: aae50ebfe6ac41f8d0dc1ad40636699bee4580fe3b3bda1d98c1565da66e94d0" podSandboxID="aae50ebfe6ac41f8d0dc1ad40636699bee4580fe3b3bda1d98c1565da66e94d0" pod="kube-system/openebs-zfs-node-bf7kk" Jul 24 08:52:47 bluth k3s[11030]: I0724 08:52:47.813298 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="9a9d2fbb90507d877353427182ae2a0e0147652b13e44a236d6531e7a9fde18b" Jul 24 08:52:47 bluth k3s[11030]: E0724 08:52:47.814468 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: aae50ebfe6ac41f8d0dc1ad40636699bee4580fe3b3bda1d98c1565da66e94d0" podSandboxID="aae50ebfe6ac41f8d0dc1ad40636699bee4580fe3b3bda1d98c1565da66e94d0" pod="kube-system/openebs-zfs-node-bf7kk" Jul 24 08:52:48 bluth k3s[11030]: E0724 08:52:48.825916 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: aae50ebfe6ac41f8d0dc1ad40636699bee4580fe3b3bda1d98c1565da66e94d0" podSandboxID="aae50ebfe6ac41f8d0dc1ad40636699bee4580fe3b3bda1d98c1565da66e94d0" pod="kube-system/openebs-zfs-node-bf7kk" Jul 24 08:52:48 bluth k3s[11030]: E0724 08:52:48.827141 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: aae50ebfe6ac41f8d0dc1ad40636699bee4580fe3b3bda1d98c1565da66e94d0" podSandboxID="aae50ebfe6ac41f8d0dc1ad40636699bee4580fe3b3bda1d98c1565da66e94d0" pod="kube-system/openebs-zfs-node-bf7kk" Jul 24 08:52:49 bluth k3s[11030]: E0724 08:52:49.746489 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:52:49 bluth k3s[11030]: E0724 08:52:49.835971 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: aae50ebfe6ac41f8d0dc1ad40636699bee4580fe3b3bda1d98c1565da66e94d0" podSandboxID="aae50ebfe6ac41f8d0dc1ad40636699bee4580fe3b3bda1d98c1565da66e94d0" pod="kube-system/openebs-zfs-node-bf7kk" Jul 24 08:52:49 bluth k3s[11030]: E0724 08:52:49.837127 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: aae50ebfe6ac41f8d0dc1ad40636699bee4580fe3b3bda1d98c1565da66e94d0" podSandboxID="aae50ebfe6ac41f8d0dc1ad40636699bee4580fe3b3bda1d98c1565da66e94d0" pod="kube-system/openebs-zfs-node-bf7kk" Jul 24 08:52:50 bluth k3s[11030]: E0724 08:52:50.846163 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: aae50ebfe6ac41f8d0dc1ad40636699bee4580fe3b3bda1d98c1565da66e94d0" podSandboxID="aae50ebfe6ac41f8d0dc1ad40636699bee4580fe3b3bda1d98c1565da66e94d0" pod="kube-system/openebs-zfs-node-bf7kk" Jul 24 08:52:50 bluth k3s[11030]: E0724 08:52:50.847349 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: aae50ebfe6ac41f8d0dc1ad40636699bee4580fe3b3bda1d98c1565da66e94d0" podSandboxID="aae50ebfe6ac41f8d0dc1ad40636699bee4580fe3b3bda1d98c1565da66e94d0" pod="kube-system/openebs-zfs-node-bf7kk" Jul 24 08:52:51 bluth k3s[11030]: E0724 08:52:51.748054 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:52:51 bluth k3s[11030]: E0724 08:52:51.857548 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: aae50ebfe6ac41f8d0dc1ad40636699bee4580fe3b3bda1d98c1565da66e94d0" podSandboxID="aae50ebfe6ac41f8d0dc1ad40636699bee4580fe3b3bda1d98c1565da66e94d0" pod="kube-system/openebs-zfs-node-bf7kk" Jul 24 08:52:51 bluth k3s[11030]: E0724 08:52:51.859259 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: aae50ebfe6ac41f8d0dc1ad40636699bee4580fe3b3bda1d98c1565da66e94d0" podSandboxID="aae50ebfe6ac41f8d0dc1ad40636699bee4580fe3b3bda1d98c1565da66e94d0" pod="kube-system/openebs-zfs-node-bf7kk" Jul 24 08:52:52 bluth k3s[11030]: E0724 08:52:52.869116 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 9bdebc0d642d1b3ba263c05b8d4e382425a87a0a2b1d16e5001ba725d5c864b8" podSandboxID="9bdebc0d642d1b3ba263c05b8d4e382425a87a0a2b1d16e5001ba725d5c864b8" pod="ix-qbittorrent/svclb-qbittorrent-ldv82" Jul 24 08:52:52 bluth k3s[11030]: E0724 08:52:52.874251 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 7a536c8b85a08aaead7cab2226eefc0398735df6a967e3b9619a82292500c5ad" podSandboxID="7a536c8b85a08aaead7cab2226eefc0398735df6a967e3b9619a82292500c5ad" pod="ix-tdarr/svclb-tdarr-44c97" Jul 24 08:52:52 bluth k3s[11030]: E0724 08:52:52.876274 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: aae50ebfe6ac41f8d0dc1ad40636699bee4580fe3b3bda1d98c1565da66e94d0" podSandboxID="aae50ebfe6ac41f8d0dc1ad40636699bee4580fe3b3bda1d98c1565da66e94d0" pod="kube-system/openebs-zfs-node-bf7kk" Jul 24 08:52:52 bluth k3s[11030]: E0724 08:52:52.878141 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: aae50ebfe6ac41f8d0dc1ad40636699bee4580fe3b3bda1d98c1565da66e94d0" podSandboxID="aae50ebfe6ac41f8d0dc1ad40636699bee4580fe3b3bda1d98c1565da66e94d0" pod="kube-system/openebs-zfs-node-bf7kk" Jul 24 08:52:53 bluth k3s[11030]: E0724 08:52:53.746657 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:52:53 bluth k3s[11030]: E0724 08:52:53.896651 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 9bdebc0d642d1b3ba263c05b8d4e382425a87a0a2b1d16e5001ba725d5c864b8" podSandboxID="9bdebc0d642d1b3ba263c05b8d4e382425a87a0a2b1d16e5001ba725d5c864b8" pod="ix-qbittorrent/svclb-qbittorrent-ldv82" Jul 24 08:52:53 bluth k3s[11030]: E0724 08:52:53.898777 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 7a536c8b85a08aaead7cab2226eefc0398735df6a967e3b9619a82292500c5ad" podSandboxID="7a536c8b85a08aaead7cab2226eefc0398735df6a967e3b9619a82292500c5ad" pod="ix-tdarr/svclb-tdarr-44c97" Jul 24 08:52:54 bluth k3s[11030]: I0724 08:52:54.676830 11030 trace.go:205] Trace[194507424]: "Create" url:/apis/events.k8s.io/v1/namespaces/ix-home-assistant/events,user-agent:k3s/v1.23.5+k3s (linux/amd64) kubernetes/fbfa51e/scheduler,audit-id:defc5038-41c8-4f9a-86d8-656291b121d5,client:127.0.0.1,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (24-Jul-2022 08:52:53.889) (total time: 787ms): Jul 24 08:52:54 bluth k3s[11030]: Trace[194507424]: ---"Object stored in database" 787ms (08:52:54.676) Jul 24 08:52:54 bluth k3s[11030]: Trace[194507424]: [787.494941ms] [787.494941ms] END Jul 24 08:52:54 bluth k3s[11030]: I0724 08:52:54.752226 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="aae50ebfe6ac41f8d0dc1ad40636699bee4580fe3b3bda1d98c1565da66e94d0" Jul 24 08:52:54 bluth k3s[11030]: E0724 08:52:54.754209 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 9bdebc0d642d1b3ba263c05b8d4e382425a87a0a2b1d16e5001ba725d5c864b8" podSandboxID="9bdebc0d642d1b3ba263c05b8d4e382425a87a0a2b1d16e5001ba725d5c864b8" pod="ix-qbittorrent/svclb-qbittorrent-ldv82" Jul 24 08:52:54 bluth k3s[11030]: E0724 08:52:54.755632 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 7a536c8b85a08aaead7cab2226eefc0398735df6a967e3b9619a82292500c5ad" podSandboxID="7a536c8b85a08aaead7cab2226eefc0398735df6a967e3b9619a82292500c5ad" pod="ix-tdarr/svclb-tdarr-44c97" Jul 24 08:52:55 bluth k3s[11030]: I0724 08:52:55.057512 11030 scope.go:110] "RemoveContainer" containerID="097557fceb4d955ca22ba5e8c9129450fda31f9a02a736c640feff3f7f75d076" Jul 24 08:52:55 bluth k3s[11030]: E0724 08:52:55.058214 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 097557fceb4d955ca22ba5e8c9129450fda31f9a02a736c640feff3f7f75d076" containerID="097557fceb4d955ca22ba5e8c9129450fda31f9a02a736c640feff3f7f75d076" Jul 24 08:52:55 bluth k3s[11030]: I0724 08:52:55.058238 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:097557fceb4d955ca22ba5e8c9129450fda31f9a02a736c640feff3f7f75d076} err="failed to get container status \"097557fceb4d955ca22ba5e8c9129450fda31f9a02a736c640feff3f7f75d076\": rpc error: code = Unknown desc = Error: No such container: 097557fceb4d955ca22ba5e8c9129450fda31f9a02a736c640feff3f7f75d076" Jul 24 08:52:55 bluth k3s[11030]: I0724 08:52:55.058246 11030 scope.go:110] "RemoveContainer" containerID="3792d5854b3664bf23548595384863a12958ee8842cfca99582b3f61ff8be7a4" Jul 24 08:52:55 bluth k3s[11030]: E0724 08:52:55.058637 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 3792d5854b3664bf23548595384863a12958ee8842cfca99582b3f61ff8be7a4" containerID="3792d5854b3664bf23548595384863a12958ee8842cfca99582b3f61ff8be7a4" Jul 24 08:52:55 bluth k3s[11030]: I0724 08:52:55.058655 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:3792d5854b3664bf23548595384863a12958ee8842cfca99582b3f61ff8be7a4} err="failed to get container status \"3792d5854b3664bf23548595384863a12958ee8842cfca99582b3f61ff8be7a4\": rpc error: code = Unknown desc = Error: No such container: 3792d5854b3664bf23548595384863a12958ee8842cfca99582b3f61ff8be7a4" Jul 24 08:52:55 bluth k3s[11030]: I0724 08:52:55.058663 11030 scope.go:110] "RemoveContainer" containerID="3792d5854b3664bf23548595384863a12958ee8842cfca99582b3f61ff8be7a4" Jul 24 08:52:55 bluth k3s[11030]: I0724 08:52:55.059000 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:3792d5854b3664bf23548595384863a12958ee8842cfca99582b3f61ff8be7a4} err="failed to get container status \"3792d5854b3664bf23548595384863a12958ee8842cfca99582b3f61ff8be7a4\": rpc error: code = Unknown desc = Error: No such container: 3792d5854b3664bf23548595384863a12958ee8842cfca99582b3f61ff8be7a4" Jul 24 08:52:55 bluth k3s[11030]: I0724 08:52:55.059009 11030 scope.go:110] "RemoveContainer" containerID="e6756b9a88de99503cb95d24440adc48d681a672619c20f7ead1ed49f6a0c23f" Jul 24 08:52:55 bluth k3s[11030]: I0724 08:52:55.728781 11030 scope.go:110] "RemoveContainer" containerID="097557fceb4d955ca22ba5e8c9129450fda31f9a02a736c640feff3f7f75d076" Jul 24 08:52:55 bluth k3s[11030]: E0724 08:52:55.729429 11030 kuberuntime_gc.go:146] "Failed to remove container" err="failed to get container status \"097557fceb4d955ca22ba5e8c9129450fda31f9a02a736c640feff3f7f75d076\": rpc error: code = Unknown desc = Error: No such container: 097557fceb4d955ca22ba5e8c9129450fda31f9a02a736c640feff3f7f75d076" containerID="097557fceb4d955ca22ba5e8c9129450fda31f9a02a736c640feff3f7f75d076" Jul 24 08:52:55 bluth k3s[11030]: I0724 08:52:55.729486 11030 scope.go:110] "RemoveContainer" containerID="0cc34af722b67d4e3567a59268d634aa0b99635c2e9bbaaa83c0ee28b84d7458" Jul 24 08:52:55 bluth k3s[11030]: E0724 08:52:55.731727 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 0cc34af722b67d4e3567a59268d634aa0b99635c2e9bbaaa83c0ee28b84d7458" containerID="0cc34af722b67d4e3567a59268d634aa0b99635c2e9bbaaa83c0ee28b84d7458" Jul 24 08:52:55 bluth k3s[11030]: E0724 08:52:55.732068 11030 kuberuntime_gc.go:146] "Failed to remove container" err="failed to get container status \"0cc34af722b67d4e3567a59268d634aa0b99635c2e9bbaaa83c0ee28b84d7458\": rpc error: code = Unknown desc = Error: No such container: 0cc34af722b67d4e3567a59268d634aa0b99635c2e9bbaaa83c0ee28b84d7458" containerID="0cc34af722b67d4e3567a59268d634aa0b99635c2e9bbaaa83c0ee28b84d7458" Jul 24 08:52:55 bluth k3s[11030]: I0724 08:52:55.732088 11030 scope.go:110] "RemoveContainer" containerID="42f6521e6785b9192e442c8f8b5075d1a19fe2ff86adb83a34c2ee72c820662f" Jul 24 08:52:55 bluth k3s[11030]: E0724 08:52:55.732405 11030 kuberuntime_gc.go:146] "Failed to remove container" err="failed to get container status \"42f6521e6785b9192e442c8f8b5075d1a19fe2ff86adb83a34c2ee72c820662f\": rpc error: code = Unknown desc = Error: No such container: 42f6521e6785b9192e442c8f8b5075d1a19fe2ff86adb83a34c2ee72c820662f" containerID="42f6521e6785b9192e442c8f8b5075d1a19fe2ff86adb83a34c2ee72c820662f" Jul 24 08:52:55 bluth k3s[11030]: I0724 08:52:55.732421 11030 scope.go:110] "RemoveContainer" containerID="54cfb20a79b2d5e5f6eaf64dadabfffd4916dde065da6801479b5a9956b68217" Jul 24 08:52:55 bluth k3s[11030]: E0724 08:52:55.732834 11030 kuberuntime_gc.go:146] "Failed to remove container" err="failed to get container status \"54cfb20a79b2d5e5f6eaf64dadabfffd4916dde065da6801479b5a9956b68217\": rpc error: code = Unknown desc = Error: No such container: 54cfb20a79b2d5e5f6eaf64dadabfffd4916dde065da6801479b5a9956b68217" containerID="54cfb20a79b2d5e5f6eaf64dadabfffd4916dde065da6801479b5a9956b68217" Jul 24 08:52:55 bluth k3s[11030]: I0724 08:52:55.732847 11030 scope.go:110] "RemoveContainer" containerID="1fecb9d93d8a8acaf3ee33d5adb5a9643f0d231113609138fb2484ddb8c4f277" Jul 24 08:52:55 bluth k3s[11030]: E0724 08:52:55.757786 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:52:55 bluth k3s[11030]: I0724 08:52:55.881574 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="9bdebc0d642d1b3ba263c05b8d4e382425a87a0a2b1d16e5001ba725d5c864b8" Jul 24 08:52:55 bluth k3s[11030]: E0724 08:52:55.886568 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 10dc1d1b176ee3cc008b0fdf620308677602d106d63e61949a95b7b618f38eae" podSandboxID="10dc1d1b176ee3cc008b0fdf620308677602d106d63e61949a95b7b618f38eae" pod="ix-plex/plex-bd956645c-95gj8" Jul 24 08:52:57 bluth k3s[11030]: E0724 08:52:57.747833 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:52:58 bluth k3s[11030]: I0724 08:52:58.740891 11030 trace.go:205] Trace[475703295]: "GuaranteedUpdate etcd3" type:*coordination.Lease (24-Jul-2022 08:52:57.340) (total time: 1400ms): Jul 24 08:52:58 bluth k3s[11030]: Trace[475703295]: ---"Transaction committed" 1400ms (08:52:58.740) Jul 24 08:52:58 bluth k3s[11030]: Trace[475703295]: [1.400778713s] [1.400778713s] END Jul 24 08:52:58 bluth k3s[11030]: I0724 08:52:58.741155 11030 trace.go:205] Trace[1561881123]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ix-truenas,user-agent:Go-http-client/2.0,audit-id:74e90095-7123-4f2b-aad3-4b2f347774b1,client:127.0.0.1,accept:application/vnd.kubernetes.protobuf,application/json,protocol:HTTP/2.0 (24-Jul-2022 08:52:57.340) (total time: 1401ms): Jul 24 08:52:58 bluth k3s[11030]: Trace[1561881123]: ---"Object stored in database" 1401ms (08:52:58.741) Jul 24 08:52:58 bluth k3s[11030]: Trace[1561881123]: [1.401103582s] [1.401103582s] END Jul 24 08:52:59 bluth k3s[11030]: I0724 08:52:59.525621 18335 event.go:278] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ix-qbittorrent", Name:"svclb-qbittorrent-ldv82", UID:"ad6ed51e-d3e1-430c-9bad-528f4ae492c5", APIVersion:"v1", ResourceVersion:"3027034", FieldPath:""}): type: 'Normal' reason: 'AddedInterface' Add eth0 [172.16.0.182/16] from ix-net Jul 24 08:52:59 bluth k3s[11030]: E0724 08:52:59.748472 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:53:00 bluth k3s[11030]: I0724 08:53:00.391960 11030 scope.go:110] "RemoveContainer" containerID="e6756b9a88de99503cb95d24440adc48d681a672619c20f7ead1ed49f6a0c23f" Jul 24 08:53:00 bluth k3s[11030]: E0724 08:53:00.392290 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: e6756b9a88de99503cb95d24440adc48d681a672619c20f7ead1ed49f6a0c23f" containerID="e6756b9a88de99503cb95d24440adc48d681a672619c20f7ead1ed49f6a0c23f" Jul 24 08:53:00 bluth k3s[11030]: I0724 08:53:00.392315 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:e6756b9a88de99503cb95d24440adc48d681a672619c20f7ead1ed49f6a0c23f} err="failed to get container status \"e6756b9a88de99503cb95d24440adc48d681a672619c20f7ead1ed49f6a0c23f\": rpc error: code = Unknown desc = Error: No such container: e6756b9a88de99503cb95d24440adc48d681a672619c20f7ead1ed49f6a0c23f" Jul 24 08:53:00 bluth k3s[11030]: I0724 08:53:00.392322 11030 scope.go:110] "RemoveContainer" containerID="83de9bb2e016582dcd1febf8fe41eaaf734e62ad8e27a5befa1c8287cd8cc1a2" Jul 24 08:53:00 bluth k3s[11030]: E0724 08:53:00.392849 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 83de9bb2e016582dcd1febf8fe41eaaf734e62ad8e27a5befa1c8287cd8cc1a2" containerID="83de9bb2e016582dcd1febf8fe41eaaf734e62ad8e27a5befa1c8287cd8cc1a2" Jul 24 08:53:00 bluth k3s[11030]: I0724 08:53:00.392874 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:83de9bb2e016582dcd1febf8fe41eaaf734e62ad8e27a5befa1c8287cd8cc1a2} err="failed to get container status \"83de9bb2e016582dcd1febf8fe41eaaf734e62ad8e27a5befa1c8287cd8cc1a2\": rpc error: code = Unknown desc = Error: No such container: 83de9bb2e016582dcd1febf8fe41eaaf734e62ad8e27a5befa1c8287cd8cc1a2" Jul 24 08:53:00 bluth k3s[11030]: I0724 08:53:00.392930 11030 scope.go:110] "RemoveContainer" containerID="83de9bb2e016582dcd1febf8fe41eaaf734e62ad8e27a5befa1c8287cd8cc1a2" Jul 24 08:53:00 bluth k3s[11030]: I0724 08:53:00.394374 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:83de9bb2e016582dcd1febf8fe41eaaf734e62ad8e27a5befa1c8287cd8cc1a2} err="failed to get container status \"83de9bb2e016582dcd1febf8fe41eaaf734e62ad8e27a5befa1c8287cd8cc1a2\": rpc error: code = Unknown desc = Error: No such container: 83de9bb2e016582dcd1febf8fe41eaaf734e62ad8e27a5befa1c8287cd8cc1a2" Jul 24 08:53:00 bluth k3s[11030]: I0724 08:53:00.394387 11030 scope.go:110] "RemoveContainer" containerID="1fecb9d93d8a8acaf3ee33d5adb5a9643f0d231113609138fb2484ddb8c4f277" Jul 24 08:53:00 bluth k3s[11030]: E0724 08:53:00.396220 11030 remote_runtime.go:505] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to remove container \"1fecb9d93d8a8acaf3ee33d5adb5a9643f0d231113609138fb2484ddb8c4f277\": Error response from daemon: removal of container 1fecb9d93d8a8acaf3ee33d5adb5a9643f0d231113609138fb2484ddb8c4f277 is already in progress" containerID="1fecb9d93d8a8acaf3ee33d5adb5a9643f0d231113609138fb2484ddb8c4f277" Jul 24 08:53:00 bluth k3s[11030]: I0724 08:53:00.396245 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:1fecb9d93d8a8acaf3ee33d5adb5a9643f0d231113609138fb2484ddb8c4f277} err="rpc error: code = Unknown desc = failed to remove container \"1fecb9d93d8a8acaf3ee33d5adb5a9643f0d231113609138fb2484ddb8c4f277\": Error response from daemon: removal of container 1fecb9d93d8a8acaf3ee33d5adb5a9643f0d231113609138fb2484ddb8c4f277 is already in progress" Jul 24 08:53:00 bluth k3s[11030]: I0724 08:53:00.396254 11030 scope.go:110] "RemoveContainer" containerID="1fecb9d93d8a8acaf3ee33d5adb5a9643f0d231113609138fb2484ddb8c4f277" Jul 24 08:53:00 bluth k3s[11030]: E0724 08:53:00.397859 11030 remote_runtime.go:505] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to remove container \"1fecb9d93d8a8acaf3ee33d5adb5a9643f0d231113609138fb2484ddb8c4f277\": Error response from daemon: removal of container 1fecb9d93d8a8acaf3ee33d5adb5a9643f0d231113609138fb2484ddb8c4f277 is already in progress" containerID="1fecb9d93d8a8acaf3ee33d5adb5a9643f0d231113609138fb2484ddb8c4f277" Jul 24 08:53:00 bluth k3s[11030]: I0724 08:53:00.397879 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:1fecb9d93d8a8acaf3ee33d5adb5a9643f0d231113609138fb2484ddb8c4f277} err="rpc error: code = Unknown desc = failed to remove container \"1fecb9d93d8a8acaf3ee33d5adb5a9643f0d231113609138fb2484ddb8c4f277\": Error response from daemon: removal of container 1fecb9d93d8a8acaf3ee33d5adb5a9643f0d231113609138fb2484ddb8c4f277 is already in progress" Jul 24 08:53:00 bluth k3s[11030]: I0724 08:53:00.397888 11030 scope.go:110] "RemoveContainer" containerID="ac2bbe356bc4ebc0e814ead5646818b8a15a87346922ef289e6a333e2a010247" Jul 24 08:53:00 bluth k3s[11030]: I0724 08:53:00.897730 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="7a536c8b85a08aaead7cab2226eefc0398735df6a967e3b9619a82292500c5ad" Jul 24 08:53:00 bluth k3s[11030]: I0724 08:53:00.909543 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="9f74ec129cf06d3c31eca044f56ab1d5213ec9cb7ca364b4f672a59039447ea5" Jul 24 08:53:01 bluth k3s[11030]: I0724 08:53:01.074728 18541 event.go:278] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ix-tdarr", Name:"svclb-tdarr-44c97", UID:"e0e2f67d-a94d-4a52-a604-fb75e8d30555", APIVersion:"v1", ResourceVersion:"3027409", FieldPath:""}): type: 'Normal' reason: 'AddedInterface' Add eth0 [172.16.0.183/16] from ix-net Jul 24 08:53:01 bluth k3s[11030]: I0724 08:53:01.190470 11030 scope.go:110] "RemoveContainer" containerID="ac2bbe356bc4ebc0e814ead5646818b8a15a87346922ef289e6a333e2a010247" Jul 24 08:53:01 bluth k3s[11030]: E0724 08:53:01.194601 11030 remote_runtime.go:505] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to remove container \"ac2bbe356bc4ebc0e814ead5646818b8a15a87346922ef289e6a333e2a010247\": Error response from daemon: removal of container ac2bbe356bc4ebc0e814ead5646818b8a15a87346922ef289e6a333e2a010247 is already in progress" containerID="ac2bbe356bc4ebc0e814ead5646818b8a15a87346922ef289e6a333e2a010247" Jul 24 08:53:01 bluth k3s[11030]: E0724 08:53:01.194625 11030 kuberuntime_gc.go:146] "Failed to remove container" err="rpc error: code = Unknown desc = failed to remove container \"ac2bbe356bc4ebc0e814ead5646818b8a15a87346922ef289e6a333e2a010247\": Error response from daemon: removal of container ac2bbe356bc4ebc0e814ead5646818b8a15a87346922ef289e6a333e2a010247 is already in progress" containerID="ac2bbe356bc4ebc0e814ead5646818b8a15a87346922ef289e6a333e2a010247" Jul 24 08:53:01 bluth k3s[11030]: I0724 08:53:01.194639 11030 scope.go:110] "RemoveContainer" containerID="e6756b9a88de99503cb95d24440adc48d681a672619c20f7ead1ed49f6a0c23f" Jul 24 08:53:01 bluth k3s[11030]: E0724 08:53:01.194970 11030 kuberuntime_gc.go:146] "Failed to remove container" err="failed to get container status \"e6756b9a88de99503cb95d24440adc48d681a672619c20f7ead1ed49f6a0c23f\": rpc error: code = Unknown desc = Error: No such container: e6756b9a88de99503cb95d24440adc48d681a672619c20f7ead1ed49f6a0c23f" containerID="e6756b9a88de99503cb95d24440adc48d681a672619c20f7ead1ed49f6a0c23f" Jul 24 08:53:01 bluth k3s[11030]: I0724 08:53:01.194982 11030 scope.go:110] "RemoveContainer" containerID="65057fd24b1abf38cfbeb68bf18ae59e155c7a3e9e3bb74aa941dd0e954dc64d" Jul 24 08:53:01 bluth k3s[11030]: E0724 08:53:01.749725 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:53:03 bluth k3s[11030]: E0724 08:53:03.752707 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:53:05 bluth k3s[11030]: E0724 08:53:05.753813 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:53:07 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T08:53:07.541+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 08:53:07 bluth k3s[11030]: E0724 08:53:07.748415 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:53:08 bluth k3s[11030]: I0724 08:53:08.019046 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="c77fab160a7d7e06fb590ab5355713015b16995bc4dae5cf05265c327f8b16c0" Jul 24 08:53:08 bluth k3s[11030]: I0724 08:53:08.032663 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="097557fceb4d955ca22ba5e8c9129450fda31f9a02a736c640feff3f7f75d076" Jul 24 08:53:08 bluth k3s[11030]: I0724 08:53:08.152700 18994 event.go:278] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ix-qbittorrent", Name:"svclb-qbittorrent-torrentudp-62ksx", UID:"c69bd8a5-5c5f-4a26-9204-b7d4036137ec", APIVersion:"v1", ResourceVersion:"3027384", FieldPath:""}): type: 'Normal' reason: 'AddedInterface' Add eth0 [172.16.0.184/16] from ix-net Jul 24 08:53:09 bluth k3s[11030]: E0724 08:53:09.749939 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:53:11 bluth k3s[11030]: E0724 08:53:11.747058 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:53:13 bluth k3s[11030]: E0724 08:53:13.754120 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:53:15 bluth k3s[11030]: I0724 08:53:15.015254 11030 scope.go:110] "RemoveContainer" containerID="ac2bbe356bc4ebc0e814ead5646818b8a15a87346922ef289e6a333e2a010247" Jul 24 08:53:15 bluth k3s[11030]: E0724 08:53:15.016047 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: ac2bbe356bc4ebc0e814ead5646818b8a15a87346922ef289e6a333e2a010247" containerID="ac2bbe356bc4ebc0e814ead5646818b8a15a87346922ef289e6a333e2a010247" Jul 24 08:53:15 bluth k3s[11030]: I0724 08:53:15.016159 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:ac2bbe356bc4ebc0e814ead5646818b8a15a87346922ef289e6a333e2a010247} err="failed to get container status \"ac2bbe356bc4ebc0e814ead5646818b8a15a87346922ef289e6a333e2a010247\": rpc error: code = Unknown desc = Error: No such container: ac2bbe356bc4ebc0e814ead5646818b8a15a87346922ef289e6a333e2a010247" Jul 24 08:53:15 bluth k3s[11030]: E0724 08:53:15.749300 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:53:17 bluth k3s[11030]: E0724 08:53:17.747047 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:53:19 bluth k3s[11030]: E0724 08:53:19.748900 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:53:21 bluth k3s[11030]: I0724 08:53:20.996352 19408 event.go:278] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ix-prowlarr", Name:"svclb-prowlarr-qhcm8", UID:"01a5e89a-9794-4261-ada8-82a5e21ccb7d", APIVersion:"v1", ResourceVersion:"3027380", FieldPath:""}): type: 'Normal' reason: 'AddedInterface' Add eth0 [172.16.0.185/16] from ix-net Jul 24 08:53:21 bluth k3s[11030]: I0724 08:53:21.055584 11030 scope.go:110] "RemoveContainer" containerID="1040ba7a6c088f0e6e080c250bce5967e0b2b718b53597d05a7fa6694324b89e" Jul 24 08:53:21 bluth k3s[11030]: E0724 08:53:21.747267 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:53:22 bluth k3s[11030]: I0724 08:53:22.431107 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="72a1d931091ebdebf3039ade3407f888aa967bfc02ef84c7e89a89ffb446a752" Jul 24 08:53:22 bluth k3s[11030]: E0724 08:53:22.463053 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: d22b452223417bc68f543d64e8bddb9472cc308939f81ac3ac0b09cf66d0999c" podSandboxID="d22b452223417bc68f543d64e8bddb9472cc308939f81ac3ac0b09cf66d0999c" pod="kube-system/coredns-d76bd69b-gk2vd" Jul 24 08:53:22 bluth k3s[11030]: I0724 08:53:22.536309 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="39750834e270cdce5354710d29f1f460795b9ceb6ff3ee5c9b4e5cb99fae63f7" Jul 24 08:53:22 bluth k3s[11030]: I0724 08:53:22.569365 19761 event.go:278] Event(v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"coredns-d76bd69b-mhb5r", UID:"3f09401b-76fa-4bff-8736-00b8e6a1a41f", APIVersion:"v1", ResourceVersion:"3027397", FieldPath:""}): type: 'Normal' reason: 'AddedInterface' Add eth0 [172.16.0.186/16] from ix-net Jul 24 08:53:23 bluth k3s[11030]: I0724 08:53:23.504471 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="43346eb69a811b3b8704ffc9efa078c14e4e665448fdb758189006db86d835f5" Jul 24 08:53:23 bluth k3s[11030]: I0724 08:53:23.642837 19955 event.go:278] Event(v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"openebs-zfs-controller-0", UID:"508c7225-82e1-4311-9b86-6a6ad8bf3740", APIVersion:"v1", ResourceVersion:"3027402", FieldPath:""}): type: 'Normal' reason: 'AddedInterface' Add eth0 [172.16.0.187/16] from ix-net Jul 24 08:53:23 bluth k3s[11030]: E0724 08:53:23.749338 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:53:24 bluth k3s[11030]: I0724 08:53:24.403279 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="54b6be0657350c916b68f097bd0839c6c28898f15501f7e078b511f54fd216a0" Jul 24 08:53:24 bluth k3s[11030]: I0724 08:53:24.506810 20009 event.go:278] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ix-radarr", Name:"svclb-radarr-nwf7v", UID:"51776aeb-f372-4d10-aa50-df5855b638e3", APIVersion:"v1", ResourceVersion:"3027404", FieldPath:""}): type: 'Normal' reason: 'AddedInterface' Add eth0 [172.16.0.188/16] from ix-net Jul 24 08:53:24 bluth k3s[11030]: I0724 08:53:24.934641 11030 cni.go:334] "CNI failed to retrieve network namespace path" err="cannot find network namespace for the terminated container \"9bf06bda197b74b428990270afa6fb445d8fd616bcf780b7af3d4862178e01bf\"" Jul 24 08:53:25 bluth k3s[11030]: I0724 08:53:25.441088 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="65057fd24b1abf38cfbeb68bf18ae59e155c7a3e9e3bb74aa941dd0e954dc64d" Jul 24 08:53:25 bluth k3s[11030]: E0724 08:53:25.750535 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:53:26 bluth k3s[11030]: I0724 08:53:26.954354 20367 event.go:278] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ix-qbittorrent", Name:"svclb-qbittorrent-torrent-8qgf8", UID:"19ef4922-0714-452c-b8ba-e3c955cabe50", APIVersion:"v1", ResourceVersion:"3027296", FieldPath:""}): type: 'Normal' reason: 'AddedInterface' Add eth0 [172.16.0.189/16] from ix-net Jul 24 08:53:26 bluth k3s[11030]: E0724 08:53:26.969469 11030 writers.go:117] apiserver was unable to write a JSON response: http: Handler timeout Jul 24 08:53:26 bluth k3s[11030]: E0724 08:53:26.969490 11030 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Jul 24 08:53:26 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T08:53:26.969+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00145d180/kine.sock","attempt":0,"error":"rpc error: code = Canceled desc = context canceled"} Jul 24 08:53:26 bluth k3s[11030]: E0724 08:53:26.970215 11030 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 12.349µs, panicked: false, err: context canceled, panic-reason: Jul 24 08:53:26 bluth k3s[11030]: E0724 08:53:26.970521 11030 writers.go:130] apiserver was unable to write a fallback JSON response: http: Handler timeout Jul 24 08:53:26 bluth k3s[11030]: E0724 08:53:26.971766 11030 timeout.go:137] post-timeout activity - time-elapsed: 3.074143ms, POST "/api/v1/namespaces/ix-qbittorrent/events" result: Jul 24 08:53:27 bluth k3s[11030]: I0724 08:53:27.192113 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="c2d4fc8c9e5c3ae9ca20561d368f76fe7c332ea314ea6d9ee35fc2fbe08ecf6c" Jul 24 08:53:27 bluth k3s[11030]: I0724 08:53:27.296315 20459 event.go:278] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ix-sonarr", Name:"svclb-sonarr-69zgj", UID:"ecd248e1-4900-4e80-ba0b-7233f4b438b4", APIVersion:"v1", ResourceVersion:"3027372", FieldPath:""}): type: 'Normal' reason: 'AddedInterface' Add eth0 [172.16.0.190/16] from ix-net Jul 24 08:53:27 bluth k3s[11030]: E0724 08:53:27.753934 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:53:29 bluth k3s[11030]: E0724 08:53:29.748794 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:53:31 bluth k3s[11030]: E0724 08:53:31.749993 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:53:32 bluth k3s[11030]: E0724 08:53:32.932517 11030 remote_runtime.go:209] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create a sandbox for pod \"svclb-tdarr-comm-npvsl\": operation timeout: context deadline exceeded" Jul 24 08:53:32 bluth k3s[11030]: E0724 08:53:32.932708 11030 kuberuntime_sandbox.go:70] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create a sandbox for pod \"svclb-tdarr-comm-npvsl\": operation timeout: context deadline exceeded" pod="ix-tdarr/svclb-tdarr-comm-npvsl" Jul 24 08:53:33 bluth k3s[11030]: I0724 08:53:33.318882 20699 event.go:278] Event(v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"intel-gpu-plugin-fxs6q", UID:"bb5993ab-6276-43d2-8017-a6fc01efb9a3", APIVersion:"v1", ResourceVersion:"3027416", FieldPath:""}): type: 'Normal' reason: 'AddedInterface' Add eth0 [172.16.0.191/16] from ix-net Jul 24 08:53:33 bluth k3s[11030]: E0724 08:53:33.751352 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:53:34 bluth k3s[11030]: I0724 08:53:34.315504 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="e6756b9a88de99503cb95d24440adc48d681a672619c20f7ead1ed49f6a0c23f" Jul 24 08:53:34 bluth k3s[11030]: I0724 08:53:34.315827 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="d22b452223417bc68f543d64e8bddb9472cc308939f81ac3ac0b09cf66d0999c" Jul 24 08:53:34 bluth k3s[11030]: I0724 08:53:34.329811 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="9c6fc7a9dfe011ec94a8d8b5a144913db6d0ccd71ee1a1872df481afd30510f2" Jul 24 08:53:34 bluth k3s[11030]: I0724 08:53:34.331237 11030 docker_sandbox.go:402] "Failed to read pod IP from plugin/docker" err="networkPlugin cni failed on the status hook for pod \"tdarr-746d65b466-b7whb_ix-tdarr\": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container \"6e77c264156395d54133c0402bc6e1ad7bbef3dc70443cd759055f59551884ee\"" Jul 24 08:53:34 bluth k3s[11030]: I0724 08:53:34.334183 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="1040ba7a6c088f0e6e080c250bce5967e0b2b718b53597d05a7fa6694324b89e" Jul 24 08:53:34 bluth k3s[11030]: I0724 08:53:34.472425 20825 event.go:278] Event(v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"coredns-d76bd69b-gk2vd", UID:"b1f2714d-382d-4638-83f4-2b2055748db3", APIVersion:"v1", ResourceVersion:"3027350", FieldPath:""}): type: 'Normal' reason: 'AddedInterface' Add eth0 [172.16.0.192/16] from ix-net Jul 24 08:53:35 bluth k3s[11030]: I0724 08:53:35.287298 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="31da4fd205a212d77f17853ba8032143c9e16e19fd5de6fa671faa2f74206994" Jul 24 08:53:35 bluth k3s[11030]: I0724 08:53:35.348928 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="f23804a7cc71c0d0fa3aa82a90423e1351064a01029da11d397488a3c3ceaad7" Jul 24 08:53:35 bluth k3s[11030]: I0724 08:53:35.404314 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="ac2bbe356bc4ebc0e814ead5646818b8a15a87346922ef289e6a333e2a010247" Jul 24 08:53:35 bluth k3s[11030]: I0724 08:53:35.437397 21008 event.go:278] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ix-esphome", Name:"svclb-esphome-q4lrj", UID:"c41628dc-0533-406b-ac00-3fbf6cd22284", APIVersion:"v1", ResourceVersion:"3027376", FieldPath:""}): type: 'Normal' reason: 'AddedInterface' Add eth0 [172.16.0.193/16] from ix-net Jul 24 08:53:35 bluth k3s[11030]: E0724 08:53:35.752182 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"71936d7f-4447-4ac2-aab8-7586f615b53e\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2 Jul 24 08:53:35 bluth k3s[11030]: E0724 08:53:35.774112 11030 kubelet.go:1730] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[kube-api-access-xr6bj config platformio], unattached volumes=[varlogs kube-api-access-xr6bj config platformio shared temp]: timed out waiting for the condition" pod="ix-esphome/esphome-658957fddc-sb8c8" Jul 24 08:53:35 bluth k3s[11030]: E0724 08:53:35.774254 11030 pod_workers.go:949] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-xr6bj config platformio], unattached volumes=[varlogs kube-api-access-xr6bj config platformio shared temp]: timed out waiting for the condition" pod="ix-esphome/esphome-658957fddc-sb8c8" podUID=05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b Jul 24 08:53:35 bluth k3s[11030]: E0724 08:53:35.777500 11030 kubelet.go:1730] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[config], unattached volumes=[shared temp varlogs list-0 kube-api-access-2v58k config]: timed out waiting for the condition" pod="ix-sonarr/sonarr-867d7dd548-4bn8s" Jul 24 08:53:35 bluth k3s[11030]: E0724 08:53:35.777570 11030 pod_workers.go:949] "Error syncing pod, skipping" err="unmounted volumes=[config], unattached volumes=[shared temp varlogs list-0 kube-api-access-2v58k config]: timed out waiting for the condition" pod="ix-sonarr/sonarr-867d7dd548-4bn8s" podUID=35ac75bf-b85f-4286-a228-ea8459dde305 Jul 24 08:53:35 bluth k3s[11030]: E0724 08:53:35.804266 11030 kubelet.go:1730] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[config], unattached volumes=[list-0 kube-api-access-2qgd6 config shared temp varlogs]: timed out waiting for the condition" pod="ix-qbittorrent/qbittorrent-6d7cf9f7cd-7n6c6" Jul 24 08:53:35 bluth k3s[11030]: E0724 08:53:35.804288 11030 pod_workers.go:949] "Error syncing pod, skipping" err="unmounted volumes=[config], unattached volumes=[list-0 kube-api-access-2qgd6 config shared temp varlogs]: timed out waiting for the condition" pod="ix-qbittorrent/qbittorrent-6d7cf9f7cd-7n6c6" podUID=2fac2fcc-a503-4887-810c-c2ff11c07852 Jul 24 08:53:35 bluth k3s[11030]: E0724 08:53:35.805353 11030 kubelet.go:1730] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[transcode-cache server logs media configs], unattached volumes=[varrun shared transcode-cache kube-api-access-56dxr varlogs server logs media temp list-0 configs]: timed out waiting for the condition" pod="ix-tdarr/tdarr-746d65b466-b7whb" Jul 24 08:53:35 bluth k3s[11030]: E0724 08:53:35.805372 11030 pod_workers.go:949] "Error syncing pod, skipping" err="unmounted volumes=[transcode-cache server logs media configs], unattached volumes=[varrun shared transcode-cache kube-api-access-56dxr varlogs server logs media temp list-0 configs]: timed out waiting for the condition" pod="ix-tdarr/tdarr-746d65b466-b7whb" podUID=6e3e9791-9bb3-487f-ae50-048fefbef8be Jul 24 08:53:35 bluth k3s[11030]: E0724 08:53:35.838384 11030 kubelet.go:1730] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[config], unattached volumes=[list-0 kube-api-access-vjp24 config shared temp varlogs]: timed out waiting for the condition" pod="ix-radarr/radarr-7486f78d7f-q4pn4" Jul 24 08:53:35 bluth k3s[11030]: E0724 08:53:35.838418 11030 pod_workers.go:949] "Error syncing pod, skipping" err="unmounted volumes=[config], unattached volumes=[list-0 kube-api-access-vjp24 config shared temp varlogs]: timed out waiting for the condition" pod="ix-radarr/radarr-7486f78d7f-q4pn4" podUID=be152b2b-fd1c-4a36-9081-ac6be2fb7901 Jul 24 08:53:36 bluth k3s[11030]: E0724 08:53:36.421552 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: bd1d7b7eda1457ba032977be046038eb7b0e4de8b978957e22459377c7dfc249" podSandboxID="bd1d7b7eda1457ba032977be046038eb7b0e4de8b978957e22459377c7dfc249" pod="ix-tdarr/svclb-tdarr-comm-npvsl" Jul 24 08:53:36 bluth k3s[11030]: E0724 08:53:36.421595 11030 pod_workers.go:949] "Error syncing pod, skipping" err="rpc error: code = Unknown desc = Error: No such container: bd1d7b7eda1457ba032977be046038eb7b0e4de8b978957e22459377c7dfc249" pod="ix-tdarr/svclb-tdarr-comm-npvsl" podUID=69463f72-1255-4fe0-b5a2-59f859d48b92 Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.470594 11030 cni.go:334] "CNI failed to retrieve network namespace path" err="cannot find network namespace for the terminated container \"e9b0b4c761ef9178a6c584628e4bad3640e0c7e17646220cc5483ae1d2d9f139\"" Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.470818 11030 cni.go:334] "CNI failed to retrieve network namespace path" err="cannot find network namespace for the terminated container \"46351dd20a65bcd3401b2ce2feaf6f4de7de1533f2dcc58d7dc4d09d8b7aacad\"" Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.471243 11030 cni.go:334] "CNI failed to retrieve network namespace path" err="cannot find network namespace for the terminated container \"7c1dcc789affc3d314b965f028c17f94aba4566963ca77a10eab57d91b1a1972\"" Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.471456 11030 cni.go:334] "CNI failed to retrieve network namespace path" err="cannot find network namespace for the terminated container \"6e77c264156395d54133c0402bc6e1ad7bbef3dc70443cd759055f59551884ee\"" Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.471604 11030 cni.go:334] "CNI failed to retrieve network namespace path" err="cannot find network namespace for the terminated container \"788c819da6910895b7711adad84d6e1f1fdb4b436f2562f6c9b94ed9c7826957\"" Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.695525 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"shared\" (UniqueName: \"kubernetes.io/empty-dir/be152b2b-fd1c-4a36-9081-ac6be2fb7901-shared\") pod \"be152b2b-fd1c-4a36-9081-ac6be2fb7901\" (UID: \"be152b2b-fd1c-4a36-9081-ac6be2fb7901\") " Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.695558 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"shared\" (UniqueName: \"kubernetes.io/empty-dir/6e3e9791-9bb3-487f-ae50-048fefbef8be-shared\") pod \"6e3e9791-9bb3-487f-ae50-048fefbef8be\" (UID: \"6e3e9791-9bb3-487f-ae50-048fefbef8be\") " Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.695574 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"shared\" (UniqueName: \"kubernetes.io/empty-dir/2fac2fcc-a503-4887-810c-c2ff11c07852-shared\") pod \"2fac2fcc-a503-4887-810c-c2ff11c07852\" (UID: \"2fac2fcc-a503-4887-810c-c2ff11c07852\") " Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.695589 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"temp\" (UniqueName: \"kubernetes.io/empty-dir/05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b-temp\") pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" (UID: \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\") " Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.695606 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"varlogs\" (UniqueName: \"kubernetes.io/empty-dir/05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b-varlogs\") pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" (UID: \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\") " Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.695619 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"shared\" (UniqueName: \"kubernetes.io/empty-dir/35ac75bf-b85f-4286-a228-ea8459dde305-shared\") pod \"35ac75bf-b85f-4286-a228-ea8459dde305\" (UID: \"35ac75bf-b85f-4286-a228-ea8459dde305\") " Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.695633 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"temp\" (UniqueName: \"kubernetes.io/empty-dir/35ac75bf-b85f-4286-a228-ea8459dde305-temp\") pod \"35ac75bf-b85f-4286-a228-ea8459dde305\" (UID: \"35ac75bf-b85f-4286-a228-ea8459dde305\") " Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.695647 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"varrun\" (UniqueName: \"kubernetes.io/empty-dir/6e3e9791-9bb3-487f-ae50-048fefbef8be-varrun\") pod \"6e3e9791-9bb3-487f-ae50-048fefbef8be\" (UID: \"6e3e9791-9bb3-487f-ae50-048fefbef8be\") " Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.695662 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56dxr\" (UniqueName: \"kubernetes.io/projected/6e3e9791-9bb3-487f-ae50-048fefbef8be-kube-api-access-56dxr\") pod \"6e3e9791-9bb3-487f-ae50-048fefbef8be\" (UID: \"6e3e9791-9bb3-487f-ae50-048fefbef8be\") " Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.695677 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"varlogs\" (UniqueName: \"kubernetes.io/empty-dir/6e3e9791-9bb3-487f-ae50-048fefbef8be-varlogs\") pod \"6e3e9791-9bb3-487f-ae50-048fefbef8be\" (UID: \"6e3e9791-9bb3-487f-ae50-048fefbef8be\") " Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.695697 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2v58k\" (UniqueName: \"kubernetes.io/projected/35ac75bf-b85f-4286-a228-ea8459dde305-kube-api-access-2v58k\") pod \"35ac75bf-b85f-4286-a228-ea8459dde305\" (UID: \"35ac75bf-b85f-4286-a228-ea8459dde305\") " Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.695709 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"temp\" (UniqueName: \"kubernetes.io/empty-dir/6e3e9791-9bb3-487f-ae50-048fefbef8be-temp\") pod \"6e3e9791-9bb3-487f-ae50-048fefbef8be\" (UID: \"6e3e9791-9bb3-487f-ae50-048fefbef8be\") " Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.695722 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"varlogs\" (UniqueName: \"kubernetes.io/empty-dir/be152b2b-fd1c-4a36-9081-ac6be2fb7901-varlogs\") pod \"be152b2b-fd1c-4a36-9081-ac6be2fb7901\" (UID: \"be152b2b-fd1c-4a36-9081-ac6be2fb7901\") " Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.695736 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"temp\" (UniqueName: \"kubernetes.io/empty-dir/be152b2b-fd1c-4a36-9081-ac6be2fb7901-temp\") pod \"be152b2b-fd1c-4a36-9081-ac6be2fb7901\" (UID: \"be152b2b-fd1c-4a36-9081-ac6be2fb7901\") " Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.695749 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"list-0\" (UniqueName: \"kubernetes.io/host-path/6e3e9791-9bb3-487f-ae50-048fefbef8be-list-0\") pod \"6e3e9791-9bb3-487f-ae50-048fefbef8be\" (UID: \"6e3e9791-9bb3-487f-ae50-048fefbef8be\") " Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.695762 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"temp\" (UniqueName: \"kubernetes.io/empty-dir/2fac2fcc-a503-4887-810c-c2ff11c07852-temp\") pod \"2fac2fcc-a503-4887-810c-c2ff11c07852\" (UID: \"2fac2fcc-a503-4887-810c-c2ff11c07852\") " Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.695773 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"list-0\" (UniqueName: \"kubernetes.io/host-path/be152b2b-fd1c-4a36-9081-ac6be2fb7901-list-0\") pod \"be152b2b-fd1c-4a36-9081-ac6be2fb7901\" (UID: \"be152b2b-fd1c-4a36-9081-ac6be2fb7901\") " Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.695785 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"varlogs\" (UniqueName: \"kubernetes.io/empty-dir/2fac2fcc-a503-4887-810c-c2ff11c07852-varlogs\") pod \"2fac2fcc-a503-4887-810c-c2ff11c07852\" (UID: \"2fac2fcc-a503-4887-810c-c2ff11c07852\") " Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.695799 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjp24\" (UniqueName: \"kubernetes.io/projected/be152b2b-fd1c-4a36-9081-ac6be2fb7901-kube-api-access-vjp24\") pod \"be152b2b-fd1c-4a36-9081-ac6be2fb7901\" (UID: \"be152b2b-fd1c-4a36-9081-ac6be2fb7901\") " Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.695813 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"list-0\" (UniqueName: \"kubernetes.io/host-path/2fac2fcc-a503-4887-810c-c2ff11c07852-list-0\") pod \"2fac2fcc-a503-4887-810c-c2ff11c07852\" (UID: \"2fac2fcc-a503-4887-810c-c2ff11c07852\") " Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.695827 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"list-0\" (UniqueName: \"kubernetes.io/host-path/35ac75bf-b85f-4286-a228-ea8459dde305-list-0\") pod \"35ac75bf-b85f-4286-a228-ea8459dde305\" (UID: \"35ac75bf-b85f-4286-a228-ea8459dde305\") " Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.695840 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"shared\" (UniqueName: \"kubernetes.io/empty-dir/05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b-shared\") pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" (UID: \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\") " Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.695854 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qgd6\" (UniqueName: \"kubernetes.io/projected/2fac2fcc-a503-4887-810c-c2ff11c07852-kube-api-access-2qgd6\") pod \"2fac2fcc-a503-4887-810c-c2ff11c07852\" (UID: \"2fac2fcc-a503-4887-810c-c2ff11c07852\") " Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.695867 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"varlogs\" (UniqueName: \"kubernetes.io/empty-dir/35ac75bf-b85f-4286-a228-ea8459dde305-varlogs\") pod \"35ac75bf-b85f-4286-a228-ea8459dde305\" (UID: \"35ac75bf-b85f-4286-a228-ea8459dde305\") " Jul 24 08:53:36 bluth k3s[11030]: W0724 08:53:36.696329 11030 empty_dir.go:517] Warning: Failed to clear quota on /var/lib/kubelet/pods/35ac75bf-b85f-4286-a228-ea8459dde305/volumes/kubernetes.io~empty-dir/varlogs: clearQuota called, but quotas disabled Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.696483 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35ac75bf-b85f-4286-a228-ea8459dde305-varlogs" (OuterVolumeSpecName: "varlogs") pod "35ac75bf-b85f-4286-a228-ea8459dde305" (UID: "35ac75bf-b85f-4286-a228-ea8459dde305"). InnerVolumeSpecName "varlogs". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Jul 24 08:53:36 bluth k3s[11030]: W0724 08:53:36.696640 11030 empty_dir.go:517] Warning: Failed to clear quota on /var/lib/kubelet/pods/be152b2b-fd1c-4a36-9081-ac6be2fb7901/volumes/kubernetes.io~empty-dir/shared: clearQuota called, but quotas disabled Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.696694 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be152b2b-fd1c-4a36-9081-ac6be2fb7901-shared" (OuterVolumeSpecName: "shared") pod "be152b2b-fd1c-4a36-9081-ac6be2fb7901" (UID: "be152b2b-fd1c-4a36-9081-ac6be2fb7901"). InnerVolumeSpecName "shared". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Jul 24 08:53:36 bluth k3s[11030]: W0724 08:53:36.696838 11030 empty_dir.go:517] Warning: Failed to clear quota on /var/lib/kubelet/pods/6e3e9791-9bb3-487f-ae50-048fefbef8be/volumes/kubernetes.io~empty-dir/shared: clearQuota called, but quotas disabled Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.696889 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e3e9791-9bb3-487f-ae50-048fefbef8be-shared" (OuterVolumeSpecName: "shared") pod "6e3e9791-9bb3-487f-ae50-048fefbef8be" (UID: "6e3e9791-9bb3-487f-ae50-048fefbef8be"). InnerVolumeSpecName "shared". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Jul 24 08:53:36 bluth k3s[11030]: W0724 08:53:36.697023 11030 empty_dir.go:517] Warning: Failed to clear quota on /var/lib/kubelet/pods/2fac2fcc-a503-4887-810c-c2ff11c07852/volumes/kubernetes.io~empty-dir/shared: clearQuota called, but quotas disabled Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.697075 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fac2fcc-a503-4887-810c-c2ff11c07852-shared" (OuterVolumeSpecName: "shared") pod "2fac2fcc-a503-4887-810c-c2ff11c07852" (UID: "2fac2fcc-a503-4887-810c-c2ff11c07852"). InnerVolumeSpecName "shared". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Jul 24 08:53:36 bluth k3s[11030]: W0724 08:53:36.697208 11030 empty_dir.go:517] Warning: Failed to clear quota on /var/lib/kubelet/pods/05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b/volumes/kubernetes.io~empty-dir/temp: clearQuota called, but quotas disabled Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.697256 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b-temp" (OuterVolumeSpecName: "temp") pod "05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b" (UID: "05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b"). InnerVolumeSpecName "temp". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Jul 24 08:53:36 bluth k3s[11030]: W0724 08:53:36.697372 11030 empty_dir.go:517] Warning: Failed to clear quota on /var/lib/kubelet/pods/05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b/volumes/kubernetes.io~empty-dir/varlogs: clearQuota called, but quotas disabled Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.697409 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b-varlogs" (OuterVolumeSpecName: "varlogs") pod "05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b" (UID: "05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b"). InnerVolumeSpecName "varlogs". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Jul 24 08:53:36 bluth k3s[11030]: W0724 08:53:36.697525 11030 empty_dir.go:517] Warning: Failed to clear quota on /var/lib/kubelet/pods/35ac75bf-b85f-4286-a228-ea8459dde305/volumes/kubernetes.io~empty-dir/shared: clearQuota called, but quotas disabled Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.697563 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35ac75bf-b85f-4286-a228-ea8459dde305-shared" (OuterVolumeSpecName: "shared") pod "35ac75bf-b85f-4286-a228-ea8459dde305" (UID: "35ac75bf-b85f-4286-a228-ea8459dde305"). InnerVolumeSpecName "shared". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.697826 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/6e3e9791-9bb3-487f-ae50-048fefbef8be-list-0" (OuterVolumeSpecName: "list-0") pod "6e3e9791-9bb3-487f-ae50-048fefbef8be" (UID: "6e3e9791-9bb3-487f-ae50-048fefbef8be"). InnerVolumeSpecName "list-0". PluginName "kubernetes.io/host-path", VolumeGidValue "" Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.698107 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/35ac75bf-b85f-4286-a228-ea8459dde305-list-0" (OuterVolumeSpecName: "list-0") pod "35ac75bf-b85f-4286-a228-ea8459dde305" (UID: "35ac75bf-b85f-4286-a228-ea8459dde305"). InnerVolumeSpecName "list-0". PluginName "kubernetes.io/host-path", VolumeGidValue "" Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.698127 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/2fac2fcc-a503-4887-810c-c2ff11c07852-list-0" (OuterVolumeSpecName: "list-0") pod "2fac2fcc-a503-4887-810c-c2ff11c07852" (UID: "2fac2fcc-a503-4887-810c-c2ff11c07852"). InnerVolumeSpecName "list-0". PluginName "kubernetes.io/host-path", VolumeGidValue "" Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.698271 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/be152b2b-fd1c-4a36-9081-ac6be2fb7901-list-0" (OuterVolumeSpecName: "list-0") pod "be152b2b-fd1c-4a36-9081-ac6be2fb7901" (UID: "be152b2b-fd1c-4a36-9081-ac6be2fb7901"). InnerVolumeSpecName "list-0". PluginName "kubernetes.io/host-path", VolumeGidValue "" Jul 24 08:53:36 bluth k3s[11030]: W0724 08:53:36.708954 11030 empty_dir.go:517] Warning: Failed to clear quota on /var/lib/kubelet/pods/05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b/volumes/kubernetes.io~empty-dir/shared: clearQuota called, but quotas disabled Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.709005 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b-shared" (OuterVolumeSpecName: "shared") pod "05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b" (UID: "05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b"). InnerVolumeSpecName "shared". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Jul 24 08:53:36 bluth k3s[11030]: W0724 08:53:36.708950 11030 empty_dir.go:517] Warning: Failed to clear quota on /var/lib/kubelet/pods/6e3e9791-9bb3-487f-ae50-048fefbef8be/volumes/kubernetes.io~empty-dir/varlogs: clearQuota called, but quotas disabled Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.709058 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e3e9791-9bb3-487f-ae50-048fefbef8be-varlogs" (OuterVolumeSpecName: "varlogs") pod "6e3e9791-9bb3-487f-ae50-048fefbef8be" (UID: "6e3e9791-9bb3-487f-ae50-048fefbef8be"). InnerVolumeSpecName "varlogs". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Jul 24 08:53:36 bluth k3s[11030]: W0724 08:53:36.709087 11030 empty_dir.go:517] Warning: Failed to clear quota on /var/lib/kubelet/pods/2fac2fcc-a503-4887-810c-c2ff11c07852/volumes/kubernetes.io~empty-dir/varlogs: clearQuota called, but quotas disabled Jul 24 08:53:36 bluth k3s[11030]: W0724 08:53:36.709096 11030 empty_dir.go:517] Warning: Failed to clear quota on /var/lib/kubelet/pods/be152b2b-fd1c-4a36-9081-ac6be2fb7901/volumes/kubernetes.io~empty-dir/varlogs: clearQuota called, but quotas disabled Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.709130 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fac2fcc-a503-4887-810c-c2ff11c07852-varlogs" (OuterVolumeSpecName: "varlogs") pod "2fac2fcc-a503-4887-810c-c2ff11c07852" (UID: "2fac2fcc-a503-4887-810c-c2ff11c07852"). InnerVolumeSpecName "varlogs". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.709139 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be152b2b-fd1c-4a36-9081-ac6be2fb7901-varlogs" (OuterVolumeSpecName: "varlogs") pod "be152b2b-fd1c-4a36-9081-ac6be2fb7901" (UID: "be152b2b-fd1c-4a36-9081-ac6be2fb7901"). InnerVolumeSpecName "varlogs". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Jul 24 08:53:36 bluth k3s[11030]: E0724 08:53:36.765494 11030 kubelet.go:1730] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[kube-api-access-lcltd], unattached volumes=[ix-host-path-home-assistant-0 ix-host-path-home-assistant-1 kube-api-access-lcltd]: timed out waiting for the condition" pod="ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-mz95k" Jul 24 08:53:36 bluth k3s[11030]: E0724 08:53:36.765524 11030 pod_workers.go:949] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-lcltd], unattached volumes=[ix-host-path-home-assistant-0 ix-host-path-home-assistant-1 kube-api-access-lcltd]: timed out waiting for the condition" pod="ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-mz95k" podUID=3f1f74fe-1d74-4b61-9bd2-083782908a35 Jul 24 08:53:36 bluth k3s[11030]: E0724 08:53:36.766955 11030 kubelet.go:1730] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[config], unattached volumes=[config shared temp varlogs kube-api-access-mc4q9]: timed out waiting for the condition" pod="ix-prowlarr/prowlarr-5bcc99cc47-rgrsl" Jul 24 08:53:36 bluth k3s[11030]: E0724 08:53:36.766976 11030 pod_workers.go:949] "Error syncing pod, skipping" err="unmounted volumes=[config], unattached volumes=[config shared temp varlogs kube-api-access-mc4q9]: timed out waiting for the condition" pod="ix-prowlarr/prowlarr-5bcc99cc47-rgrsl" podUID=afd760a7-8b6b-470e-aa87-8ed16adb34bb Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.796891 11030 reconciler.go:300] "Volume detached for volume \"varlogs\" (UniqueName: \"kubernetes.io/empty-dir/6e3e9791-9bb3-487f-ae50-048fefbef8be-varlogs\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.796916 11030 reconciler.go:300] "Volume detached for volume \"varlogs\" (UniqueName: \"kubernetes.io/empty-dir/be152b2b-fd1c-4a36-9081-ac6be2fb7901-varlogs\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.796925 11030 reconciler.go:300] "Volume detached for volume \"list-0\" (UniqueName: \"kubernetes.io/host-path/6e3e9791-9bb3-487f-ae50-048fefbef8be-list-0\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.796932 11030 reconciler.go:300] "Volume detached for volume \"list-0\" (UniqueName: \"kubernetes.io/host-path/be152b2b-fd1c-4a36-9081-ac6be2fb7901-list-0\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.796938 11030 reconciler.go:300] "Volume detached for volume \"varlogs\" (UniqueName: \"kubernetes.io/empty-dir/2fac2fcc-a503-4887-810c-c2ff11c07852-varlogs\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.796945 11030 reconciler.go:300] "Volume detached for volume \"list-0\" (UniqueName: \"kubernetes.io/host-path/2fac2fcc-a503-4887-810c-c2ff11c07852-list-0\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.796953 11030 reconciler.go:300] "Volume detached for volume \"list-0\" (UniqueName: \"kubernetes.io/host-path/35ac75bf-b85f-4286-a228-ea8459dde305-list-0\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.796960 11030 reconciler.go:300] "Volume detached for volume \"shared\" (UniqueName: \"kubernetes.io/empty-dir/05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b-shared\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.796968 11030 reconciler.go:300] "Volume detached for volume \"varlogs\" (UniqueName: \"kubernetes.io/empty-dir/35ac75bf-b85f-4286-a228-ea8459dde305-varlogs\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.796974 11030 reconciler.go:300] "Volume detached for volume \"shared\" (UniqueName: \"kubernetes.io/empty-dir/be152b2b-fd1c-4a36-9081-ac6be2fb7901-shared\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.796981 11030 reconciler.go:300] "Volume detached for volume \"shared\" (UniqueName: \"kubernetes.io/empty-dir/6e3e9791-9bb3-487f-ae50-048fefbef8be-shared\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.796987 11030 reconciler.go:300] "Volume detached for volume \"shared\" (UniqueName: \"kubernetes.io/empty-dir/2fac2fcc-a503-4887-810c-c2ff11c07852-shared\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.796993 11030 reconciler.go:300] "Volume detached for volume \"temp\" (UniqueName: \"kubernetes.io/empty-dir/05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b-temp\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.796999 11030 reconciler.go:300] "Volume detached for volume \"varlogs\" (UniqueName: \"kubernetes.io/empty-dir/05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b-varlogs\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:53:36 bluth k3s[11030]: I0724 08:53:36.797006 11030 reconciler.go:300] "Volume detached for volume \"shared\" (UniqueName: \"kubernetes.io/empty-dir/35ac75bf-b85f-4286-a228-ea8459dde305-shared\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:53:37 bluth k3s[11030]: E0724 08:53:37.478561 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: bd1d7b7eda1457ba032977be046038eb7b0e4de8b978957e22459377c7dfc249" podSandboxID="bd1d7b7eda1457ba032977be046038eb7b0e4de8b978957e22459377c7dfc249" pod="ix-tdarr/svclb-tdarr-comm-npvsl" Jul 24 08:53:37 bluth k3s[11030]: E0724 08:53:37.479620 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: bd1d7b7eda1457ba032977be046038eb7b0e4de8b978957e22459377c7dfc249" podSandboxID="bd1d7b7eda1457ba032977be046038eb7b0e4de8b978957e22459377c7dfc249" pod="ix-tdarr/svclb-tdarr-comm-npvsl" Jul 24 08:53:37 bluth k3s[11030]: I0724 08:53:37.481087 11030 cni.go:334] "CNI failed to retrieve network namespace path" err="cannot find network namespace for the terminated container \"93ce773435d99e6c171bcc350a4dec1fa738edda5dc537ff4e0e2ccb1dd443be\"" Jul 24 08:53:37 bluth k3s[11030]: I0724 08:53:37.481339 11030 cni.go:334] "CNI failed to retrieve network namespace path" err="cannot find network namespace for the terminated container \"9ac66d3a4484fab3c3fc091d5fb61f0f6cdd9b19f8d48a132bdea423b0ca1e77\"" Jul 24 08:53:37 bluth k3s[11030]: I0724 08:53:37.602190 11030 reconciler.go:300] "Volume detached for volume \"pvc-06521873-c179-499e-9079-194ec6618fdd\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-06521873-c179-499e-9079-194ec6618fdd\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:53:37 bluth k3s[11030]: I0724 08:53:37.602208 11030 reconciler.go:300] "Volume detached for volume \"pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:53:37 bluth k3s[11030]: I0724 08:53:37.602217 11030 reconciler.go:300] "Volume detached for volume \"kube-api-access-xr6bj\" (UniqueName: \"kubernetes.io/projected/05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b-kube-api-access-xr6bj\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:53:37 bluth k3s[11030]: I0724 08:53:37.747656 11030 kubelet_volumes.go:133] "Cleaned up orphaned volume from pod" podUID=05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b path="/var/lib/kubelet/pods/05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b/volumes/kubernetes.io~csi/pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e/mount" Jul 24 08:53:37 bluth k3s[11030]: I0724 08:53:37.747708 11030 kubelet_volumes.go:133] "Cleaned up orphaned volume from pod" podUID=05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b path="/var/lib/kubelet/pods/05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b/volumes/kubernetes.io~csi/pvc-06521873-c179-499e-9079-194ec6618fdd/mount" Jul 24 08:53:37 bluth k3s[11030]: I0724 08:53:37.747749 11030 kubelet_volumes.go:133] "Cleaned up orphaned volume from pod" podUID=05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b path="/var/lib/kubelet/pods/05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b/volumes/kubernetes.io~projected/kube-api-access-xr6bj" Jul 24 08:53:37 bluth k3s[11030]: E0724 08:53:37.748563 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=3 Jul 24 08:53:38 bluth k3s[11030]: E0724 08:53:38.489570 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: bd1d7b7eda1457ba032977be046038eb7b0e4de8b978957e22459377c7dfc249" podSandboxID="bd1d7b7eda1457ba032977be046038eb7b0e4de8b978957e22459377c7dfc249" pod="ix-tdarr/svclb-tdarr-comm-npvsl" Jul 24 08:53:38 bluth k3s[11030]: E0724 08:53:38.490678 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: bd1d7b7eda1457ba032977be046038eb7b0e4de8b978957e22459377c7dfc249" podSandboxID="bd1d7b7eda1457ba032977be046038eb7b0e4de8b978957e22459377c7dfc249" pod="ix-tdarr/svclb-tdarr-comm-npvsl" Jul 24 08:53:38 bluth k3s[11030]: I0724 08:53:38.911362 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mc4q9\" (UniqueName: \"kubernetes.io/projected/afd760a7-8b6b-470e-aa87-8ed16adb34bb-kube-api-access-mc4q9\") pod \"afd760a7-8b6b-470e-aa87-8ed16adb34bb\" (UID: \"afd760a7-8b6b-470e-aa87-8ed16adb34bb\") " Jul 24 08:53:38 bluth k3s[11030]: I0724 08:53:38.911407 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"ix-host-path-home-assistant-0\" (UniqueName: \"kubernetes.io/host-path/3f1f74fe-1d74-4b61-9bd2-083782908a35-ix-host-path-home-assistant-0\") pod \"3f1f74fe-1d74-4b61-9bd2-083782908a35\" (UID: \"3f1f74fe-1d74-4b61-9bd2-083782908a35\") " Jul 24 08:53:38 bluth k3s[11030]: I0724 08:53:38.911423 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"temp\" (UniqueName: \"kubernetes.io/empty-dir/afd760a7-8b6b-470e-aa87-8ed16adb34bb-temp\") pod \"afd760a7-8b6b-470e-aa87-8ed16adb34bb\" (UID: \"afd760a7-8b6b-470e-aa87-8ed16adb34bb\") " Jul 24 08:53:38 bluth k3s[11030]: I0724 08:53:38.911448 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"ix-host-path-home-assistant-1\" (UniqueName: \"kubernetes.io/host-path/3f1f74fe-1d74-4b61-9bd2-083782908a35-ix-host-path-home-assistant-1\") pod \"3f1f74fe-1d74-4b61-9bd2-083782908a35\" (UID: \"3f1f74fe-1d74-4b61-9bd2-083782908a35\") " Jul 24 08:53:38 bluth k3s[11030]: I0724 08:53:38.911462 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"shared\" (UniqueName: \"kubernetes.io/empty-dir/afd760a7-8b6b-470e-aa87-8ed16adb34bb-shared\") pod \"afd760a7-8b6b-470e-aa87-8ed16adb34bb\" (UID: \"afd760a7-8b6b-470e-aa87-8ed16adb34bb\") " Jul 24 08:53:38 bluth k3s[11030]: I0724 08:53:38.911476 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"varlogs\" (UniqueName: \"kubernetes.io/empty-dir/afd760a7-8b6b-470e-aa87-8ed16adb34bb-varlogs\") pod \"afd760a7-8b6b-470e-aa87-8ed16adb34bb\" (UID: \"afd760a7-8b6b-470e-aa87-8ed16adb34bb\") " Jul 24 08:53:38 bluth k3s[11030]: W0724 08:53:38.911725 11030 empty_dir.go:517] Warning: Failed to clear quota on /var/lib/kubelet/pods/afd760a7-8b6b-470e-aa87-8ed16adb34bb/volumes/kubernetes.io~empty-dir/varlogs: clearQuota called, but quotas disabled Jul 24 08:53:38 bluth k3s[11030]: I0724 08:53:38.911762 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/3f1f74fe-1d74-4b61-9bd2-083782908a35-ix-host-path-home-assistant-0" (OuterVolumeSpecName: "ix-host-path-home-assistant-0") pod "3f1f74fe-1d74-4b61-9bd2-083782908a35" (UID: "3f1f74fe-1d74-4b61-9bd2-083782908a35"). InnerVolumeSpecName "ix-host-path-home-assistant-0". PluginName "kubernetes.io/host-path", VolumeGidValue "" Jul 24 08:53:38 bluth k3s[11030]: I0724 08:53:38.911781 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd760a7-8b6b-470e-aa87-8ed16adb34bb-varlogs" (OuterVolumeSpecName: "varlogs") pod "afd760a7-8b6b-470e-aa87-8ed16adb34bb" (UID: "afd760a7-8b6b-470e-aa87-8ed16adb34bb"). InnerVolumeSpecName "varlogs". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Jul 24 08:53:38 bluth k3s[11030]: I0724 08:53:38.911789 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/3f1f74fe-1d74-4b61-9bd2-083782908a35-ix-host-path-home-assistant-1" (OuterVolumeSpecName: "ix-host-path-home-assistant-1") pod "3f1f74fe-1d74-4b61-9bd2-083782908a35" (UID: "3f1f74fe-1d74-4b61-9bd2-083782908a35"). InnerVolumeSpecName "ix-host-path-home-assistant-1". PluginName "kubernetes.io/host-path", VolumeGidValue "" Jul 24 08:53:38 bluth k3s[11030]: W0724 08:53:38.911898 11030 empty_dir.go:517] Warning: Failed to clear quota on /var/lib/kubelet/pods/afd760a7-8b6b-470e-aa87-8ed16adb34bb/volumes/kubernetes.io~empty-dir/shared: clearQuota called, but quotas disabled Jul 24 08:53:38 bluth k3s[11030]: I0724 08:53:38.911937 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd760a7-8b6b-470e-aa87-8ed16adb34bb-shared" (OuterVolumeSpecName: "shared") pod "afd760a7-8b6b-470e-aa87-8ed16adb34bb" (UID: "afd760a7-8b6b-470e-aa87-8ed16adb34bb"). InnerVolumeSpecName "shared". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Jul 24 08:53:39 bluth k3s[11030]: I0724 08:53:39.012548 11030 reconciler.go:300] "Volume detached for volume \"ix-host-path-home-assistant-0\" (UniqueName: \"kubernetes.io/host-path/3f1f74fe-1d74-4b61-9bd2-083782908a35-ix-host-path-home-assistant-0\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:53:39 bluth k3s[11030]: I0724 08:53:39.012568 11030 reconciler.go:300] "Volume detached for volume \"ix-host-path-home-assistant-1\" (UniqueName: \"kubernetes.io/host-path/3f1f74fe-1d74-4b61-9bd2-083782908a35-ix-host-path-home-assistant-1\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:53:39 bluth k3s[11030]: I0724 08:53:39.012580 11030 reconciler.go:300] "Volume detached for volume \"shared\" (UniqueName: \"kubernetes.io/empty-dir/afd760a7-8b6b-470e-aa87-8ed16adb34bb-shared\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:53:39 bluth k3s[11030]: I0724 08:53:39.012589 11030 reconciler.go:300] "Volume detached for volume \"varlogs\" (UniqueName: \"kubernetes.io/empty-dir/afd760a7-8b6b-470e-aa87-8ed16adb34bb-varlogs\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:53:39 bluth k3s[11030]: E0724 08:53:39.500568 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: bd1d7b7eda1457ba032977be046038eb7b0e4de8b978957e22459377c7dfc249" podSandboxID="bd1d7b7eda1457ba032977be046038eb7b0e4de8b978957e22459377c7dfc249" pod="ix-tdarr/svclb-tdarr-comm-npvsl" Jul 24 08:53:39 bluth k3s[11030]: E0724 08:53:39.501694 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: bd1d7b7eda1457ba032977be046038eb7b0e4de8b978957e22459377c7dfc249" podSandboxID="bd1d7b7eda1457ba032977be046038eb7b0e4de8b978957e22459377c7dfc249" pod="ix-tdarr/svclb-tdarr-comm-npvsl" Jul 24 08:53:39 bluth k3s[11030]: I0724 08:53:39.615541 11030 reconciler.go:300] "Volume detached for volume \"kube-api-access-lcltd\" (UniqueName: \"kubernetes.io/projected/3f1f74fe-1d74-4b61-9bd2-083782908a35-kube-api-access-lcltd\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:53:39 bluth k3s[11030]: I0724 08:53:39.752664 11030 kubelet_volumes.go:133] "Cleaned up orphaned volume from pod" podUID=3f1f74fe-1d74-4b61-9bd2-083782908a35 path="/var/lib/kubelet/pods/3f1f74fe-1d74-4b61-9bd2-083782908a35/volumes/kubernetes.io~projected/kube-api-access-lcltd" Jul 24 08:53:39 bluth k3s[11030]: I0724 08:53:39.752791 11030 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=3f1f74fe-1d74-4b61-9bd2-083782908a35 path="/var/lib/kubelet/pods/3f1f74fe-1d74-4b61-9bd2-083782908a35/volumes" Jul 24 08:53:39 bluth k3s[11030]: E0724 08:53:39.753819 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=3 Jul 24 08:53:40 bluth k3s[11030]: E0724 08:53:40.512346 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: bd1d7b7eda1457ba032977be046038eb7b0e4de8b978957e22459377c7dfc249" podSandboxID="bd1d7b7eda1457ba032977be046038eb7b0e4de8b978957e22459377c7dfc249" pod="ix-tdarr/svclb-tdarr-comm-npvsl" Jul 24 08:53:40 bluth k3s[11030]: E0724 08:53:40.513440 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: bd1d7b7eda1457ba032977be046038eb7b0e4de8b978957e22459377c7dfc249" podSandboxID="bd1d7b7eda1457ba032977be046038eb7b0e4de8b978957e22459377c7dfc249" pod="ix-tdarr/svclb-tdarr-comm-npvsl" Jul 24 08:53:41 bluth k3s[11030]: E0724 08:53:41.524480 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: bd1d7b7eda1457ba032977be046038eb7b0e4de8b978957e22459377c7dfc249" podSandboxID="bd1d7b7eda1457ba032977be046038eb7b0e4de8b978957e22459377c7dfc249" pod="ix-tdarr/svclb-tdarr-comm-npvsl" Jul 24 08:53:41 bluth k3s[11030]: E0724 08:53:41.526124 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: bd1d7b7eda1457ba032977be046038eb7b0e4de8b978957e22459377c7dfc249" podSandboxID="bd1d7b7eda1457ba032977be046038eb7b0e4de8b978957e22459377c7dfc249" pod="ix-tdarr/svclb-tdarr-comm-npvsl" Jul 24 08:53:41 bluth k3s[11030]: E0724 08:53:41.750826 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=3 Jul 24 08:53:42 bluth k3s[11030]: I0724 08:53:42.001157 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be152b2b-fd1c-4a36-9081-ac6be2fb7901-temp" (OuterVolumeSpecName: "temp") pod "be152b2b-fd1c-4a36-9081-ac6be2fb7901" (UID: "be152b2b-fd1c-4a36-9081-ac6be2fb7901"). InnerVolumeSpecName "temp". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Jul 24 08:53:42 bluth k3s[11030]: I0724 08:53:42.001157 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd760a7-8b6b-470e-aa87-8ed16adb34bb-temp" (OuterVolumeSpecName: "temp") pod "afd760a7-8b6b-470e-aa87-8ed16adb34bb" (UID: "afd760a7-8b6b-470e-aa87-8ed16adb34bb"). InnerVolumeSpecName "temp". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Jul 24 08:53:42 bluth k3s[11030]: I0724 08:53:42.001455 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e3e9791-9bb3-487f-ae50-048fefbef8be-varrun" (OuterVolumeSpecName: "varrun") pod "6e3e9791-9bb3-487f-ae50-048fefbef8be" (UID: "6e3e9791-9bb3-487f-ae50-048fefbef8be"). InnerVolumeSpecName "varrun". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Jul 24 08:53:42 bluth k3s[11030]: I0724 08:53:42.001544 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fac2fcc-a503-4887-810c-c2ff11c07852-kube-api-access-2qgd6" (OuterVolumeSpecName: "kube-api-access-2qgd6") pod "2fac2fcc-a503-4887-810c-c2ff11c07852" (UID: "2fac2fcc-a503-4887-810c-c2ff11c07852"). InnerVolumeSpecName "kube-api-access-2qgd6". PluginName "kubernetes.io/projected", VolumeGidValue "" Jul 24 08:53:42 bluth k3s[11030]: I0724 08:53:42.001621 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e3e9791-9bb3-487f-ae50-048fefbef8be-temp" (OuterVolumeSpecName: "temp") pod "6e3e9791-9bb3-487f-ae50-048fefbef8be" (UID: "6e3e9791-9bb3-487f-ae50-048fefbef8be"). InnerVolumeSpecName "temp". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Jul 24 08:53:42 bluth k3s[11030]: I0724 08:53:42.001802 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afd760a7-8b6b-470e-aa87-8ed16adb34bb-kube-api-access-mc4q9" (OuterVolumeSpecName: "kube-api-access-mc4q9") pod "afd760a7-8b6b-470e-aa87-8ed16adb34bb" (UID: "afd760a7-8b6b-470e-aa87-8ed16adb34bb"). InnerVolumeSpecName "kube-api-access-mc4q9". PluginName "kubernetes.io/projected", VolumeGidValue "" Jul 24 08:53:42 bluth k3s[11030]: I0724 08:53:42.001981 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35ac75bf-b85f-4286-a228-ea8459dde305-kube-api-access-2v58k" (OuterVolumeSpecName: "kube-api-access-2v58k") pod "35ac75bf-b85f-4286-a228-ea8459dde305" (UID: "35ac75bf-b85f-4286-a228-ea8459dde305"). InnerVolumeSpecName "kube-api-access-2v58k". PluginName "kubernetes.io/projected", VolumeGidValue "" Jul 24 08:53:42 bluth k3s[11030]: I0724 08:53:42.002298 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fac2fcc-a503-4887-810c-c2ff11c07852-temp" (OuterVolumeSpecName: "temp") pod "2fac2fcc-a503-4887-810c-c2ff11c07852" (UID: "2fac2fcc-a503-4887-810c-c2ff11c07852"). InnerVolumeSpecName "temp". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Jul 24 08:53:42 bluth k3s[11030]: I0724 08:53:42.006828 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35ac75bf-b85f-4286-a228-ea8459dde305-temp" (OuterVolumeSpecName: "temp") pod "35ac75bf-b85f-4286-a228-ea8459dde305" (UID: "35ac75bf-b85f-4286-a228-ea8459dde305"). InnerVolumeSpecName "temp". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Jul 24 08:53:42 bluth k3s[11030]: I0724 08:53:42.007576 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e3e9791-9bb3-487f-ae50-048fefbef8be-kube-api-access-56dxr" (OuterVolumeSpecName: "kube-api-access-56dxr") pod "6e3e9791-9bb3-487f-ae50-048fefbef8be" (UID: "6e3e9791-9bb3-487f-ae50-048fefbef8be"). InnerVolumeSpecName "kube-api-access-56dxr". PluginName "kubernetes.io/projected", VolumeGidValue "" Jul 24 08:53:42 bluth k3s[11030]: I0724 08:53:42.007636 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be152b2b-fd1c-4a36-9081-ac6be2fb7901-kube-api-access-vjp24" (OuterVolumeSpecName: "kube-api-access-vjp24") pod "be152b2b-fd1c-4a36-9081-ac6be2fb7901" (UID: "be152b2b-fd1c-4a36-9081-ac6be2fb7901"). InnerVolumeSpecName "kube-api-access-vjp24". PluginName "kubernetes.io/projected", VolumeGidValue "" Jul 24 08:53:42 bluth k3s[11030]: I0724 08:53:42.029326 11030 reconciler.go:300] "Volume detached for volume \"temp\" (UniqueName: \"kubernetes.io/empty-dir/2fac2fcc-a503-4887-810c-c2ff11c07852-temp\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:53:42 bluth k3s[11030]: I0724 08:53:42.029344 11030 reconciler.go:300] "Volume detached for volume \"temp\" (UniqueName: \"kubernetes.io/empty-dir/be152b2b-fd1c-4a36-9081-ac6be2fb7901-temp\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:53:42 bluth k3s[11030]: I0724 08:53:42.029353 11030 reconciler.go:300] "Volume detached for volume \"kube-api-access-vjp24\" (UniqueName: \"kubernetes.io/projected/be152b2b-fd1c-4a36-9081-ac6be2fb7901-kube-api-access-vjp24\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:53:42 bluth k3s[11030]: I0724 08:53:42.029361 11030 reconciler.go:300] "Volume detached for volume \"kube-api-access-2qgd6\" (UniqueName: \"kubernetes.io/projected/2fac2fcc-a503-4887-810c-c2ff11c07852-kube-api-access-2qgd6\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:53:42 bluth k3s[11030]: I0724 08:53:42.029369 11030 reconciler.go:300] "Volume detached for volume \"kube-api-access-mc4q9\" (UniqueName: \"kubernetes.io/projected/afd760a7-8b6b-470e-aa87-8ed16adb34bb-kube-api-access-mc4q9\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:53:42 bluth k3s[11030]: I0724 08:53:42.029375 11030 reconciler.go:300] "Volume detached for volume \"kube-api-access-56dxr\" (UniqueName: \"kubernetes.io/projected/6e3e9791-9bb3-487f-ae50-048fefbef8be-kube-api-access-56dxr\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:53:42 bluth k3s[11030]: I0724 08:53:42.029382 11030 reconciler.go:300] "Volume detached for volume \"temp\" (UniqueName: \"kubernetes.io/empty-dir/35ac75bf-b85f-4286-a228-ea8459dde305-temp\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:53:42 bluth k3s[11030]: I0724 08:53:42.029388 11030 reconciler.go:300] "Volume detached for volume \"varrun\" (UniqueName: \"kubernetes.io/empty-dir/6e3e9791-9bb3-487f-ae50-048fefbef8be-varrun\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:53:42 bluth k3s[11030]: I0724 08:53:42.029394 11030 reconciler.go:300] "Volume detached for volume \"temp\" (UniqueName: \"kubernetes.io/empty-dir/afd760a7-8b6b-470e-aa87-8ed16adb34bb-temp\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:53:42 bluth k3s[11030]: I0724 08:53:42.029400 11030 reconciler.go:300] "Volume detached for volume \"temp\" (UniqueName: \"kubernetes.io/empty-dir/6e3e9791-9bb3-487f-ae50-048fefbef8be-temp\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:53:42 bluth k3s[11030]: I0724 08:53:42.029407 11030 reconciler.go:300] "Volume detached for volume \"kube-api-access-2v58k\" (UniqueName: \"kubernetes.io/projected/35ac75bf-b85f-4286-a228-ea8459dde305-kube-api-access-2v58k\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:53:42 bluth k3s[11030]: I0724 08:53:42.334155 11030 reconciler.go:300] "Volume detached for volume \"pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:53:42 bluth k3s[11030]: I0724 08:53:42.335226 11030 reconciler.go:300] "Volume detached for volume \"pvc-58562e12-4adc-4e83-8131-848f1b39bf00\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-58562e12-4adc-4e83-8131-848f1b39bf00\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:53:42 bluth k3s[11030]: I0724 08:53:42.435670 11030 reconciler.go:300] "Volume detached for volume \"pvc-4e6fcc6b-6890-4751-a09a-75bb6cde146f\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-4e6fcc6b-6890-4751-a09a-75bb6cde146f\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:53:42 bluth k3s[11030]: I0724 08:53:42.435800 11030 reconciler.go:300] "Volume detached for volume \"pvc-2e6b3d10-435c-4ce5-806d-61d551dcef05\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-2e6b3d10-435c-4ce5-806d-61d551dcef05\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:53:42 bluth k3s[11030]: I0724 08:53:42.435880 11030 reconciler.go:300] "Volume detached for volume \"pvc-028dc34a-8c2e-4a0a-b507-c33a02541407\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-028dc34a-8c2e-4a0a-b507-c33a02541407\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:53:42 bluth k3s[11030]: I0724 08:53:42.435954 11030 reconciler.go:300] "Volume detached for volume \"pvc-195f1a32-9c93-4737-983a-8dacee32e453\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-195f1a32-9c93-4737-983a-8dacee32e453\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:53:42 bluth k3s[11030]: I0724 08:53:42.436025 11030 reconciler.go:300] "Volume detached for volume \"pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:53:42 bluth k3s[11030]: I0724 08:53:42.436083 11030 reconciler.go:300] "Volume detached for volume \"pvc-c41ca0c8-d532-4ca2-b6ca-fe3e9ddf87d2\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c41ca0c8-d532-4ca2-b6ca-fe3e9ddf87d2\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:53:42 bluth k3s[11030]: I0724 08:53:42.436151 11030 reconciler.go:300] "Volume detached for volume \"pvc-3706f211-9694-4ec9-88b9-55f66b73b802\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-3706f211-9694-4ec9-88b9-55f66b73b802\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:53:42 bluth k3s[11030]: E0724 08:53:42.536436 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: bd1d7b7eda1457ba032977be046038eb7b0e4de8b978957e22459377c7dfc249" podSandboxID="bd1d7b7eda1457ba032977be046038eb7b0e4de8b978957e22459377c7dfc249" pod="ix-tdarr/svclb-tdarr-comm-npvsl" Jul 24 08:53:42 bluth k3s[11030]: E0724 08:53:42.538090 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: bd1d7b7eda1457ba032977be046038eb7b0e4de8b978957e22459377c7dfc249" podSandboxID="bd1d7b7eda1457ba032977be046038eb7b0e4de8b978957e22459377c7dfc249" pod="ix-tdarr/svclb-tdarr-comm-npvsl" Jul 24 08:53:43 bluth k3s[11030]: I0724 08:53:43.552436 11030 docker_sandbox.go:402] "Failed to read pod IP from plugin/docker" err="networkPlugin cni failed on the status hook for pod \"svclb-tdarr-comm-npvsl_ix-tdarr\": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container \"bd1d7b7eda1457ba032977be046038eb7b0e4de8b978957e22459377c7dfc249\"" Jul 24 08:53:43 bluth k3s[11030]: I0724 08:53:43.554968 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="bd1d7b7eda1457ba032977be046038eb7b0e4de8b978957e22459377c7dfc249" Jul 24 08:53:43 bluth k3s[11030]: I0724 08:53:43.558578 11030 cni.go:334] "CNI failed to retrieve network namespace path" err="cannot find network namespace for the terminated container \"bd1d7b7eda1457ba032977be046038eb7b0e4de8b978957e22459377c7dfc249\"" Jul 24 08:53:43 bluth k3s[11030]: I0724 08:53:43.747829 11030 kubelet_volumes.go:133] "Cleaned up orphaned volume from pod" podUID=2fac2fcc-a503-4887-810c-c2ff11c07852 path="/var/lib/kubelet/pods/2fac2fcc-a503-4887-810c-c2ff11c07852/volumes/kubernetes.io~csi/pvc-58562e12-4adc-4e83-8131-848f1b39bf00/mount" Jul 24 08:53:43 bluth k3s[11030]: I0724 08:53:43.748267 11030 kubelet_volumes.go:133] "Cleaned up orphaned volume from pod" podUID=35ac75bf-b85f-4286-a228-ea8459dde305 path="/var/lib/kubelet/pods/35ac75bf-b85f-4286-a228-ea8459dde305/volumes/kubernetes.io~csi/pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4/mount" Jul 24 08:53:43 bluth k3s[11030]: I0724 08:53:43.748722 11030 kubelet_volumes.go:133] "Cleaned up orphaned volume from pod" podUID=6e3e9791-9bb3-487f-ae50-048fefbef8be path="/var/lib/kubelet/pods/6e3e9791-9bb3-487f-ae50-048fefbef8be/volumes/kubernetes.io~csi/pvc-3706f211-9694-4ec9-88b9-55f66b73b802/mount" Jul 24 08:53:43 bluth k3s[11030]: I0724 08:53:43.748766 11030 kubelet_volumes.go:133] "Cleaned up orphaned volume from pod" podUID=6e3e9791-9bb3-487f-ae50-048fefbef8be path="/var/lib/kubelet/pods/6e3e9791-9bb3-487f-ae50-048fefbef8be/volumes/kubernetes.io~csi/pvc-028dc34a-8c2e-4a0a-b507-c33a02541407/mount" Jul 24 08:53:43 bluth k3s[11030]: I0724 08:53:43.748806 11030 kubelet_volumes.go:133] "Cleaned up orphaned volume from pod" podUID=6e3e9791-9bb3-487f-ae50-048fefbef8be path="/var/lib/kubelet/pods/6e3e9791-9bb3-487f-ae50-048fefbef8be/volumes/kubernetes.io~csi/pvc-4e6fcc6b-6890-4751-a09a-75bb6cde146f/mount" Jul 24 08:53:43 bluth k3s[11030]: I0724 08:53:43.748850 11030 kubelet_volumes.go:133] "Cleaned up orphaned volume from pod" podUID=6e3e9791-9bb3-487f-ae50-048fefbef8be path="/var/lib/kubelet/pods/6e3e9791-9bb3-487f-ae50-048fefbef8be/volumes/kubernetes.io~csi/pvc-c41ca0c8-d532-4ca2-b6ca-fe3e9ddf87d2/mount" Jul 24 08:53:43 bluth k3s[11030]: I0724 08:53:43.748891 11030 kubelet_volumes.go:133] "Cleaned up orphaned volume from pod" podUID=6e3e9791-9bb3-487f-ae50-048fefbef8be path="/var/lib/kubelet/pods/6e3e9791-9bb3-487f-ae50-048fefbef8be/volumes/kubernetes.io~csi/pvc-2e6b3d10-435c-4ce5-806d-61d551dcef05/mount" Jul 24 08:53:43 bluth k3s[11030]: I0724 08:53:43.749948 11030 kubelet_volumes.go:133] "Cleaned up orphaned volume from pod" podUID=afd760a7-8b6b-470e-aa87-8ed16adb34bb path="/var/lib/kubelet/pods/afd760a7-8b6b-470e-aa87-8ed16adb34bb/volumes/kubernetes.io~csi/pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274/mount" Jul 24 08:53:43 bluth k3s[11030]: I0724 08:53:43.750352 11030 kubelet_volumes.go:133] "Cleaned up orphaned volume from pod" podUID=be152b2b-fd1c-4a36-9081-ac6be2fb7901 path="/var/lib/kubelet/pods/be152b2b-fd1c-4a36-9081-ac6be2fb7901/volumes/kubernetes.io~csi/pvc-195f1a32-9c93-4737-983a-8dacee32e453/mount" Jul 24 08:53:43 bluth k3s[11030]: E0724 08:53:43.750597 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:53:44 bluth k3s[11030]: I0724 08:53:44.584123 11030 event.go:294] "Event occurred" object="ix-tdarr/svclb-tdarr-comm" kind="DaemonSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulCreate" message="Created pod: svclb-tdarr-comm-lgh8b" Jul 24 08:53:44 bluth k3s[11030]: I0724 08:53:44.587213 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:53:44 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T08:53:44.876+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 08:53:45 bluth k3s[11030]: I0724 08:53:45.750748 11030 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=69463f72-1255-4fe0-b5a2-59f859d48b92 path="/var/lib/kubelet/pods/69463f72-1255-4fe0-b5a2-59f859d48b92/volumes" Jul 24 08:53:45 bluth k3s[11030]: E0724 08:53:45.752523 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:53:46 bluth k3s[11030]: E0724 08:53:46.595084 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: caecd739834fb496e4f85acb4bbf6596b6d5f784a17abf9b90bd230218f227f3" containerID="caecd739834fb496e4f85acb4bbf6596b6d5f784a17abf9b90bd230218f227f3" Jul 24 08:53:46 bluth k3s[11030]: E0724 08:53:46.595107 11030 kuberuntime_manager.go:1079] "getPodContainerStatuses for pod failed" err="rpc error: code = Unknown desc = Error: No such container: caecd739834fb496e4f85acb4bbf6596b6d5f784a17abf9b90bd230218f227f3" pod="kube-system/openebs-zfs-node-bf7kk" Jul 24 08:53:47 bluth k3s[11030]: E0724 08:53:47.773907 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:53:49 bluth k3s[11030]: E0724 08:53:49.687357 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 45dcd9b69c3fc1c416b4d575574763088bdacb840d4037acfad820d24bbe7f7b" containerID="45dcd9b69c3fc1c416b4d575574763088bdacb840d4037acfad820d24bbe7f7b" Jul 24 08:53:49 bluth k3s[11030]: E0724 08:53:49.687379 11030 kuberuntime_manager.go:1079] "getPodContainerStatuses for pod failed" err="rpc error: code = Unknown desc = Error: No such container: 45dcd9b69c3fc1c416b4d575574763088bdacb840d4037acfad820d24bbe7f7b" pod="ix-qbittorrent/svclb-qbittorrent-ldv82" Jul 24 08:53:49 bluth k3s[11030]: E0724 08:53:49.782249 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:53:51 bluth k3s[11030]: I0724 08:53:51.099958 11030 trace.go:205] Trace[692737572]: "GuaranteedUpdate etcd3" type:*coordination.Lease (24-Jul-2022 08:53:49.702) (total time: 1397ms): Jul 24 08:53:51 bluth k3s[11030]: Trace[692737572]: ---"Transaction committed" 1397ms (08:53:51.099) Jul 24 08:53:51 bluth k3s[11030]: Trace[692737572]: [1.397558008s] [1.397558008s] END Jul 24 08:53:51 bluth k3s[11030]: I0724 08:53:51.100251 11030 trace.go:205] Trace[182642058]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ix-truenas,user-agent:Go-http-client/2.0,audit-id:a7f61e3b-bb20-4568-8bab-63ab612c9140,client:127.0.0.1,accept:application/vnd.kubernetes.protobuf,application/json,protocol:HTTP/2.0 (24-Jul-2022 08:53:49.702) (total time: 1397ms): Jul 24 08:53:51 bluth k3s[11030]: Trace[182642058]: ---"Object stored in database" 1397ms (08:53:51.100) Jul 24 08:53:51 bluth k3s[11030]: Trace[182642058]: [1.39792836s] [1.39792836s] END Jul 24 08:53:51 bluth k3s[11030]: E0724 08:53:51.756343 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:53:53 bluth k3s[11030]: E0724 08:53:53.752888 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:53:53 bluth k3s[11030]: E0724 08:53:53.752987 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 8173768d1c65a73c0aaefa5256406ec639f08671ae7df7013210d6ce62e5a052" containerID="8173768d1c65a73c0aaefa5256406ec639f08671ae7df7013210d6ce62e5a052" Jul 24 08:53:53 bluth k3s[11030]: E0724 08:53:53.753218 11030 kuberuntime_manager.go:1079] "getPodContainerStatuses for pod failed" err="rpc error: code = Unknown desc = Error: No such container: 8173768d1c65a73c0aaefa5256406ec639f08671ae7df7013210d6ce62e5a052" pod="ix-qbittorrent/svclb-qbittorrent-torrentudp-62ksx" Jul 24 08:53:53 bluth k3s[11030]: E0724 08:53:53.756074 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 782924fe93893e3f55e40b0f59f0d67d687452e1d614160a7b2b0975d992454a" containerID="782924fe93893e3f55e40b0f59f0d67d687452e1d614160a7b2b0975d992454a" Jul 24 08:53:53 bluth k3s[11030]: E0724 08:53:53.756093 11030 kuberuntime_manager.go:1079] "getPodContainerStatuses for pod failed" err="rpc error: code = Unknown desc = Error: No such container: 782924fe93893e3f55e40b0f59f0d67d687452e1d614160a7b2b0975d992454a" pod="ix-plex/plex-bd956645c-95gj8" Jul 24 08:53:54 bluth k3s[11030]: E0724 08:53:54.786451 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: b36856ea81b4de326bf77efba4b7623b64e883bdfb64f26e1b89ece06ae9bced" containerID="b36856ea81b4de326bf77efba4b7623b64e883bdfb64f26e1b89ece06ae9bced" Jul 24 08:53:54 bluth k3s[11030]: E0724 08:53:54.786475 11030 kuberuntime_manager.go:1079] "getPodContainerStatuses for pod failed" err="rpc error: code = Unknown desc = Error: No such container: b36856ea81b4de326bf77efba4b7623b64e883bdfb64f26e1b89ece06ae9bced" pod="ix-prowlarr/svclb-prowlarr-qhcm8" Jul 24 08:53:55 bluth k3s[11030]: E0724 08:53:55.783407 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:53:57 bluth k3s[11030]: E0724 08:53:57.755010 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:53:59 bluth k3s[11030]: E0724 08:53:59.753278 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:54:01 bluth k3s[11030]: E0724 08:54:01.758061 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:54:03 bluth k3s[11030]: E0724 08:54:03.784363 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:54:05 bluth k3s[11030]: E0724 08:54:05.771421 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:54:06 bluth k3s[11030]: E0724 08:54:06.195667 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: f0c517a8f8e10361e1a200c82e3452308e4a1d821ffd44d7f03d256c49220965" containerID="f0c517a8f8e10361e1a200c82e3452308e4a1d821ffd44d7f03d256c49220965" Jul 24 08:54:06 bluth k3s[11030]: E0724 08:54:06.195690 11030 kuberuntime_manager.go:1079] "getPodContainerStatuses for pod failed" err="rpc error: code = Unknown desc = Error: No such container: f0c517a8f8e10361e1a200c82e3452308e4a1d821ffd44d7f03d256c49220965" pod="ix-tdarr/svclb-tdarr-44c97" Jul 24 08:54:06 bluth k3s[11030]: time="2022-07-24T08:54:06+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-qbittorrent\", Name:\"qbittorrent\", UID:\"728a4977-c72e-46a0-96f2-a9279b009171\", APIVersion:\"v1\", ResourceVersion:\"3027187\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-qbittorrent/svclb-qbittorrent" Jul 24 08:54:06 bluth k3s[11030]: time="2022-07-24T08:54:06+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-qbittorrent\", Name:\"qbittorrent\", UID:\"728a4977-c72e-46a0-96f2-a9279b009171\", APIVersion:\"v1\", ResourceVersion:\"3027187\", FieldPath:\"\"}): type: 'Normal' reason: 'UpdatedIngressIP' LoadBalancer Ingress IP addresses updated: 192.168.0.143" Jul 24 08:54:06 bluth k3s[11030]: time="2022-07-24T08:54:06+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-qbittorrent\", Name:\"qbittorrent\", UID:\"728a4977-c72e-46a0-96f2-a9279b009171\", APIVersion:\"v1\", ResourceVersion:\"3027645\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-qbittorrent/svclb-qbittorrent" Jul 24 08:54:07 bluth k3s[11030]: E0724 08:54:07.750377 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:54:07 bluth k3s[11030]: E0724 08:54:07.831134 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: e0a015a83f4f225c152c833960e5c4fcee97f9db0f8aa22d1b75e978b477cf6d" containerID="e0a015a83f4f225c152c833960e5c4fcee97f9db0f8aa22d1b75e978b477cf6d" Jul 24 08:54:07 bluth k3s[11030]: E0724 08:54:07.831156 11030 kuberuntime_manager.go:1079] "getPodContainerStatuses for pod failed" err="rpc error: code = Unknown desc = Error: No such container: e0a015a83f4f225c152c833960e5c4fcee97f9db0f8aa22d1b75e978b477cf6d" pod="kube-system/openebs-zfs-controller-0" Jul 24 08:54:07 bluth k3s[11030]: E0724 08:54:07.836418 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 5a2bde0c9fe43ee4870ac69ebca1c9f130c43dd8b4120fa0d3b147738a284b12" containerID="5a2bde0c9fe43ee4870ac69ebca1c9f130c43dd8b4120fa0d3b147738a284b12" Jul 24 08:54:07 bluth k3s[11030]: E0724 08:54:07.836442 11030 kuberuntime_manager.go:1079] "getPodContainerStatuses for pod failed" err="rpc error: code = Unknown desc = Error: No such container: 5a2bde0c9fe43ee4870ac69ebca1c9f130c43dd8b4120fa0d3b147738a284b12" pod="ix-radarr/svclb-radarr-nwf7v" Jul 24 08:54:07 bluth k3s[11030]: time="2022-07-24T08:54:07+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-qbittorrent\", Name:\"qbittorrent-torrentudp\", UID:\"1d92e74f-b912-46f1-b780-c740f80bb9ca\", APIVersion:\"v1\", ResourceVersion:\"2504584\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-qbittorrent/svclb-qbittorrent-torrentudp" Jul 24 08:54:07 bluth k3s[11030]: E0724 08:54:07.874781 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 09b6bdd013ac1def83a478d3daa31d3df518a05424e45777166f72eb7a761612" containerID="09b6bdd013ac1def83a478d3daa31d3df518a05424e45777166f72eb7a761612" Jul 24 08:54:07 bluth k3s[11030]: E0724 08:54:07.874824 11030 kuberuntime_manager.go:1079] "getPodContainerStatuses for pod failed" err="rpc error: code = Unknown desc = Error: No such container: 09b6bdd013ac1def83a478d3daa31d3df518a05424e45777166f72eb7a761612" pod="kube-system/coredns-d76bd69b-mhb5r" Jul 24 08:54:07 bluth k3s[11030]: time="2022-07-24T08:54:07+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-prowlarr\", Name:\"prowlarr\", UID:\"bd43faf9-73db-4bcd-a678-c673a7b54098\", APIVersion:\"v1\", ResourceVersion:\"2334148\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-prowlarr/svclb-prowlarr" Jul 24 08:54:09 bluth k3s[11030]: E0724 08:54:09.764827 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:54:11 bluth k3s[11030]: E0724 08:54:11.763869 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:54:13 bluth k3s[11030]: E0724 08:54:13.750308 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:54:14 bluth k3s[11030]: I0724 08:54:14.855618 11030 scope.go:110] "RemoveContainer" containerID="977f04ce2d09fd39f08c474227a4f5a023ba618a6c2b92f6c2d0cf7fdcc70578" Jul 24 08:54:15 bluth k3s[11030]: E0724 08:54:15.789289 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:54:17 bluth k3s[11030]: E0724 08:54:17.755204 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:54:19 bluth k3s[11030]: E0724 08:54:19.758037 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:54:20 bluth k3s[11030]: E0724 08:54:20.719258 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: fe7bdbe3401af33e1d6685ff8499d4e2ac6f1cdf9c60e748dc590c5b44b791af" containerID="fe7bdbe3401af33e1d6685ff8499d4e2ac6f1cdf9c60e748dc590c5b44b791af" Jul 24 08:54:20 bluth k3s[11030]: E0724 08:54:20.719313 11030 kuberuntime_manager.go:1079] "getPodContainerStatuses for pod failed" err="rpc error: code = Unknown desc = Error: No such container: fe7bdbe3401af33e1d6685ff8499d4e2ac6f1cdf9c60e748dc590c5b44b791af" pod="ix-qbittorrent/svclb-qbittorrent-torrent-8qgf8" Jul 24 08:54:20 bluth k3s[11030]: time="2022-07-24T08:54:20+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-radarr\", Name:\"radarr\", UID:\"2ceba47d-fa27-462f-9816-af4352781795\", APIVersion:\"v1\", ResourceVersion:\"2504670\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-radarr/svclb-radarr" Jul 24 08:54:21 bluth k3s[11030]: E0724 08:54:21.756050 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 2c03ead7d0c9acbaa5c5287c0fc7c095b28c281ee20ff47d5f657ccb9db92cbb" containerID="2c03ead7d0c9acbaa5c5287c0fc7c095b28c281ee20ff47d5f657ccb9db92cbb" Jul 24 08:54:21 bluth k3s[11030]: E0724 08:54:21.756074 11030 kuberuntime_manager.go:1079] "getPodContainerStatuses for pod failed" err="rpc error: code = Unknown desc = Error: No such container: 2c03ead7d0c9acbaa5c5287c0fc7c095b28c281ee20ff47d5f657ccb9db92cbb" pod="ix-sonarr/svclb-sonarr-69zgj" Jul 24 08:54:21 bluth k3s[11030]: E0724 08:54:21.759735 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:54:21 bluth k3s[11030]: E0724 08:54:21.760089 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 0116727ef517122669f9ac99183cd270fe069d99a1a6d6be05567380ffdfd0af" containerID="0116727ef517122669f9ac99183cd270fe069d99a1a6d6be05567380ffdfd0af" Jul 24 08:54:21 bluth k3s[11030]: E0724 08:54:21.760107 11030 kuberuntime_manager.go:1079] "getPodContainerStatuses for pod failed" err="rpc error: code = Unknown desc = Error: No such container: 0116727ef517122669f9ac99183cd270fe069d99a1a6d6be05567380ffdfd0af" pod="ix-esphome/svclb-esphome-q4lrj" Jul 24 08:54:23 bluth k3s[11030]: E0724 08:54:23.754759 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:54:25 bluth k3s[11030]: E0724 08:54:25.757536 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:54:27 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T08:54:27.467+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 08:54:27 bluth k3s[11030]: E0724 08:54:27.757791 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:54:29 bluth k3s[11030]: E0724 08:54:29.794702 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:54:29 bluth k3s[11030]: I0724 08:54:29.794753 11030 scope.go:110] "RemoveContainer" containerID="8173768d1c65a73c0aaefa5256406ec639f08671ae7df7013210d6ce62e5a052" Jul 24 08:54:30 bluth k3s[11030]: time="2022-07-24T08:54:30+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-tdarr\", Name:\"tdarr\", UID:\"90709eb9-3d84-4b91-b6ab-73ece8ffcb86\", APIVersion:\"v1\", ResourceVersion:\"2504489\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-tdarr/svclb-tdarr" Jul 24 08:54:30 bluth k3s[11030]: time="2022-07-24T08:54:30+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-qbittorrent\", Name:\"qbittorrent-torrent\", UID:\"29b172f8-d799-4e0c-aac7-78ec5af5c01b\", APIVersion:\"v1\", ResourceVersion:\"2334164\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-qbittorrent/svclb-qbittorrent-torrent" Jul 24 08:54:30 bluth k3s[11030]: time="2022-07-24T08:54:30+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-prowlarr\", Name:\"prowlarr\", UID:\"bd43faf9-73db-4bcd-a678-c673a7b54098\", APIVersion:\"v1\", ResourceVersion:\"2334148\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-prowlarr/svclb-prowlarr" Jul 24 08:54:30 bluth k3s[11030]: time="2022-07-24T08:54:30+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-prowlarr\", Name:\"prowlarr\", UID:\"bd43faf9-73db-4bcd-a678-c673a7b54098\", APIVersion:\"v1\", ResourceVersion:\"2334148\", FieldPath:\"\"}): type: 'Normal' reason: 'UpdatedIngressIP' LoadBalancer Ingress IP addresses updated: " Jul 24 08:54:30 bluth k3s[11030]: I0724 08:54:30.522294 11030 event.go:294] "Event occurred" object="ix-prowlarr/svclb-prowlarr" kind="DaemonSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulCreate" message="Created pod: svclb-prowlarr-jgkkl" Jul 24 08:54:30 bluth k3s[11030]: time="2022-07-24T08:54:30+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-prowlarr\", Name:\"prowlarr\", UID:\"bd43faf9-73db-4bcd-a678-c673a7b54098\", APIVersion:\"v1\", ResourceVersion:\"3027702\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-prowlarr/svclb-prowlarr" Jul 24 08:54:30 bluth k3s[11030]: I0724 08:54:30.528677 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:54:30 bluth k3s[11030]: I0724 08:54:30.538507 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/host-path/495a78dd-7609-44be-bb7e-7036e65c860b-data\") pod \"495a78dd-7609-44be-bb7e-7036e65c860b\" (UID: \"495a78dd-7609-44be-bb7e-7036e65c860b\") " Jul 24 08:54:30 bluth k3s[11030]: I0724 08:54:30.538541 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"shared\" (UniqueName: \"kubernetes.io/empty-dir/495a78dd-7609-44be-bb7e-7036e65c860b-shared\") pod \"495a78dd-7609-44be-bb7e-7036e65c860b\" (UID: \"495a78dd-7609-44be-bb7e-7036e65c860b\") " Jul 24 08:54:30 bluth k3s[11030]: I0724 08:54:30.538555 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/495a78dd-7609-44be-bb7e-7036e65c860b-config\") pod \"495a78dd-7609-44be-bb7e-7036e65c860b\" (UID: \"495a78dd-7609-44be-bb7e-7036e65c860b\") " Jul 24 08:54:30 bluth k3s[11030]: I0724 08:54:30.538572 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdlrl\" (UniqueName: \"kubernetes.io/projected/495a78dd-7609-44be-bb7e-7036e65c860b-kube-api-access-xdlrl\") pod \"495a78dd-7609-44be-bb7e-7036e65c860b\" (UID: \"495a78dd-7609-44be-bb7e-7036e65c860b\") " Jul 24 08:54:30 bluth k3s[11030]: I0724 08:54:30.538586 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"transcode\" (UniqueName: \"kubernetes.io/host-path/495a78dd-7609-44be-bb7e-7036e65c860b-transcode\") pod \"495a78dd-7609-44be-bb7e-7036e65c860b\" (UID: \"495a78dd-7609-44be-bb7e-7036e65c860b\") " Jul 24 08:54:30 bluth k3s[11030]: I0724 08:54:30.538601 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"plex-probe-check\" (UniqueName: \"kubernetes.io/configmap/495a78dd-7609-44be-bb7e-7036e65c860b-plex-probe-check\") pod \"495a78dd-7609-44be-bb7e-7036e65c860b\" (UID: \"495a78dd-7609-44be-bb7e-7036e65c860b\") " Jul 24 08:54:30 bluth k3s[11030]: I0724 08:54:30.538603 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/495a78dd-7609-44be-bb7e-7036e65c860b-data" (OuterVolumeSpecName: "data") pod "495a78dd-7609-44be-bb7e-7036e65c860b" (UID: "495a78dd-7609-44be-bb7e-7036e65c860b"). InnerVolumeSpecName "data". PluginName "kubernetes.io/host-path", VolumeGidValue "" Jul 24 08:54:30 bluth k3s[11030]: I0724 08:54:30.538613 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"shared-logs\" (UniqueName: \"kubernetes.io/empty-dir/495a78dd-7609-44be-bb7e-7036e65c860b-shared-logs\") pod \"495a78dd-7609-44be-bb7e-7036e65c860b\" (UID: \"495a78dd-7609-44be-bb7e-7036e65c860b\") " Jul 24 08:54:30 bluth k3s[11030]: I0724 08:54:30.538637 11030 reconciler.go:300] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/host-path/495a78dd-7609-44be-bb7e-7036e65c860b-data\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:54:30 bluth k3s[11030]: I0724 08:54:30.538625 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/495a78dd-7609-44be-bb7e-7036e65c860b-transcode" (OuterVolumeSpecName: "transcode") pod "495a78dd-7609-44be-bb7e-7036e65c860b" (UID: "495a78dd-7609-44be-bb7e-7036e65c860b"). InnerVolumeSpecName "transcode". PluginName "kubernetes.io/host-path", VolumeGidValue "" Jul 24 08:54:30 bluth k3s[11030]: I0724 08:54:30.538603 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/495a78dd-7609-44be-bb7e-7036e65c860b-config" (OuterVolumeSpecName: "config") pod "495a78dd-7609-44be-bb7e-7036e65c860b" (UID: "495a78dd-7609-44be-bb7e-7036e65c860b"). InnerVolumeSpecName "config". PluginName "kubernetes.io/host-path", VolumeGidValue "" Jul 24 08:54:30 bluth k3s[11030]: W0724 08:54:30.538816 11030 empty_dir.go:517] Warning: Failed to clear quota on /var/lib/kubelet/pods/495a78dd-7609-44be-bb7e-7036e65c860b/volumes/kubernetes.io~empty-dir/shared-logs: clearQuota called, but quotas disabled Jul 24 08:54:30 bluth k3s[11030]: W0724 08:54:30.539203 11030 empty_dir.go:517] Warning: Failed to clear quota on /var/lib/kubelet/pods/495a78dd-7609-44be-bb7e-7036e65c860b/volumes/kubernetes.io~empty-dir/shared: clearQuota called, but quotas disabled Jul 24 08:54:30 bluth k3s[11030]: I0724 08:54:30.539254 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/495a78dd-7609-44be-bb7e-7036e65c860b-shared" (OuterVolumeSpecName: "shared") pod "495a78dd-7609-44be-bb7e-7036e65c860b" (UID: "495a78dd-7609-44be-bb7e-7036e65c860b"). InnerVolumeSpecName "shared". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Jul 24 08:54:30 bluth k3s[11030]: I0724 08:54:30.540372 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/495a78dd-7609-44be-bb7e-7036e65c860b-kube-api-access-xdlrl" (OuterVolumeSpecName: "kube-api-access-xdlrl") pod "495a78dd-7609-44be-bb7e-7036e65c860b" (UID: "495a78dd-7609-44be-bb7e-7036e65c860b"). InnerVolumeSpecName "kube-api-access-xdlrl". PluginName "kubernetes.io/projected", VolumeGidValue "" Jul 24 08:54:30 bluth k3s[11030]: W0724 08:54:30.542203 11030 mount_helper_common.go:133] Warning: "/var/lib/kubelet/pods/495a78dd-7609-44be-bb7e-7036e65c860b/volume-subpaths/plex-probe-check/plex/5" is not a mountpoint, deleting Jul 24 08:54:30 bluth k3s[11030]: W0724 08:54:30.542498 11030 empty_dir.go:517] Warning: Failed to clear quota on /var/lib/kubelet/pods/495a78dd-7609-44be-bb7e-7036e65c860b/volumes/kubernetes.io~configmap/plex-probe-check: clearQuota called, but quotas disabled Jul 24 08:54:30 bluth k3s[11030]: I0724 08:54:30.542675 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/495a78dd-7609-44be-bb7e-7036e65c860b-plex-probe-check" (OuterVolumeSpecName: "plex-probe-check") pod "495a78dd-7609-44be-bb7e-7036e65c860b" (UID: "495a78dd-7609-44be-bb7e-7036e65c860b"). InnerVolumeSpecName "plex-probe-check". PluginName "kubernetes.io/configmap", VolumeGidValue "" Jul 24 08:54:30 bluth k3s[11030]: I0724 08:54:30.639802 11030 reconciler.go:300] "Volume detached for volume \"shared\" (UniqueName: \"kubernetes.io/empty-dir/495a78dd-7609-44be-bb7e-7036e65c860b-shared\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:54:30 bluth k3s[11030]: I0724 08:54:30.639867 11030 reconciler.go:300] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/host-path/495a78dd-7609-44be-bb7e-7036e65c860b-config\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:54:30 bluth k3s[11030]: I0724 08:54:30.639904 11030 reconciler.go:300] "Volume detached for volume \"kube-api-access-xdlrl\" (UniqueName: \"kubernetes.io/projected/495a78dd-7609-44be-bb7e-7036e65c860b-kube-api-access-xdlrl\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:54:30 bluth k3s[11030]: I0724 08:54:30.639933 11030 reconciler.go:300] "Volume detached for volume \"transcode\" (UniqueName: \"kubernetes.io/host-path/495a78dd-7609-44be-bb7e-7036e65c860b-transcode\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:54:30 bluth k3s[11030]: I0724 08:54:30.639974 11030 reconciler.go:300] "Volume detached for volume \"plex-probe-check\" (UniqueName: \"kubernetes.io/configmap/495a78dd-7609-44be-bb7e-7036e65c860b-plex-probe-check\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:54:31 bluth k3s[11030]: I0724 08:54:31.159708 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/495a78dd-7609-44be-bb7e-7036e65c860b-shared-logs" (OuterVolumeSpecName: "shared-logs") pod "495a78dd-7609-44be-bb7e-7036e65c860b" (UID: "495a78dd-7609-44be-bb7e-7036e65c860b"). InnerVolumeSpecName "shared-logs". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Jul 24 08:54:31 bluth k3s[11030]: I0724 08:54:31.251755 11030 reconciler.go:300] "Volume detached for volume \"shared-logs\" (UniqueName: \"kubernetes.io/empty-dir/495a78dd-7609-44be-bb7e-7036e65c860b-shared-logs\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:54:31 bluth k3s[11030]: time="2022-07-24T08:54:31+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-prowlarr\", Name:\"prowlarr\", UID:\"bd43faf9-73db-4bcd-a678-c673a7b54098\", APIVersion:\"v1\", ResourceVersion:\"3027702\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-prowlarr/svclb-prowlarr" Jul 24 08:54:31 bluth k3s[11030]: I0724 08:54:31.753256 11030 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=01a5e89a-9794-4261-ada8-82a5e21ccb7d path="/var/lib/kubelet/pods/01a5e89a-9794-4261-ada8-82a5e21ccb7d/volumes" Jul 24 08:54:31 bluth k3s[11030]: E0724 08:54:31.757634 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:54:32 bluth k3s[11030]: I0724 08:54:32.816980 11030 event.go:294] "Event occurred" object="ix-qbittorrent/svclb-qbittorrent" kind="DaemonSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulCreate" message="Created pod: svclb-qbittorrent-7jf9d" Jul 24 08:54:32 bluth k3s[11030]: I0724 08:54:32.825061 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:54:33 bluth k3s[11030]: I0724 08:54:33.755012 11030 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=ad6ed51e-d3e1-430c-9bad-528f4ae492c5 path="/var/lib/kubelet/pods/ad6ed51e-d3e1-430c-9bad-528f4ae492c5/volumes" Jul 24 08:54:33 bluth k3s[11030]: E0724 08:54:33.755848 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:54:33 bluth k3s[11030]: I0724 08:54:33.874768 11030 trace.go:205] Trace[490241159]: "GuaranteedUpdate etcd3" type:*core.Pod (24-Jul-2022 08:54:33.278) (total time: 596ms): Jul 24 08:54:33 bluth k3s[11030]: Trace[490241159]: ---"Transaction committed" 587ms (08:54:33.874) Jul 24 08:54:33 bluth k3s[11030]: Trace[490241159]: [596.552578ms] [596.552578ms] END Jul 24 08:54:33 bluth k3s[11030]: I0724 08:54:33.876675 11030 trace.go:205] Trace[1040271206]: "Patch" url:/api/v1/namespaces/kube-system/pods/coredns-d76bd69b-mhb5r/status,user-agent:Go-http-client/2.0,audit-id:2dbd7f8c-b344-4c0e-bb80-7024abae76a6,client:127.0.0.1,accept:application/vnd.kubernetes.protobuf,application/json,protocol:HTTP/2.0 (24-Jul-2022 08:54:33.277) (total time: 598ms): Jul 24 08:54:33 bluth k3s[11030]: Trace[1040271206]: ---"Object stored in database" 589ms (08:54:33.876) Jul 24 08:54:33 bluth k3s[11030]: Trace[1040271206]: [598.662459ms] [598.662459ms] END Jul 24 08:54:33 bluth k3s[11030]: time="2022-07-24T08:54:33+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-radarr\", Name:\"radarr\", UID:\"2ceba47d-fa27-462f-9816-af4352781795\", APIVersion:\"v1\", ResourceVersion:\"2504670\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-radarr/svclb-radarr" Jul 24 08:54:33 bluth k3s[11030]: time="2022-07-24T08:54:33+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-radarr\", Name:\"radarr\", UID:\"2ceba47d-fa27-462f-9816-af4352781795\", APIVersion:\"v1\", ResourceVersion:\"2504670\", FieldPath:\"\"}): type: 'Normal' reason: 'UpdatedIngressIP' LoadBalancer Ingress IP addresses updated: " Jul 24 08:54:33 bluth k3s[11030]: time="2022-07-24T08:54:33+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-radarr\", Name:\"radarr\", UID:\"2ceba47d-fa27-462f-9816-af4352781795\", APIVersion:\"v1\", ResourceVersion:\"3027740\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-radarr/svclb-radarr" Jul 24 08:54:33 bluth k3s[11030]: time="2022-07-24T08:54:33+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-radarr\", Name:\"radarr\", UID:\"2ceba47d-fa27-462f-9816-af4352781795\", APIVersion:\"v1\", ResourceVersion:\"3027740\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-radarr/svclb-radarr" Jul 24 08:54:33 bluth k3s[11030]: I0724 08:54:33.928248 11030 event.go:294] "Event occurred" object="ix-radarr/svclb-radarr" kind="DaemonSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulCreate" message="Created pod: svclb-radarr-s72px" Jul 24 08:54:33 bluth k3s[11030]: I0724 08:54:33.931740 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:54:34 bluth k3s[11030]: time="2022-07-24T08:54:34+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-qbittorrent\", Name:\"qbittorrent-torrentudp\", UID:\"1d92e74f-b912-46f1-b780-c740f80bb9ca\", APIVersion:\"v1\", ResourceVersion:\"2504584\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-qbittorrent/svclb-qbittorrent-torrentudp" Jul 24 08:54:34 bluth k3s[11030]: time="2022-07-24T08:54:34+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-qbittorrent\", Name:\"qbittorrent-torrentudp\", UID:\"1d92e74f-b912-46f1-b780-c740f80bb9ca\", APIVersion:\"v1\", ResourceVersion:\"2504584\", FieldPath:\"\"}): type: 'Normal' reason: 'UpdatedIngressIP' LoadBalancer Ingress IP addresses updated: " Jul 24 08:54:34 bluth k3s[11030]: time="2022-07-24T08:54:34+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-qbittorrent\", Name:\"qbittorrent-torrentudp\", UID:\"1d92e74f-b912-46f1-b780-c740f80bb9ca\", APIVersion:\"v1\", ResourceVersion:\"3027754\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-qbittorrent/svclb-qbittorrent-torrentudp" Jul 24 08:54:34 bluth k3s[11030]: time="2022-07-24T08:54:34+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-qbittorrent\", Name:\"qbittorrent-torrentudp\", UID:\"1d92e74f-b912-46f1-b780-c740f80bb9ca\", APIVersion:\"v1\", ResourceVersion:\"3027754\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-qbittorrent/svclb-qbittorrent-torrentudp" Jul 24 08:54:34 bluth k3s[11030]: I0724 08:54:34.714028 11030 event.go:294] "Event occurred" object="ix-qbittorrent/svclb-qbittorrent-torrentudp" kind="DaemonSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulCreate" message="Created pod: svclb-qbittorrent-torrentudp-wjbmf" Jul 24 08:54:34 bluth k3s[11030]: I0724 08:54:34.719491 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:54:34 bluth k3s[11030]: time="2022-07-24T08:54:34+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-radarr\", Name:\"radarr\", UID:\"2ceba47d-fa27-462f-9816-af4352781795\", APIVersion:\"v1\", ResourceVersion:\"3027740\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-radarr/svclb-radarr" Jul 24 08:54:35 bluth k3s[11030]: I0724 08:54:35.037352 11030 scope.go:110] "RemoveContainer" containerID="996ab65b7608e8fb0d744ce68a7c7d875d830d4728c5bf75a0d8431536dc2e8f" Jul 24 08:54:35 bluth k3s[11030]: time="2022-07-24T08:54:35+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-esphome\", Name:\"esphome\", UID:\"9d7f07bb-0ed3-4a9b-b9b3-ce707e119669\", APIVersion:\"v1\", ResourceVersion:\"2334230\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-esphome/svclb-esphome" Jul 24 08:54:35 bluth k3s[11030]: time="2022-07-24T08:54:35+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-qbittorrent\", Name:\"qbittorrent-torrentudp\", UID:\"1d92e74f-b912-46f1-b780-c740f80bb9ca\", APIVersion:\"v1\", ResourceVersion:\"3027754\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-qbittorrent/svclb-qbittorrent-torrentudp" Jul 24 08:54:35 bluth k3s[11030]: I0724 08:54:35.776348 11030 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=51776aeb-f372-4d10-aa50-df5855b638e3 path="/var/lib/kubelet/pods/51776aeb-f372-4d10-aa50-df5855b638e3/volumes" Jul 24 08:54:35 bluth k3s[11030]: I0724 08:54:35.784856 11030 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=c69bd8a5-5c5f-4a26-9204-b7d4036137ec path="/var/lib/kubelet/pods/c69bd8a5-5c5f-4a26-9204-b7d4036137ec/volumes" Jul 24 08:54:35 bluth k3s[11030]: E0724 08:54:35.786180 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:54:37 bluth k3s[11030]: I0724 08:54:37.403356 27709 event.go:278] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ix-tdarr", Name:"svclb-tdarr-comm-lgh8b", UID:"209cc8e3-874d-4958-9a64-dd130dc956ee", APIVersion:"v1", ResourceVersion:"3027616", FieldPath:""}): type: 'Normal' reason: 'AddedInterface' Add eth0 [172.16.0.194/16] from ix-net Jul 24 08:54:37 bluth k3s[11030]: E0724 08:54:37.771943 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:54:39 bluth k3s[11030]: E0724 08:54:39.756987 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:54:41 bluth k3s[11030]: E0724 08:54:41.756754 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:54:42 bluth k3s[11030]: I0724 08:54:42.016307 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67x4q\" (UniqueName: \"kubernetes.io/projected/b1f2714d-382d-4638-83f4-2b2055748db3-kube-api-access-67x4q\") pod \"b1f2714d-382d-4638-83f4-2b2055748db3\" (UID: \"b1f2714d-382d-4638-83f4-2b2055748db3\") " Jul 24 08:54:42 bluth k3s[11030]: I0724 08:54:42.016402 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b1f2714d-382d-4638-83f4-2b2055748db3-config-volume\") pod \"b1f2714d-382d-4638-83f4-2b2055748db3\" (UID: \"b1f2714d-382d-4638-83f4-2b2055748db3\") " Jul 24 08:54:42 bluth k3s[11030]: I0724 08:54:42.016464 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"custom-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1f2714d-382d-4638-83f4-2b2055748db3-custom-config-volume\") pod \"b1f2714d-382d-4638-83f4-2b2055748db3\" (UID: \"b1f2714d-382d-4638-83f4-2b2055748db3\") " Jul 24 08:54:42 bluth k3s[11030]: W0724 08:54:42.017316 11030 empty_dir.go:517] Warning: Failed to clear quota on /var/lib/kubelet/pods/b1f2714d-382d-4638-83f4-2b2055748db3/volumes/kubernetes.io~configmap/config-volume: clearQuota called, but quotas disabled Jul 24 08:54:42 bluth k3s[11030]: W0724 08:54:42.017390 11030 empty_dir.go:517] Warning: Failed to clear quota on /var/lib/kubelet/pods/b1f2714d-382d-4638-83f4-2b2055748db3/volumes/kubernetes.io~configmap/custom-config-volume: clearQuota called, but quotas disabled Jul 24 08:54:42 bluth k3s[11030]: I0724 08:54:42.018588 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1f2714d-382d-4638-83f4-2b2055748db3-custom-config-volume" (OuterVolumeSpecName: "custom-config-volume") pod "b1f2714d-382d-4638-83f4-2b2055748db3" (UID: "b1f2714d-382d-4638-83f4-2b2055748db3"). InnerVolumeSpecName "custom-config-volume". PluginName "kubernetes.io/configmap", VolumeGidValue "" Jul 24 08:54:42 bluth k3s[11030]: I0724 08:54:42.018930 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1f2714d-382d-4638-83f4-2b2055748db3-config-volume" (OuterVolumeSpecName: "config-volume") pod "b1f2714d-382d-4638-83f4-2b2055748db3" (UID: "b1f2714d-382d-4638-83f4-2b2055748db3"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/configmap", VolumeGidValue "" Jul 24 08:54:42 bluth k3s[11030]: I0724 08:54:42.020244 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1f2714d-382d-4638-83f4-2b2055748db3-kube-api-access-67x4q" (OuterVolumeSpecName: "kube-api-access-67x4q") pod "b1f2714d-382d-4638-83f4-2b2055748db3" (UID: "b1f2714d-382d-4638-83f4-2b2055748db3"). InnerVolumeSpecName "kube-api-access-67x4q". PluginName "kubernetes.io/projected", VolumeGidValue "" Jul 24 08:54:42 bluth k3s[11030]: I0724 08:54:42.116879 11030 reconciler.go:300] "Volume detached for volume \"kube-api-access-67x4q\" (UniqueName: \"kubernetes.io/projected/b1f2714d-382d-4638-83f4-2b2055748db3-kube-api-access-67x4q\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:54:42 bluth k3s[11030]: I0724 08:54:42.116954 11030 reconciler.go:300] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b1f2714d-382d-4638-83f4-2b2055748db3-config-volume\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:54:42 bluth k3s[11030]: I0724 08:54:42.116989 11030 reconciler.go:300] "Volume detached for volume \"custom-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1f2714d-382d-4638-83f4-2b2055748db3-custom-config-volume\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:54:42 bluth k3s[11030]: I0724 08:54:42.240402 11030 scope.go:110] "RemoveContainer" containerID="cb842219d9539846ec5a654aaa10e3cdd7dc39127be57f7a670c77c09fb23b95" Jul 24 08:54:42 bluth k3s[11030]: I0724 08:54:42.719703 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"nfd-source-hooks\" (UniqueName: \"kubernetes.io/host-path/bb5993ab-6276-43d2-8017-a6fc01efb9a3-nfd-source-hooks\") pod \"bb5993ab-6276-43d2-8017-a6fc01efb9a3\" (UID: \"bb5993ab-6276-43d2-8017-a6fc01efb9a3\") " Jul 24 08:54:42 bluth k3s[11030]: I0724 08:54:42.719794 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"kubeletsockets\" (UniqueName: \"kubernetes.io/host-path/bb5993ab-6276-43d2-8017-a6fc01efb9a3-kubeletsockets\") pod \"bb5993ab-6276-43d2-8017-a6fc01efb9a3\" (UID: \"bb5993ab-6276-43d2-8017-a6fc01efb9a3\") " Jul 24 08:54:42 bluth k3s[11030]: I0724 08:54:42.719851 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"devfs\" (UniqueName: \"kubernetes.io/host-path/bb5993ab-6276-43d2-8017-a6fc01efb9a3-devfs\") pod \"bb5993ab-6276-43d2-8017-a6fc01efb9a3\" (UID: \"bb5993ab-6276-43d2-8017-a6fc01efb9a3\") " Jul 24 08:54:42 bluth k3s[11030]: I0724 08:54:42.719901 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"sysfs\" (UniqueName: \"kubernetes.io/host-path/bb5993ab-6276-43d2-8017-a6fc01efb9a3-sysfs\") pod \"bb5993ab-6276-43d2-8017-a6fc01efb9a3\" (UID: \"bb5993ab-6276-43d2-8017-a6fc01efb9a3\") " Jul 24 08:54:42 bluth k3s[11030]: I0724 08:54:42.719930 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/bb5993ab-6276-43d2-8017-a6fc01efb9a3-kubeletsockets" (OuterVolumeSpecName: "kubeletsockets") pod "bb5993ab-6276-43d2-8017-a6fc01efb9a3" (UID: "bb5993ab-6276-43d2-8017-a6fc01efb9a3"). InnerVolumeSpecName "kubeletsockets". PluginName "kubernetes.io/host-path", VolumeGidValue "" Jul 24 08:54:42 bluth k3s[11030]: I0724 08:54:42.719969 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smtsz\" (UniqueName: \"kubernetes.io/projected/bb5993ab-6276-43d2-8017-a6fc01efb9a3-kube-api-access-smtsz\") pod \"bb5993ab-6276-43d2-8017-a6fc01efb9a3\" (UID: \"bb5993ab-6276-43d2-8017-a6fc01efb9a3\") " Jul 24 08:54:42 bluth k3s[11030]: I0724 08:54:42.720017 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/bb5993ab-6276-43d2-8017-a6fc01efb9a3-sysfs" (OuterVolumeSpecName: "sysfs") pod "bb5993ab-6276-43d2-8017-a6fc01efb9a3" (UID: "bb5993ab-6276-43d2-8017-a6fc01efb9a3"). InnerVolumeSpecName "sysfs". PluginName "kubernetes.io/host-path", VolumeGidValue "" Jul 24 08:54:42 bluth k3s[11030]: I0724 08:54:42.720014 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/bb5993ab-6276-43d2-8017-a6fc01efb9a3-devfs" (OuterVolumeSpecName: "devfs") pod "bb5993ab-6276-43d2-8017-a6fc01efb9a3" (UID: "bb5993ab-6276-43d2-8017-a6fc01efb9a3"). InnerVolumeSpecName "devfs". PluginName "kubernetes.io/host-path", VolumeGidValue "" Jul 24 08:54:42 bluth k3s[11030]: I0724 08:54:42.720042 11030 reconciler.go:300] "Volume detached for volume \"kubeletsockets\" (UniqueName: \"kubernetes.io/host-path/bb5993ab-6276-43d2-8017-a6fc01efb9a3-kubeletsockets\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:54:42 bluth k3s[11030]: I0724 08:54:42.719932 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/bb5993ab-6276-43d2-8017-a6fc01efb9a3-nfd-source-hooks" (OuterVolumeSpecName: "nfd-source-hooks") pod "bb5993ab-6276-43d2-8017-a6fc01efb9a3" (UID: "bb5993ab-6276-43d2-8017-a6fc01efb9a3"). InnerVolumeSpecName "nfd-source-hooks". PluginName "kubernetes.io/host-path", VolumeGidValue "" Jul 24 08:54:42 bluth k3s[11030]: I0724 08:54:42.723466 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb5993ab-6276-43d2-8017-a6fc01efb9a3-kube-api-access-smtsz" (OuterVolumeSpecName: "kube-api-access-smtsz") pod "bb5993ab-6276-43d2-8017-a6fc01efb9a3" (UID: "bb5993ab-6276-43d2-8017-a6fc01efb9a3"). InnerVolumeSpecName "kube-api-access-smtsz". PluginName "kubernetes.io/projected", VolumeGidValue "" Jul 24 08:54:42 bluth k3s[11030]: I0724 08:54:42.820576 11030 reconciler.go:300] "Volume detached for volume \"kube-api-access-smtsz\" (UniqueName: \"kubernetes.io/projected/bb5993ab-6276-43d2-8017-a6fc01efb9a3-kube-api-access-smtsz\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:54:42 bluth k3s[11030]: I0724 08:54:42.820641 11030 reconciler.go:300] "Volume detached for volume \"devfs\" (UniqueName: \"kubernetes.io/host-path/bb5993ab-6276-43d2-8017-a6fc01efb9a3-devfs\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:54:42 bluth k3s[11030]: I0724 08:54:42.820670 11030 reconciler.go:300] "Volume detached for volume \"sysfs\" (UniqueName: \"kubernetes.io/host-path/bb5993ab-6276-43d2-8017-a6fc01efb9a3-sysfs\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:54:42 bluth k3s[11030]: I0724 08:54:42.820700 11030 reconciler.go:300] "Volume detached for volume \"nfd-source-hooks\" (UniqueName: \"kubernetes.io/host-path/bb5993ab-6276-43d2-8017-a6fc01efb9a3-nfd-source-hooks\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:54:43 bluth k3s[11030]: E0724 08:54:43.753793 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:54:45 bluth k3s[11030]: E0724 08:54:45.755973 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:54:47 bluth k3s[11030]: I0724 08:54:47.334719 11030 docker_sandbox.go:241] "Both sandbox container and checkpoint could not be found. Proceed without further sandbox information." podSandboxID="ae53aab9115f1640d86df436ceacb7f96dae2fa96c88a8e0d57d08aa26cc25b1" Jul 24 08:54:47 bluth k3s[11030]: I0724 08:54:47.335469 11030 cni.go:334] "CNI failed to retrieve network namespace path" err="Error: No such container: ae53aab9115f1640d86df436ceacb7f96dae2fa96c88a8e0d57d08aa26cc25b1" Jul 24 08:54:47 bluth k3s[11030]: E0724 08:54:47.411227 11030 cni.go:381] "Error deleting pod from network" err="Multus: [/]: error getting pod: resource name may not be empty" pod="" podSandboxID={Type:docker ID:ae53aab9115f1640d86df436ceacb7f96dae2fa96c88a8e0d57d08aa26cc25b1} podNetnsPath="" networkType="multus" networkName="multus-cni-network" Jul 24 08:54:47 bluth k3s[11030]: E0724 08:54:47.412903 11030 remote_runtime.go:245] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = networkPlugin cni failed to teardown pod \"_\" network: Multus: [/]: error getting pod: resource name may not be empty" podSandboxID="ae53aab9115f1640d86df436ceacb7f96dae2fa96c88a8e0d57d08aa26cc25b1" Jul 24 08:54:47 bluth k3s[11030]: E0724 08:54:47.412970 11030 kuberuntime_manager.go:1013] "Failed to stop sandbox" podSandboxID={Type:docker ID:ae53aab9115f1640d86df436ceacb7f96dae2fa96c88a8e0d57d08aa26cc25b1} Jul 24 08:54:47 bluth k3s[11030]: E0724 08:54:47.413040 11030 kubelet.go:1799] failed to "KillPodSandbox" for "c41628dc-0533-406b-ac00-3fbf6cd22284" with KillPodSandboxError: "rpc error: code = Unknown desc = networkPlugin cni failed to teardown pod \"_\" network: Multus: [/]: error getting pod: resource name may not be empty" Jul 24 08:54:47 bluth k3s[11030]: E0724 08:54:47.413089 11030 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"KillPodSandbox\" for \"c41628dc-0533-406b-ac00-3fbf6cd22284\" with KillPodSandboxError: \"rpc error: code = Unknown desc = networkPlugin cni failed to teardown pod \\\"_\\\" network: Multus: [/]: error getting pod: resource name may not be empty\"" pod="ix-esphome/svclb-esphome-q4lrj" podUID=c41628dc-0533-406b-ac00-3fbf6cd22284 Jul 24 08:54:47 bluth k3s[11030]: E0724 08:54:47.778984 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:54:49 bluth k3s[11030]: E0724 08:54:49.769518 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:54:49 bluth k3s[11030]: time="2022-07-24T08:54:49+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-sonarr\", Name:\"sonarr\", UID:\"b24571a2-a018-4716-ab7d-d6ab31e66e44\", APIVersion:\"v1\", ResourceVersion:\"2334145\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-sonarr/svclb-sonarr" Jul 24 08:54:49 bluth k3s[11030]: W0724 08:54:49.997541 11030 endpointslice_controller.go:306] Error syncing endpoint slices for service "ix-plex/plex-udp", retrying. Error: EndpointSlice informer cache is out of date Jul 24 08:54:50 bluth k3s[11030]: W0724 08:54:50.001162 11030 endpointslice_controller.go:306] Error syncing endpoint slices for service "ix-plex/plex-tcp-cluster-ip", retrying. Error: EndpointSlice informer cache is out of date Jul 24 08:54:50 bluth k3s[11030]: I0724 08:54:50.006393 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="5bee41d5ba349868a37c5cfcd76020fda6fad2615c08c4e6b0858e801d96c273" Jul 24 08:54:50 bluth k3s[11030]: I0724 08:54:50.988924 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:54:50 bluth k3s[11030]: I0724 08:54:50.994770 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:54:50 bluth k3s[11030]: I0724 08:54:50.995302 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:54:50 bluth k3s[11030]: I0724 08:54:50.997887 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:54:50 bluth k3s[11030]: I0724 08:54:50.998011 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:54:50 bluth k3s[11030]: I0724 08:54:50.998132 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:54:51 bluth k3s[11030]: I0724 08:54:51.004837 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:54:51 bluth k3s[11030]: I0724 08:54:51.005013 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:54:51 bluth k3s[11030]: I0724 08:54:51.058100 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="31da4fd205a212d77f17853ba8032143c9e16e19fd5de6fa671faa2f74206994" Jul 24 08:54:51 bluth k3s[11030]: E0724 08:54:51.148695 11030 kuberuntime_container.go:598] "PreStop hook failed" err="command '/bin/sh -c rm -rf /registration/zfs-localpv /registration/zfs-localpv-reg.sock' exited with 126: " pod="kube-system/openebs-zfs-node-bf7kk" podUID=d312e744-054d-4a0d-8f39-46a83c98c808 containerName="csi-node-driver-registrar" containerID="docker://caecd739834fb496e4f85acb4bbf6596b6d5f784a17abf9b90bd230218f227f3" Jul 24 08:54:51 bluth k3s[11030]: I0724 08:54:51.756564 11030 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=495a78dd-7609-44be-bb7e-7036e65c860b path="/var/lib/kubelet/pods/495a78dd-7609-44be-bb7e-7036e65c860b/volumes" Jul 24 08:54:51 bluth k3s[11030]: I0724 08:54:51.758729 11030 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=b1f2714d-382d-4638-83f4-2b2055748db3 path="/var/lib/kubelet/pods/b1f2714d-382d-4638-83f4-2b2055748db3/volumes" Jul 24 08:54:51 bluth k3s[11030]: E0724 08:54:51.759609 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:54:52 bluth k3s[11030]: I0724 08:54:52.152400 11030 request.go:665] Waited for 1.145879345s due to client-side throttling, not priority and fairness, request: GET:https://127.0.0.1:6443/api/v1/namespaces/ix-sonarr/configmaps?fieldSelector=metadata.name%3Dkube-root-ca.crt&limit=500&resourceVersion=0 Jul 24 08:54:52 bluth k3s[11030]: I0724 08:54:52.778703 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e\") pod \"esphome-658957fddc-6z2jm\" (UID: \"19254d20-7ad1-49a1-b408-f6e4213e5b4e\") " pod="ix-esphome/esphome-658957fddc-6z2jm" Jul 24 08:54:52 bluth k3s[11030]: E0724 08:54:52.881033 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:54:53.380991846 +0100 IST m=+219.868848779 (durationBeforeRetry 500ms). Error: MountVolume.MountDevice failed for volume "pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e") pod "esphome-658957fddc-6z2jm" (UID: "19254d20-7ad1-49a1-b408-f6e4213e5b4e") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:54:53 bluth k3s[11030]: E0724 08:54:53.384527 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:54:54.38451288 +0100 IST m=+220.872369774 (durationBeforeRetry 1s). Error: MountVolume.MountDevice failed for volume "pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e") pod "esphome-658957fddc-6z2jm" (UID: "19254d20-7ad1-49a1-b408-f6e4213e5b4e") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:54:53 bluth k3s[11030]: I0724 08:54:53.585422 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-06521873-c179-499e-9079-194ec6618fdd\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-06521873-c179-499e-9079-194ec6618fdd\") pod \"esphome-658957fddc-6z2jm\" (UID: \"19254d20-7ad1-49a1-b408-f6e4213e5b4e\") " pod="ix-esphome/esphome-658957fddc-6z2jm" Jul 24 08:54:53 bluth k3s[11030]: I0724 08:54:53.585525 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"shared\" (UniqueName: \"kubernetes.io/empty-dir/19254d20-7ad1-49a1-b408-f6e4213e5b4e-shared\") pod \"esphome-658957fddc-6z2jm\" (UID: \"19254d20-7ad1-49a1-b408-f6e4213e5b4e\") " pod="ix-esphome/esphome-658957fddc-6z2jm" Jul 24 08:54:53 bluth k3s[11030]: I0724 08:54:53.585602 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ix-host-path-home-assistant-1\" (UniqueName: \"kubernetes.io/host-path/3f2c328d-b981-469a-b3d6-2850e53c7d15-ix-host-path-home-assistant-1\") pod \"home-assistant-ix-chart-54bf5d89c8-5rpjh\" (UID: \"3f2c328d-b981-469a-b3d6-2850e53c7d15\") " pod="ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-5rpjh" Jul 24 08:54:53 bluth k3s[11030]: I0724 08:54:53.585742 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ix-host-path-home-assistant-0\" (UniqueName: \"kubernetes.io/host-path/3f2c328d-b981-469a-b3d6-2850e53c7d15-ix-host-path-home-assistant-0\") pod \"home-assistant-ix-chart-54bf5d89c8-5rpjh\" (UID: \"3f2c328d-b981-469a-b3d6-2850e53c7d15\") " pod="ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-5rpjh" Jul 24 08:54:53 bluth k3s[11030]: I0724 08:54:53.585831 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq65c\" (UniqueName: \"kubernetes.io/projected/19254d20-7ad1-49a1-b408-f6e4213e5b4e-kube-api-access-pq65c\") pod \"esphome-658957fddc-6z2jm\" (UID: \"19254d20-7ad1-49a1-b408-f6e4213e5b4e\") " pod="ix-esphome/esphome-658957fddc-6z2jm" Jul 24 08:54:53 bluth k3s[11030]: I0724 08:54:53.585897 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"varlogs\" (UniqueName: \"kubernetes.io/empty-dir/19254d20-7ad1-49a1-b408-f6e4213e5b4e-varlogs\") pod \"esphome-658957fddc-6z2jm\" (UID: \"19254d20-7ad1-49a1-b408-f6e4213e5b4e\") " pod="ix-esphome/esphome-658957fddc-6z2jm" Jul 24 08:54:53 bluth k3s[11030]: I0724 08:54:53.585961 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwnwp\" (UniqueName: \"kubernetes.io/projected/3da0644e-8d9b-4f63-b8dc-0199c4f76027-kube-api-access-bwnwp\") pod \"cloudflared-ix-chart-687fff6d55-vzff2\" (UID: \"3da0644e-8d9b-4f63-b8dc-0199c4f76027\") " pod="ix-cloudflared/cloudflared-ix-chart-687fff6d55-vzff2" Jul 24 08:54:53 bluth k3s[11030]: I0724 08:54:53.586021 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp\" (UniqueName: \"kubernetes.io/empty-dir/19254d20-7ad1-49a1-b408-f6e4213e5b4e-temp\") pod \"esphome-658957fddc-6z2jm\" (UID: \"19254d20-7ad1-49a1-b408-f6e4213e5b4e\") " pod="ix-esphome/esphome-658957fddc-6z2jm" Jul 24 08:54:53 bluth k3s[11030]: I0724 08:54:53.586082 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhsff\" (UniqueName: \"kubernetes.io/projected/3f2c328d-b981-469a-b3d6-2850e53c7d15-kube-api-access-fhsff\") pod \"home-assistant-ix-chart-54bf5d89c8-5rpjh\" (UID: \"3f2c328d-b981-469a-b3d6-2850e53c7d15\") " pod="ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-5rpjh" Jul 24 08:54:53 bluth k3s[11030]: E0724 08:54:53.687927 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-06521873-c179-499e-9079-194ec6618fdd podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:54:54.187886996 +0100 IST m=+220.675743931 (durationBeforeRetry 500ms). Error: MountVolume.MountDevice failed for volume "pvc-06521873-c179-499e-9079-194ec6618fdd" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-06521873-c179-499e-9079-194ec6618fdd") pod "esphome-658957fddc-6z2jm" (UID: "19254d20-7ad1-49a1-b408-f6e4213e5b4e") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:54:53 bluth k3s[11030]: E0724 08:54:53.755079 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:54:54 bluth k3s[11030]: E0724 08:54:54.190998 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-06521873-c179-499e-9079-194ec6618fdd podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:54:55.190964073 +0100 IST m=+221.678820998 (durationBeforeRetry 1s). Error: MountVolume.MountDevice failed for volume "pvc-06521873-c179-499e-9079-194ec6618fdd" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-06521873-c179-499e-9079-194ec6618fdd") pod "esphome-658957fddc-6z2jm" (UID: "19254d20-7ad1-49a1-b408-f6e4213e5b4e") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:54:54 bluth k3s[11030]: E0724 08:54:54.392311 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:54:56.39227728 +0100 IST m=+222.880134205 (durationBeforeRetry 2s). Error: MountVolume.MountDevice failed for volume "pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e") pod "esphome-658957fddc-6z2jm" (UID: "19254d20-7ad1-49a1-b408-f6e4213e5b4e") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:54:54 bluth k3s[11030]: I0724 08:54:54.997006 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274\") pod \"prowlarr-5bcc99cc47-v86hm\" (UID: \"5d1f2955-de8a-4e25-b1f8-e607463ed369\") " pod="ix-prowlarr/prowlarr-5bcc99cc47-v86hm" Jul 24 08:54:54 bluth k3s[11030]: I0724 08:54:54.997116 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"shared\" (UniqueName: \"kubernetes.io/empty-dir/5d1f2955-de8a-4e25-b1f8-e607463ed369-shared\") pod \"prowlarr-5bcc99cc47-v86hm\" (UID: \"5d1f2955-de8a-4e25-b1f8-e607463ed369\") " pod="ix-prowlarr/prowlarr-5bcc99cc47-v86hm" Jul 24 08:54:54 bluth k3s[11030]: I0724 08:54:54.997323 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp\" (UniqueName: \"kubernetes.io/empty-dir/5d1f2955-de8a-4e25-b1f8-e607463ed369-temp\") pod \"prowlarr-5bcc99cc47-v86hm\" (UID: \"5d1f2955-de8a-4e25-b1f8-e607463ed369\") " pod="ix-prowlarr/prowlarr-5bcc99cc47-v86hm" Jul 24 08:54:54 bluth k3s[11030]: I0724 08:54:54.997416 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"varlogs\" (UniqueName: \"kubernetes.io/empty-dir/5d1f2955-de8a-4e25-b1f8-e607463ed369-varlogs\") pod \"prowlarr-5bcc99cc47-v86hm\" (UID: \"5d1f2955-de8a-4e25-b1f8-e607463ed369\") " pod="ix-prowlarr/prowlarr-5bcc99cc47-v86hm" Jul 24 08:54:54 bluth k3s[11030]: I0724 08:54:54.997519 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8v9k\" (UniqueName: \"kubernetes.io/projected/5d1f2955-de8a-4e25-b1f8-e607463ed369-kube-api-access-c8v9k\") pod \"prowlarr-5bcc99cc47-v86hm\" (UID: \"5d1f2955-de8a-4e25-b1f8-e607463ed369\") " pod="ix-prowlarr/prowlarr-5bcc99cc47-v86hm" Jul 24 08:54:55 bluth k3s[11030]: E0724 08:54:55.099296 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:54:55.599261614 +0100 IST m=+222.087118558 (durationBeforeRetry 500ms). Error: MountVolume.MountDevice failed for volume "pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274") pod "prowlarr-5bcc99cc47-v86hm" (UID: "5d1f2955-de8a-4e25-b1f8-e607463ed369") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:54:55 bluth k3s[11030]: E0724 08:54:55.200058 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-06521873-c179-499e-9079-194ec6618fdd podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:54:57.200018405 +0100 IST m=+223.687875338 (durationBeforeRetry 2s). Error: MountVolume.MountDevice failed for volume "pvc-06521873-c179-499e-9079-194ec6618fdd" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-06521873-c179-499e-9079-194ec6618fdd") pod "esphome-658957fddc-6z2jm" (UID: "19254d20-7ad1-49a1-b408-f6e4213e5b4e") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:54:55 bluth k3s[11030]: E0724 08:54:55.602306 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:54:56.602297012 +0100 IST m=+223.090153892 (durationBeforeRetry 1s). Error: MountVolume.MountDevice failed for volume "pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274") pod "prowlarr-5bcc99cc47-v86hm" (UID: "5d1f2955-de8a-4e25-b1f8-e607463ed369") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:54:55 bluth k3s[11030]: I0724 08:54:55.673806 11030 trace.go:205] Trace[1268991238]: "GuaranteedUpdate etcd3" type:*core.Event (24-Jul-2022 08:54:54.193) (total time: 1480ms): Jul 24 08:54:55 bluth k3s[11030]: Trace[1268991238]: ---"Transaction committed" 1476ms (08:54:55.673) Jul 24 08:54:55 bluth k3s[11030]: Trace[1268991238]: [1.480002746s] [1.480002746s] END Jul 24 08:54:55 bluth k3s[11030]: I0724 08:54:55.674098 11030 trace.go:205] Trace[1397597835]: "Patch" url:/api/v1/namespaces/ix-esphome/events/esphome-658957fddc-6z2jm.1704b4bea53795c8,user-agent:k3s/v1.23.5+k3s (linux/amd64) kubernetes/fbfa51e,audit-id:55f1f8ae-280f-4992-9418-bb5b2bd0f4d2,client:127.0.0.1,accept:application/vnd.kubernetes.protobuf,application/json,protocol:HTTP/2.0 (24-Jul-2022 08:54:54.193) (total time: 1480ms): Jul 24 08:54:55 bluth k3s[11030]: Trace[1397597835]: ---"Object stored in database" 1476ms (08:54:55.673) Jul 24 08:54:55 bluth k3s[11030]: Trace[1397597835]: [1.480430253s] [1.480430253s] END Jul 24 08:54:55 bluth k3s[11030]: E0724 08:54:55.751131 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:54:56 bluth k3s[11030]: I0724 08:54:56.005047 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ix-host-path-status-html-0\" (UniqueName: \"kubernetes.io/host-path/290c61bb-9f19-47b9-b548-312e738a0bf1-ix-host-path-status-html-0\") pod \"status-html-ix-chart-765b966694-c4nwz\" (UID: \"290c61bb-9f19-47b9-b548-312e738a0bf1\") " pod="ix-status-html/status-html-ix-chart-765b966694-c4nwz" Jul 24 08:54:56 bluth k3s[11030]: I0724 08:54:56.005204 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-58562e12-4adc-4e83-8131-848f1b39bf00\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-58562e12-4adc-4e83-8131-848f1b39bf00\") pod \"qbittorrent-6d7cf9f7cd-xrqwj\" (UID: \"4d43a190-0b41-4f84-a84d-ce295694fcb5\") " pod="ix-qbittorrent/qbittorrent-6d7cf9f7cd-xrqwj" Jul 24 08:54:56 bluth k3s[11030]: I0724 08:54:56.005271 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp\" (UniqueName: \"kubernetes.io/empty-dir/4d43a190-0b41-4f84-a84d-ce295694fcb5-temp\") pod \"qbittorrent-6d7cf9f7cd-xrqwj\" (UID: \"4d43a190-0b41-4f84-a84d-ce295694fcb5\") " pod="ix-qbittorrent/qbittorrent-6d7cf9f7cd-xrqwj" Jul 24 08:54:56 bluth k3s[11030]: I0724 08:54:56.005343 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpklx\" (UniqueName: \"kubernetes.io/projected/290c61bb-9f19-47b9-b548-312e738a0bf1-kube-api-access-cpklx\") pod \"status-html-ix-chart-765b966694-c4nwz\" (UID: \"290c61bb-9f19-47b9-b548-312e738a0bf1\") " pod="ix-status-html/status-html-ix-chart-765b966694-c4nwz" Jul 24 08:54:56 bluth k3s[11030]: I0724 08:54:56.005470 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"varlogs\" (UniqueName: \"kubernetes.io/empty-dir/4d43a190-0b41-4f84-a84d-ce295694fcb5-varlogs\") pod \"qbittorrent-6d7cf9f7cd-xrqwj\" (UID: \"4d43a190-0b41-4f84-a84d-ce295694fcb5\") " pod="ix-qbittorrent/qbittorrent-6d7cf9f7cd-xrqwj" Jul 24 08:54:56 bluth k3s[11030]: I0724 08:54:56.005566 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"shared\" (UniqueName: \"kubernetes.io/empty-dir/4d43a190-0b41-4f84-a84d-ce295694fcb5-shared\") pod \"qbittorrent-6d7cf9f7cd-xrqwj\" (UID: \"4d43a190-0b41-4f84-a84d-ce295694fcb5\") " pod="ix-qbittorrent/qbittorrent-6d7cf9f7cd-xrqwj" Jul 24 08:54:56 bluth k3s[11030]: I0724 08:54:56.005679 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"list-0\" (UniqueName: \"kubernetes.io/host-path/4d43a190-0b41-4f84-a84d-ce295694fcb5-list-0\") pod \"qbittorrent-6d7cf9f7cd-xrqwj\" (UID: \"4d43a190-0b41-4f84-a84d-ce295694fcb5\") " pod="ix-qbittorrent/qbittorrent-6d7cf9f7cd-xrqwj" Jul 24 08:54:56 bluth k3s[11030]: I0724 08:54:56.005744 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df98j\" (UniqueName: \"kubernetes.io/projected/4d43a190-0b41-4f84-a84d-ce295694fcb5-kube-api-access-df98j\") pod \"qbittorrent-6d7cf9f7cd-xrqwj\" (UID: \"4d43a190-0b41-4f84-a84d-ce295694fcb5\") " pod="ix-qbittorrent/qbittorrent-6d7cf9f7cd-xrqwj" Jul 24 08:54:56 bluth k3s[11030]: E0724 08:54:56.107885 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-58562e12-4adc-4e83-8131-848f1b39bf00 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:54:56.607849798 +0100 IST m=+223.095706720 (durationBeforeRetry 500ms). Error: MountVolume.MountDevice failed for volume "pvc-58562e12-4adc-4e83-8131-848f1b39bf00" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-58562e12-4adc-4e83-8131-848f1b39bf00") pod "qbittorrent-6d7cf9f7cd-xrqwj" (UID: "4d43a190-0b41-4f84-a84d-ce295694fcb5") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:54:56 bluth k3s[11030]: E0724 08:54:56.409968 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:55:00.409932792 +0100 IST m=+226.897789736 (durationBeforeRetry 4s). Error: MountVolume.MountDevice failed for volume "pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e") pod "esphome-658957fddc-6z2jm" (UID: "19254d20-7ad1-49a1-b408-f6e4213e5b4e") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:54:56 bluth k3s[11030]: E0724 08:54:56.610960 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-58562e12-4adc-4e83-8131-848f1b39bf00 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:54:57.610949231 +0100 IST m=+224.098806117 (durationBeforeRetry 1s). Error: MountVolume.MountDevice failed for volume "pvc-58562e12-4adc-4e83-8131-848f1b39bf00" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-58562e12-4adc-4e83-8131-848f1b39bf00") pod "qbittorrent-6d7cf9f7cd-xrqwj" (UID: "4d43a190-0b41-4f84-a84d-ce295694fcb5") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:54:56 bluth k3s[11030]: E0724 08:54:56.610978 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:54:58.610972789 +0100 IST m=+225.098829668 (durationBeforeRetry 2s). Error: MountVolume.MountDevice failed for volume "pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274") pod "prowlarr-5bcc99cc47-v86hm" (UID: "5d1f2955-de8a-4e25-b1f8-e607463ed369") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:54:57 bluth k3s[11030]: E0724 08:54:57.041223 11030 authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, service account token has been invalidated]" Jul 24 08:54:57 bluth k3s[11030]: I0724 08:54:57.216987 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-195f1a32-9c93-4737-983a-8dacee32e453\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-195f1a32-9c93-4737-983a-8dacee32e453\") pod \"radarr-7486f78d7f-ntt44\" (UID: \"9ab9d8fe-e669-40bb-a0ce-7f7a4e5a69b5\") " pod="ix-radarr/radarr-7486f78d7f-ntt44" Jul 24 08:54:57 bluth k3s[11030]: I0724 08:54:57.217114 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"list-0\" (UniqueName: \"kubernetes.io/host-path/9ab9d8fe-e669-40bb-a0ce-7f7a4e5a69b5-list-0\") pod \"radarr-7486f78d7f-ntt44\" (UID: \"9ab9d8fe-e669-40bb-a0ce-7f7a4e5a69b5\") " pod="ix-radarr/radarr-7486f78d7f-ntt44" Jul 24 08:54:57 bluth k3s[11030]: I0724 08:54:57.217180 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp\" (UniqueName: \"kubernetes.io/empty-dir/9ab9d8fe-e669-40bb-a0ce-7f7a4e5a69b5-temp\") pod \"radarr-7486f78d7f-ntt44\" (UID: \"9ab9d8fe-e669-40bb-a0ce-7f7a4e5a69b5\") " pod="ix-radarr/radarr-7486f78d7f-ntt44" Jul 24 08:54:57 bluth k3s[11030]: I0724 08:54:57.217237 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"shared\" (UniqueName: \"kubernetes.io/empty-dir/9ab9d8fe-e669-40bb-a0ce-7f7a4e5a69b5-shared\") pod \"radarr-7486f78d7f-ntt44\" (UID: \"9ab9d8fe-e669-40bb-a0ce-7f7a4e5a69b5\") " pod="ix-radarr/radarr-7486f78d7f-ntt44" Jul 24 08:54:57 bluth k3s[11030]: I0724 08:54:57.217294 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"varlogs\" (UniqueName: \"kubernetes.io/empty-dir/9ab9d8fe-e669-40bb-a0ce-7f7a4e5a69b5-varlogs\") pod \"radarr-7486f78d7f-ntt44\" (UID: \"9ab9d8fe-e669-40bb-a0ce-7f7a4e5a69b5\") " pod="ix-radarr/radarr-7486f78d7f-ntt44" Jul 24 08:54:57 bluth k3s[11030]: I0724 08:54:57.217352 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wqqq\" (UniqueName: \"kubernetes.io/projected/9ab9d8fe-e669-40bb-a0ce-7f7a4e5a69b5-kube-api-access-4wqqq\") pod \"radarr-7486f78d7f-ntt44\" (UID: \"9ab9d8fe-e669-40bb-a0ce-7f7a4e5a69b5\") " pod="ix-radarr/radarr-7486f78d7f-ntt44" Jul 24 08:54:57 bluth k3s[11030]: E0724 08:54:57.217747 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-06521873-c179-499e-9079-194ec6618fdd podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:55:01.217713351 +0100 IST m=+227.705570295 (durationBeforeRetry 4s). Error: MountVolume.MountDevice failed for volume "pvc-06521873-c179-499e-9079-194ec6618fdd" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-06521873-c179-499e-9079-194ec6618fdd") pod "esphome-658957fddc-6z2jm" (UID: "19254d20-7ad1-49a1-b408-f6e4213e5b4e") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:54:57 bluth k3s[11030]: I0724 08:54:57.246143 11030 scope.go:110] "RemoveContainer" containerID="b36856ea81b4de326bf77efba4b7623b64e883bdfb64f26e1b89ece06ae9bced" Jul 24 08:54:57 bluth k3s[11030]: E0724 08:54:57.319862 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-195f1a32-9c93-4737-983a-8dacee32e453 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:54:57.819825976 +0100 IST m=+224.307682899 (durationBeforeRetry 500ms). Error: MountVolume.MountDevice failed for volume "pvc-195f1a32-9c93-4737-983a-8dacee32e453" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-195f1a32-9c93-4737-983a-8dacee32e453") pod "radarr-7486f78d7f-ntt44" (UID: "9ab9d8fe-e669-40bb-a0ce-7f7a4e5a69b5") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:54:57 bluth k3s[11030]: E0724 08:54:57.622181 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-58562e12-4adc-4e83-8131-848f1b39bf00 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:54:59.622143137 +0100 IST m=+226.110000066 (durationBeforeRetry 2s). Error: MountVolume.MountDevice failed for volume "pvc-58562e12-4adc-4e83-8131-848f1b39bf00" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-58562e12-4adc-4e83-8131-848f1b39bf00") pod "qbittorrent-6d7cf9f7cd-xrqwj" (UID: "4d43a190-0b41-4f84-a84d-ce295694fcb5") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:54:57 bluth k3s[11030]: E0724 08:54:57.778429 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:54:57 bluth k3s[11030]: E0724 08:54:57.824335 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-195f1a32-9c93-4737-983a-8dacee32e453 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:54:58.824298064 +0100 IST m=+225.312155006 (durationBeforeRetry 1s). Error: MountVolume.MountDevice failed for volume "pvc-195f1a32-9c93-4737-983a-8dacee32e453" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-195f1a32-9c93-4737-983a-8dacee32e453") pod "radarr-7486f78d7f-ntt44" (UID: "9ab9d8fe-e669-40bb-a0ce-7f7a4e5a69b5") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:54:58 bluth k3s[11030]: I0724 08:54:58.226675 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8kjp\" (UniqueName: \"kubernetes.io/projected/1a1846b0-3361-4f4b-a1cd-d5f958d6a7ac-kube-api-access-m8kjp\") pod \"sonarr-867d7dd548-fw6z4\" (UID: \"1a1846b0-3361-4f4b-a1cd-d5f958d6a7ac\") " pod="ix-sonarr/sonarr-867d7dd548-fw6z4" Jul 24 08:54:58 bluth k3s[11030]: I0724 08:54:58.226833 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"shared\" (UniqueName: \"kubernetes.io/empty-dir/1a1846b0-3361-4f4b-a1cd-d5f958d6a7ac-shared\") pod \"sonarr-867d7dd548-fw6z4\" (UID: \"1a1846b0-3361-4f4b-a1cd-d5f958d6a7ac\") " pod="ix-sonarr/sonarr-867d7dd548-fw6z4" Jul 24 08:54:58 bluth k3s[11030]: I0724 08:54:58.226900 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"varlogs\" (UniqueName: \"kubernetes.io/empty-dir/1a1846b0-3361-4f4b-a1cd-d5f958d6a7ac-varlogs\") pod \"sonarr-867d7dd548-fw6z4\" (UID: \"1a1846b0-3361-4f4b-a1cd-d5f958d6a7ac\") " pod="ix-sonarr/sonarr-867d7dd548-fw6z4" Jul 24 08:54:58 bluth k3s[11030]: I0724 08:54:58.227038 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4\") pod \"sonarr-867d7dd548-fw6z4\" (UID: \"1a1846b0-3361-4f4b-a1cd-d5f958d6a7ac\") " pod="ix-sonarr/sonarr-867d7dd548-fw6z4" Jul 24 08:54:58 bluth k3s[11030]: I0724 08:54:58.227209 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp\" (UniqueName: \"kubernetes.io/empty-dir/1a1846b0-3361-4f4b-a1cd-d5f958d6a7ac-temp\") pod \"sonarr-867d7dd548-fw6z4\" (UID: \"1a1846b0-3361-4f4b-a1cd-d5f958d6a7ac\") " pod="ix-sonarr/sonarr-867d7dd548-fw6z4" Jul 24 08:54:58 bluth k3s[11030]: I0724 08:54:58.227301 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"list-0\" (UniqueName: \"kubernetes.io/host-path/1a1846b0-3361-4f4b-a1cd-d5f958d6a7ac-list-0\") pod \"sonarr-867d7dd548-fw6z4\" (UID: \"1a1846b0-3361-4f4b-a1cd-d5f958d6a7ac\") " pod="ix-sonarr/sonarr-867d7dd548-fw6z4" Jul 24 08:54:58 bluth k3s[11030]: E0724 08:54:58.329253 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:54:58.82921601 +0100 IST m=+225.317072934 (durationBeforeRetry 500ms). Error: MountVolume.MountDevice failed for volume "pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4") pod "sonarr-867d7dd548-fw6z4" (UID: "1a1846b0-3361-4f4b-a1cd-d5f958d6a7ac") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:54:58 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T08:54:58.388+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 08:54:58 bluth k3s[11030]: E0724 08:54:58.630785 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:55:02.630751149 +0100 IST m=+229.118608074 (durationBeforeRetry 4s). Error: MountVolume.MountDevice failed for volume "pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274") pod "prowlarr-5bcc99cc47-v86hm" (UID: "5d1f2955-de8a-4e25-b1f8-e607463ed369") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:54:58 bluth k3s[11030]: E0724 08:54:58.832423 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:54:59.83238996 +0100 IST m=+226.320246890 (durationBeforeRetry 1s). Error: MountVolume.MountDevice failed for volume "pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4") pod "sonarr-867d7dd548-fw6z4" (UID: "1a1846b0-3361-4f4b-a1cd-d5f958d6a7ac") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:54:58 bluth k3s[11030]: E0724 08:54:58.832491 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-195f1a32-9c93-4737-983a-8dacee32e453 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:55:00.832468862 +0100 IST m=+227.320325788 (durationBeforeRetry 2s). Error: MountVolume.MountDevice failed for volume "pvc-195f1a32-9c93-4737-983a-8dacee32e453" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-195f1a32-9c93-4737-983a-8dacee32e453") pod "radarr-7486f78d7f-ntt44" (UID: "9ab9d8fe-e669-40bb-a0ce-7f7a4e5a69b5") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:54:59 bluth k3s[11030]: I0724 08:54:59.398846 11030 event.go:294] "Event occurred" object="kube-system/intel-gpu-plugin" kind="DaemonSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulCreate" message="Created pod: intel-gpu-plugin-9t629" Jul 24 08:54:59 bluth k3s[11030]: I0724 08:54:59.412270 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:54:59 bluth k3s[11030]: I0724 08:54:59.539389 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sysfs\" (UniqueName: \"kubernetes.io/host-path/a93d958f-5cba-4480-8ca8-61ca64464a2b-sysfs\") pod \"intel-gpu-plugin-9t629\" (UID: \"a93d958f-5cba-4480-8ca8-61ca64464a2b\") " pod="kube-system/intel-gpu-plugin-9t629" Jul 24 08:54:59 bluth k3s[11030]: I0724 08:54:59.539435 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8bfc\" (UniqueName: \"kubernetes.io/projected/a93d958f-5cba-4480-8ca8-61ca64464a2b-kube-api-access-c8bfc\") pod \"intel-gpu-plugin-9t629\" (UID: \"a93d958f-5cba-4480-8ca8-61ca64464a2b\") " pod="kube-system/intel-gpu-plugin-9t629" Jul 24 08:54:59 bluth k3s[11030]: I0724 08:54:59.539452 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"devfs\" (UniqueName: \"kubernetes.io/host-path/a93d958f-5cba-4480-8ca8-61ca64464a2b-devfs\") pod \"intel-gpu-plugin-9t629\" (UID: \"a93d958f-5cba-4480-8ca8-61ca64464a2b\") " pod="kube-system/intel-gpu-plugin-9t629" Jul 24 08:54:59 bluth k3s[11030]: I0724 08:54:59.539532 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubeletsockets\" (UniqueName: \"kubernetes.io/host-path/a93d958f-5cba-4480-8ca8-61ca64464a2b-kubeletsockets\") pod \"intel-gpu-plugin-9t629\" (UID: \"a93d958f-5cba-4480-8ca8-61ca64464a2b\") " pod="kube-system/intel-gpu-plugin-9t629" Jul 24 08:54:59 bluth k3s[11030]: I0724 08:54:59.539554 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nfd-source-hooks\" (UniqueName: \"kubernetes.io/host-path/a93d958f-5cba-4480-8ca8-61ca64464a2b-nfd-source-hooks\") pod \"intel-gpu-plugin-9t629\" (UID: \"a93d958f-5cba-4480-8ca8-61ca64464a2b\") " pod="kube-system/intel-gpu-plugin-9t629" Jul 24 08:54:59 bluth k3s[11030]: E0724 08:54:59.641460 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-58562e12-4adc-4e83-8131-848f1b39bf00 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:55:03.641435721 +0100 IST m=+230.129292627 (durationBeforeRetry 4s). Error: MountVolume.MountDevice failed for volume "pvc-58562e12-4adc-4e83-8131-848f1b39bf00" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-58562e12-4adc-4e83-8131-848f1b39bf00") pod "qbittorrent-6d7cf9f7cd-xrqwj" (UID: "4d43a190-0b41-4f84-a84d-ce295694fcb5") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:54:59 bluth k3s[11030]: I0724 08:54:59.757262 11030 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=bb5993ab-6276-43d2-8017-a6fc01efb9a3 path="/var/lib/kubelet/pods/bb5993ab-6276-43d2-8017-a6fc01efb9a3/volumes" Jul 24 08:54:59 bluth k3s[11030]: E0724 08:54:59.758914 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:54:59 bluth k3s[11030]: time="2022-07-24T08:54:59+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-esphome\", Name:\"esphome\", UID:\"9d7f07bb-0ed3-4a9b-b9b3-ce707e119669\", APIVersion:\"v1\", ResourceVersion:\"2334230\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-esphome/svclb-esphome" Jul 24 08:54:59 bluth k3s[11030]: time="2022-07-24T08:54:59+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-esphome\", Name:\"esphome\", UID:\"9d7f07bb-0ed3-4a9b-b9b3-ce707e119669\", APIVersion:\"v1\", ResourceVersion:\"2334230\", FieldPath:\"\"}): type: 'Normal' reason: 'UpdatedIngressIP' LoadBalancer Ingress IP addresses updated: " Jul 24 08:54:59 bluth k3s[11030]: time="2022-07-24T08:54:59+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-esphome\", Name:\"esphome\", UID:\"9d7f07bb-0ed3-4a9b-b9b3-ce707e119669\", APIVersion:\"v1\", ResourceVersion:\"3027860\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-esphome/svclb-esphome" Jul 24 08:54:59 bluth k3s[11030]: E0724 08:54:59.843572 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:55:01.843538726 +0100 IST m=+228.331395648 (durationBeforeRetry 2s). Error: MountVolume.MountDevice failed for volume "pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4") pod "sonarr-867d7dd548-fw6z4" (UID: "1a1846b0-3361-4f4b-a1cd-d5f958d6a7ac") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:55:00 bluth k3s[11030]: time="2022-07-24T08:55:00+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-esphome\", Name:\"esphome\", UID:\"9d7f07bb-0ed3-4a9b-b9b3-ce707e119669\", APIVersion:\"v1\", ResourceVersion:\"3027860\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-esphome/svclb-esphome" Jul 24 08:55:00 bluth k3s[11030]: I0724 08:55:00.163617 11030 event.go:294] "Event occurred" object="ix-esphome/svclb-esphome" kind="DaemonSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulCreate" message="Created pod: svclb-esphome-5q48s" Jul 24 08:55:00 bluth k3s[11030]: I0724 08:55:00.167615 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:55:00 bluth k3s[11030]: E0724 08:55:00.449760 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:55:08.44972307 +0100 IST m=+234.937580007 (durationBeforeRetry 8s). Error: MountVolume.MountDevice failed for volume "pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e") pod "esphome-658957fddc-6z2jm" (UID: "19254d20-7ad1-49a1-b408-f6e4213e5b4e") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:55:00 bluth k3s[11030]: time="2022-07-24T08:55:00+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-qbittorrent\", Name:\"qbittorrent-torrent\", UID:\"29b172f8-d799-4e0c-aac7-78ec5af5c01b\", APIVersion:\"v1\", ResourceVersion:\"2334164\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-qbittorrent/svclb-qbittorrent-torrent" Jul 24 08:55:00 bluth k3s[11030]: time="2022-07-24T08:55:00+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-qbittorrent\", Name:\"qbittorrent-torrent\", UID:\"29b172f8-d799-4e0c-aac7-78ec5af5c01b\", APIVersion:\"v1\", ResourceVersion:\"2334164\", FieldPath:\"\"}): type: 'Normal' reason: 'UpdatedIngressIP' LoadBalancer Ingress IP addresses updated: " Jul 24 08:55:00 bluth k3s[11030]: time="2022-07-24T08:55:00+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-qbittorrent\", Name:\"qbittorrent-torrent\", UID:\"29b172f8-d799-4e0c-aac7-78ec5af5c01b\", APIVersion:\"v1\", ResourceVersion:\"3027879\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-qbittorrent/svclb-qbittorrent-torrent" Jul 24 08:55:00 bluth k3s[11030]: time="2022-07-24T08:55:00+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-qbittorrent\", Name:\"qbittorrent-torrent\", UID:\"29b172f8-d799-4e0c-aac7-78ec5af5c01b\", APIVersion:\"v1\", ResourceVersion:\"3027879\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-qbittorrent/svclb-qbittorrent-torrent" Jul 24 08:55:00 bluth k3s[11030]: I0724 08:55:00.780201 11030 event.go:294] "Event occurred" object="ix-qbittorrent/svclb-qbittorrent-torrent" kind="DaemonSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulCreate" message="Created pod: svclb-qbittorrent-torrent-dnxlw" Jul 24 08:55:00 bluth k3s[11030]: I0724 08:55:00.791214 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:55:00 bluth k3s[11030]: E0724 08:55:00.851914 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-195f1a32-9c93-4737-983a-8dacee32e453 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:55:04.851905133 +0100 IST m=+231.339762014 (durationBeforeRetry 4s). Error: MountVolume.MountDevice failed for volume "pvc-195f1a32-9c93-4737-983a-8dacee32e453" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-195f1a32-9c93-4737-983a-8dacee32e453") pod "radarr-7486f78d7f-ntt44" (UID: "9ab9d8fe-e669-40bb-a0ce-7f7a4e5a69b5") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:55:01 bluth k3s[11030]: time="2022-07-24T08:55:01+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-esphome\", Name:\"esphome\", UID:\"9d7f07bb-0ed3-4a9b-b9b3-ce707e119669\", APIVersion:\"v1\", ResourceVersion:\"3027860\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-esphome/svclb-esphome" Jul 24 08:55:01 bluth k3s[11030]: time="2022-07-24T08:55:01+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-tdarr\", Name:\"tdarr\", UID:\"90709eb9-3d84-4b91-b6ab-73ece8ffcb86\", APIVersion:\"v1\", ResourceVersion:\"2504489\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-tdarr/svclb-tdarr" Jul 24 08:55:01 bluth k3s[11030]: time="2022-07-24T08:55:01+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-tdarr\", Name:\"tdarr\", UID:\"90709eb9-3d84-4b91-b6ab-73ece8ffcb86\", APIVersion:\"v1\", ResourceVersion:\"2504489\", FieldPath:\"\"}): type: 'Normal' reason: 'UpdatedIngressIP' LoadBalancer Ingress IP addresses updated: " Jul 24 08:55:01 bluth k3s[11030]: time="2022-07-24T08:55:01+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-tdarr\", Name:\"tdarr\", UID:\"90709eb9-3d84-4b91-b6ab-73ece8ffcb86\", APIVersion:\"v1\", ResourceVersion:\"3027898\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-tdarr/svclb-tdarr" Jul 24 08:55:01 bluth k3s[11030]: E0724 08:55:01.254942 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-06521873-c179-499e-9079-194ec6618fdd podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:55:09.25491532 +0100 IST m=+235.742772229 (durationBeforeRetry 8s). Error: MountVolume.MountDevice failed for volume "pvc-06521873-c179-499e-9079-194ec6618fdd" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-06521873-c179-499e-9079-194ec6618fdd") pod "esphome-658957fddc-6z2jm" (UID: "19254d20-7ad1-49a1-b408-f6e4213e5b4e") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:55:01 bluth k3s[11030]: time="2022-07-24T08:55:01+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-tdarr\", Name:\"tdarr\", UID:\"90709eb9-3d84-4b91-b6ab-73ece8ffcb86\", APIVersion:\"v1\", ResourceVersion:\"3027898\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-tdarr/svclb-tdarr" Jul 24 08:55:01 bluth k3s[11030]: I0724 08:55:01.389715 11030 event.go:294] "Event occurred" object="ix-tdarr/svclb-tdarr" kind="DaemonSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulCreate" message="Created pod: svclb-tdarr-wfk5l" Jul 24 08:55:01 bluth k3s[11030]: I0724 08:55:01.402357 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:55:01 bluth k3s[11030]: I0724 08:55:01.749216 11030 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=19ef4922-0714-452c-b8ba-e3c955cabe50 path="/var/lib/kubelet/pods/19ef4922-0714-452c-b8ba-e3c955cabe50/volumes" Jul 24 08:55:01 bluth k3s[11030]: I0724 08:55:01.751774 11030 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=c41628dc-0533-406b-ac00-3fbf6cd22284 path="/var/lib/kubelet/pods/c41628dc-0533-406b-ac00-3fbf6cd22284/volumes" Jul 24 08:55:01 bluth k3s[11030]: I0724 08:55:01.752166 11030 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=e0e2f67d-a94d-4a52-a604-fb75e8d30555 path="/var/lib/kubelet/pods/e0e2f67d-a94d-4a52-a604-fb75e8d30555/volumes" Jul 24 08:55:01 bluth k3s[11030]: E0724 08:55:01.752452 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:55:01 bluth k3s[11030]: time="2022-07-24T08:55:01+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-qbittorrent\", Name:\"qbittorrent-torrent\", UID:\"29b172f8-d799-4e0c-aac7-78ec5af5c01b\", APIVersion:\"v1\", ResourceVersion:\"3027879\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-qbittorrent/svclb-qbittorrent-torrent" Jul 24 08:55:01 bluth k3s[11030]: E0724 08:55:01.859301 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:55:05.859267085 +0100 IST m=+232.347124009 (durationBeforeRetry 4s). Error: MountVolume.MountDevice failed for volume "pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4") pod "sonarr-867d7dd548-fw6z4" (UID: "1a1846b0-3361-4f4b-a1cd-d5f958d6a7ac") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:55:02 bluth k3s[11030]: time="2022-07-24T08:55:02+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-tdarr\", Name:\"tdarr\", UID:\"90709eb9-3d84-4b91-b6ab-73ece8ffcb86\", APIVersion:\"v1\", ResourceVersion:\"3027898\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-tdarr/svclb-tdarr" Jul 24 08:55:02 bluth k3s[11030]: E0724 08:55:02.663755 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:55:10.663720464 +0100 IST m=+237.151577406 (durationBeforeRetry 8s). Error: MountVolume.MountDevice failed for volume "pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274") pod "prowlarr-5bcc99cc47-v86hm" (UID: "5d1f2955-de8a-4e25-b1f8-e607463ed369") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:55:03 bluth k3s[11030]: E0724 08:55:03.671283 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-58562e12-4adc-4e83-8131-848f1b39bf00 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:55:11.671249304 +0100 IST m=+238.159106246 (durationBeforeRetry 8s). Error: MountVolume.MountDevice failed for volume "pvc-58562e12-4adc-4e83-8131-848f1b39bf00" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-58562e12-4adc-4e83-8131-848f1b39bf00") pod "qbittorrent-6d7cf9f7cd-xrqwj" (UID: "4d43a190-0b41-4f84-a84d-ce295694fcb5") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:55:03 bluth k3s[11030]: E0724 08:55:03.755447 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:55:04 bluth k3s[11030]: E0724 08:55:04.880977 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-195f1a32-9c93-4737-983a-8dacee32e453 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:55:12.88094377 +0100 IST m=+239.368800713 (durationBeforeRetry 8s). Error: MountVolume.MountDevice failed for volume "pvc-195f1a32-9c93-4737-983a-8dacee32e453" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-195f1a32-9c93-4737-983a-8dacee32e453") pod "radarr-7486f78d7f-ntt44" (UID: "9ab9d8fe-e669-40bb-a0ce-7f7a4e5a69b5") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:55:05 bluth k3s[11030]: I0724 08:55:05.262072 11030 scope.go:110] "RemoveContainer" containerID="6ad0b3c8059d50b878addcfca6272e5cb2118b8128b92b8010fcef833e38ee3d" Jul 24 08:55:05 bluth k3s[11030]: E0724 08:55:05.755904 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:55:05 bluth k3s[11030]: E0724 08:55:05.888830 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:55:13.888793618 +0100 IST m=+240.376650551 (durationBeforeRetry 8s). Error: MountVolume.MountDevice failed for volume "pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4") pod "sonarr-867d7dd548-fw6z4" (UID: "1a1846b0-3361-4f4b-a1cd-d5f958d6a7ac") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:55:07 bluth k3s[11030]: E0724 08:55:07.777086 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:55:08 bluth k3s[11030]: E0724 08:55:08.513851 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:55:24.513817626 +0100 IST m=+251.001674567 (durationBeforeRetry 16s). Error: MountVolume.MountDevice failed for volume "pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e") pod "esphome-658957fddc-6z2jm" (UID: "19254d20-7ad1-49a1-b408-f6e4213e5b4e") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:55:09 bluth k3s[11030]: E0724 08:55:09.321436 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-06521873-c179-499e-9079-194ec6618fdd podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:55:25.321400404 +0100 IST m=+251.809257343 (durationBeforeRetry 16s). Error: MountVolume.MountDevice failed for volume "pvc-06521873-c179-499e-9079-194ec6618fdd" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-06521873-c179-499e-9079-194ec6618fdd") pod "esphome-658957fddc-6z2jm" (UID: "19254d20-7ad1-49a1-b408-f6e4213e5b4e") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:55:09 bluth k3s[11030]: I0724 08:55:09.723861 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"chroot-zfs\" (UniqueName: \"kubernetes.io/configmap/d312e744-054d-4a0d-8f39-46a83c98c808-chroot-zfs\") pod \"d312e744-054d-4a0d-8f39-46a83c98c808\" (UID: \"d312e744-054d-4a0d-8f39-46a83c98c808\") " Jul 24 08:55:09 bluth k3s[11030]: I0724 08:55:09.723951 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"host-root\" (UniqueName: \"kubernetes.io/host-path/d312e744-054d-4a0d-8f39-46a83c98c808-host-root\") pod \"d312e744-054d-4a0d-8f39-46a83c98c808\" (UID: \"d312e744-054d-4a0d-8f39-46a83c98c808\") " Jul 24 08:55:09 bluth k3s[11030]: I0724 08:55:09.724008 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"plugin-dir\" (UniqueName: \"kubernetes.io/host-path/d312e744-054d-4a0d-8f39-46a83c98c808-plugin-dir\") pod \"d312e744-054d-4a0d-8f39-46a83c98c808\" (UID: \"d312e744-054d-4a0d-8f39-46a83c98c808\") " Jul 24 08:55:09 bluth k3s[11030]: I0724 08:55:09.724065 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d312e744-054d-4a0d-8f39-46a83c98c808-registration-dir\") pod \"d312e744-054d-4a0d-8f39-46a83c98c808\" (UID: \"d312e744-054d-4a0d-8f39-46a83c98c808\") " Jul 24 08:55:09 bluth k3s[11030]: I0724 08:55:09.724122 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d312e744-054d-4a0d-8f39-46a83c98c808-device-dir\") pod \"d312e744-054d-4a0d-8f39-46a83c98c808\" (UID: \"d312e744-054d-4a0d-8f39-46a83c98c808\") " Jul 24 08:55:09 bluth k3s[11030]: I0724 08:55:09.724180 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vv85\" (UniqueName: \"kubernetes.io/projected/d312e744-054d-4a0d-8f39-46a83c98c808-kube-api-access-8vv85\") pod \"d312e744-054d-4a0d-8f39-46a83c98c808\" (UID: \"d312e744-054d-4a0d-8f39-46a83c98c808\") " Jul 24 08:55:09 bluth k3s[11030]: I0724 08:55:09.724163 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/d312e744-054d-4a0d-8f39-46a83c98c808-host-root" (OuterVolumeSpecName: "host-root") pod "d312e744-054d-4a0d-8f39-46a83c98c808" (UID: "d312e744-054d-4a0d-8f39-46a83c98c808"). InnerVolumeSpecName "host-root". PluginName "kubernetes.io/host-path", VolumeGidValue "" Jul 24 08:55:09 bluth k3s[11030]: I0724 08:55:09.724206 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/d312e744-054d-4a0d-8f39-46a83c98c808-plugin-dir" (OuterVolumeSpecName: "plugin-dir") pod "d312e744-054d-4a0d-8f39-46a83c98c808" (UID: "d312e744-054d-4a0d-8f39-46a83c98c808"). InnerVolumeSpecName "plugin-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Jul 24 08:55:09 bluth k3s[11030]: I0724 08:55:09.724231 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"pods-mount-dir\" (UniqueName: \"kubernetes.io/host-path/d312e744-054d-4a0d-8f39-46a83c98c808-pods-mount-dir\") pod \"d312e744-054d-4a0d-8f39-46a83c98c808\" (UID: \"d312e744-054d-4a0d-8f39-46a83c98c808\") " Jul 24 08:55:09 bluth k3s[11030]: I0724 08:55:09.724302 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/d312e744-054d-4a0d-8f39-46a83c98c808-device-dir" (OuterVolumeSpecName: "device-dir") pod "d312e744-054d-4a0d-8f39-46a83c98c808" (UID: "d312e744-054d-4a0d-8f39-46a83c98c808"). InnerVolumeSpecName "device-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Jul 24 08:55:09 bluth k3s[11030]: I0724 08:55:09.724320 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/d312e744-054d-4a0d-8f39-46a83c98c808-pods-mount-dir" (OuterVolumeSpecName: "pods-mount-dir") pod "d312e744-054d-4a0d-8f39-46a83c98c808" (UID: "d312e744-054d-4a0d-8f39-46a83c98c808"). InnerVolumeSpecName "pods-mount-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Jul 24 08:55:09 bluth k3s[11030]: I0724 08:55:09.724346 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"encr-keys\" (UniqueName: \"kubernetes.io/host-path/d312e744-054d-4a0d-8f39-46a83c98c808-encr-keys\") pod \"d312e744-054d-4a0d-8f39-46a83c98c808\" (UID: \"d312e744-054d-4a0d-8f39-46a83c98c808\") " Jul 24 08:55:09 bluth k3s[11030]: I0724 08:55:09.724407 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/d312e744-054d-4a0d-8f39-46a83c98c808-registration-dir" (OuterVolumeSpecName: "registration-dir") pod "d312e744-054d-4a0d-8f39-46a83c98c808" (UID: "d312e744-054d-4a0d-8f39-46a83c98c808"). InnerVolumeSpecName "registration-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Jul 24 08:55:09 bluth k3s[11030]: I0724 08:55:09.724610 11030 reconciler.go:300] "Volume detached for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d312e744-054d-4a0d-8f39-46a83c98c808-registration-dir\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:55:09 bluth k3s[11030]: I0724 08:55:09.724646 11030 reconciler.go:300] "Volume detached for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d312e744-054d-4a0d-8f39-46a83c98c808-device-dir\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:55:09 bluth k3s[11030]: I0724 08:55:09.724650 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/d312e744-054d-4a0d-8f39-46a83c98c808-encr-keys" (OuterVolumeSpecName: "encr-keys") pod "d312e744-054d-4a0d-8f39-46a83c98c808" (UID: "d312e744-054d-4a0d-8f39-46a83c98c808"). InnerVolumeSpecName "encr-keys". PluginName "kubernetes.io/host-path", VolumeGidValue "" Jul 24 08:55:09 bluth k3s[11030]: I0724 08:55:09.724677 11030 reconciler.go:300] "Volume detached for volume \"pods-mount-dir\" (UniqueName: \"kubernetes.io/host-path/d312e744-054d-4a0d-8f39-46a83c98c808-pods-mount-dir\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:55:09 bluth k3s[11030]: I0724 08:55:09.724708 11030 reconciler.go:300] "Volume detached for volume \"host-root\" (UniqueName: \"kubernetes.io/host-path/d312e744-054d-4a0d-8f39-46a83c98c808-host-root\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:55:09 bluth k3s[11030]: I0724 08:55:09.724737 11030 reconciler.go:300] "Volume detached for volume \"plugin-dir\" (UniqueName: \"kubernetes.io/host-path/d312e744-054d-4a0d-8f39-46a83c98c808-plugin-dir\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:55:09 bluth k3s[11030]: I0724 08:55:09.727886 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d312e744-054d-4a0d-8f39-46a83c98c808-kube-api-access-8vv85" (OuterVolumeSpecName: "kube-api-access-8vv85") pod "d312e744-054d-4a0d-8f39-46a83c98c808" (UID: "d312e744-054d-4a0d-8f39-46a83c98c808"). InnerVolumeSpecName "kube-api-access-8vv85". PluginName "kubernetes.io/projected", VolumeGidValue "" Jul 24 08:55:09 bluth k3s[11030]: W0724 08:55:09.730218 11030 mount_helper_common.go:133] Warning: "/var/lib/kubelet/pods/d312e744-054d-4a0d-8f39-46a83c98c808/volume-subpaths/chroot-zfs/openebs-zfs-plugin/3" is not a mountpoint, deleting Jul 24 08:55:09 bluth k3s[11030]: W0724 08:55:09.731261 11030 empty_dir.go:517] Warning: Failed to clear quota on /var/lib/kubelet/pods/d312e744-054d-4a0d-8f39-46a83c98c808/volumes/kubernetes.io~configmap/chroot-zfs: clearQuota called, but quotas disabled Jul 24 08:55:09 bluth k3s[11030]: I0724 08:55:09.731987 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d312e744-054d-4a0d-8f39-46a83c98c808-chroot-zfs" (OuterVolumeSpecName: "chroot-zfs") pod "d312e744-054d-4a0d-8f39-46a83c98c808" (UID: "d312e744-054d-4a0d-8f39-46a83c98c808"). InnerVolumeSpecName "chroot-zfs". PluginName "kubernetes.io/configmap", VolumeGidValue "" Jul 24 08:55:09 bluth k3s[11030]: E0724 08:55:09.753475 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:55:09 bluth k3s[11030]: I0724 08:55:09.825938 11030 reconciler.go:300] "Volume detached for volume \"kube-api-access-8vv85\" (UniqueName: \"kubernetes.io/projected/d312e744-054d-4a0d-8f39-46a83c98c808-kube-api-access-8vv85\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:55:09 bluth k3s[11030]: I0724 08:55:09.826004 11030 reconciler.go:300] "Volume detached for volume \"encr-keys\" (UniqueName: \"kubernetes.io/host-path/d312e744-054d-4a0d-8f39-46a83c98c808-encr-keys\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:55:09 bluth k3s[11030]: I0724 08:55:09.826037 11030 reconciler.go:300] "Volume detached for volume \"chroot-zfs\" (UniqueName: \"kubernetes.io/configmap/d312e744-054d-4a0d-8f39-46a83c98c808-chroot-zfs\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:55:10 bluth k3s[11030]: E0724 08:55:10.734224 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:55:26.734213877 +0100 IST m=+253.222070758 (durationBeforeRetry 16s). Error: MountVolume.MountDevice failed for volume "pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274") pod "prowlarr-5bcc99cc47-v86hm" (UID: "5d1f2955-de8a-4e25-b1f8-e607463ed369") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:55:10 bluth k3s[11030]: E0724 08:55:10.954536 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: d4aed5c62635e97bfd4e98215c786fc4419297c23985754ae4f57fc76cc6c21c" containerID="d4aed5c62635e97bfd4e98215c786fc4419297c23985754ae4f57fc76cc6c21c" Jul 24 08:55:10 bluth k3s[11030]: E0724 08:55:10.954607 11030 kuberuntime_manager.go:1079] "getPodContainerStatuses for pod failed" err="rpc error: code = Unknown desc = Error: No such container: d4aed5c62635e97bfd4e98215c786fc4419297c23985754ae4f57fc76cc6c21c" pod="kube-system/openebs-zfs-controller-0" Jul 24 08:55:11 bluth k3s[11030]: E0724 08:55:11.742498 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-58562e12-4adc-4e83-8131-848f1b39bf00 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:55:27.742461771 +0100 IST m=+254.230318702 (durationBeforeRetry 16s). Error: MountVolume.MountDevice failed for volume "pvc-58562e12-4adc-4e83-8131-848f1b39bf00" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-58562e12-4adc-4e83-8131-848f1b39bf00") pod "qbittorrent-6d7cf9f7cd-xrqwj" (UID: "4d43a190-0b41-4f84-a84d-ce295694fcb5") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:55:11 bluth k3s[11030]: E0724 08:55:11.755151 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:55:12 bluth k3s[11030]: I0724 08:55:12.140936 11030 trace.go:205] Trace[869327891]: "GuaranteedUpdate etcd3" type:*core.Pod (24-Jul-2022 08:55:10.953) (total time: 1187ms): Jul 24 08:55:12 bluth k3s[11030]: Trace[869327891]: ---"Transaction committed" 1178ms (08:55:12.140) Jul 24 08:55:12 bluth k3s[11030]: Trace[869327891]: [1.187813748s] [1.187813748s] END Jul 24 08:55:12 bluth k3s[11030]: I0724 08:55:12.141279 11030 trace.go:205] Trace[1345154259]: "Patch" url:/api/v1/namespaces/kube-system/pods/openebs-zfs-node-bf7kk/status,user-agent:Go-http-client/2.0,audit-id:0e0fd9c3-e83b-47c8-aee6-348892776a59,client:127.0.0.1,accept:application/vnd.kubernetes.protobuf,application/json,protocol:HTTP/2.0 (24-Jul-2022 08:55:10.952) (total time: 1188ms): Jul 24 08:55:12 bluth k3s[11030]: Trace[1345154259]: ---"Object stored in database" 1179ms (08:55:12.140) Jul 24 08:55:12 bluth k3s[11030]: Trace[1345154259]: [1.188393625s] [1.188393625s] END Jul 24 08:55:12 bluth k3s[11030]: I0724 08:55:12.683658 11030 event.go:294] "Event occurred" object="kube-system/openebs-zfs-node" kind="DaemonSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulCreate" message="Created pod: openebs-zfs-node-vk949" Jul 24 08:55:12 bluth k3s[11030]: I0724 08:55:12.694922 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:55:12 bluth k3s[11030]: time="2022-07-24T08:55:12+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-sonarr\", Name:\"sonarr\", UID:\"b24571a2-a018-4716-ab7d-d6ab31e66e44\", APIVersion:\"v1\", ResourceVersion:\"2334145\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-sonarr/svclb-sonarr" Jul 24 08:55:12 bluth k3s[11030]: time="2022-07-24T08:55:12+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-sonarr\", Name:\"sonarr\", UID:\"b24571a2-a018-4716-ab7d-d6ab31e66e44\", APIVersion:\"v1\", ResourceVersion:\"2334145\", FieldPath:\"\"}): type: 'Normal' reason: 'UpdatedIngressIP' LoadBalancer Ingress IP addresses updated: " Jul 24 08:55:12 bluth k3s[11030]: time="2022-07-24T08:55:12+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-sonarr\", Name:\"sonarr\", UID:\"b24571a2-a018-4716-ab7d-d6ab31e66e44\", APIVersion:\"v1\", ResourceVersion:\"3027945\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-sonarr/svclb-sonarr" Jul 24 08:55:12 bluth k3s[11030]: time="2022-07-24T08:55:12+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-sonarr\", Name:\"sonarr\", UID:\"b24571a2-a018-4716-ab7d-d6ab31e66e44\", APIVersion:\"v1\", ResourceVersion:\"3027945\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-sonarr/svclb-sonarr" Jul 24 08:55:12 bluth k3s[11030]: I0724 08:55:12.741518 11030 event.go:294] "Event occurred" object="ix-sonarr/svclb-sonarr" kind="DaemonSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulCreate" message="Created pod: svclb-sonarr-7tc6n" Jul 24 08:55:12 bluth k3s[11030]: I0724 08:55:12.745645 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:55:12 bluth k3s[11030]: I0724 08:55:12.850711 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/74df464f-f315-4c9f-8e03-c682d351af8c-device-dir\") pod \"openebs-zfs-node-vk949\" (UID: \"74df464f-f315-4c9f-8e03-c682d351af8c\") " pod="kube-system/openebs-zfs-node-vk949" Jul 24 08:55:12 bluth k3s[11030]: I0724 08:55:12.850811 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"encr-keys\" (UniqueName: \"kubernetes.io/host-path/74df464f-f315-4c9f-8e03-c682d351af8c-encr-keys\") pod \"openebs-zfs-node-vk949\" (UID: \"74df464f-f315-4c9f-8e03-c682d351af8c\") " pod="kube-system/openebs-zfs-node-vk949" Jul 24 08:55:12 bluth k3s[11030]: I0724 08:55:12.850875 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-dir\" (UniqueName: \"kubernetes.io/host-path/74df464f-f315-4c9f-8e03-c682d351af8c-plugin-dir\") pod \"openebs-zfs-node-vk949\" (UID: \"74df464f-f315-4c9f-8e03-c682d351af8c\") " pod="kube-system/openebs-zfs-node-vk949" Jul 24 08:55:12 bluth k3s[11030]: I0724 08:55:12.851007 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pods-mount-dir\" (UniqueName: \"kubernetes.io/host-path/74df464f-f315-4c9f-8e03-c682d351af8c-pods-mount-dir\") pod \"openebs-zfs-node-vk949\" (UID: \"74df464f-f315-4c9f-8e03-c682d351af8c\") " pod="kube-system/openebs-zfs-node-vk949" Jul 24 08:55:12 bluth k3s[11030]: I0724 08:55:12.851114 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpxkg\" (UniqueName: \"kubernetes.io/projected/74df464f-f315-4c9f-8e03-c682d351af8c-kube-api-access-gpxkg\") pod \"openebs-zfs-node-vk949\" (UID: \"74df464f-f315-4c9f-8e03-c682d351af8c\") " pod="kube-system/openebs-zfs-node-vk949" Jul 24 08:55:12 bluth k3s[11030]: I0724 08:55:12.851277 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-root\" (UniqueName: \"kubernetes.io/host-path/74df464f-f315-4c9f-8e03-c682d351af8c-host-root\") pod \"openebs-zfs-node-vk949\" (UID: \"74df464f-f315-4c9f-8e03-c682d351af8c\") " pod="kube-system/openebs-zfs-node-vk949" Jul 24 08:55:12 bluth k3s[11030]: I0724 08:55:12.851361 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"chroot-zfs\" (UniqueName: \"kubernetes.io/configmap/74df464f-f315-4c9f-8e03-c682d351af8c-chroot-zfs\") pod \"openebs-zfs-node-vk949\" (UID: \"74df464f-f315-4c9f-8e03-c682d351af8c\") " pod="kube-system/openebs-zfs-node-vk949" Jul 24 08:55:12 bluth k3s[11030]: I0724 08:55:12.851537 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/74df464f-f315-4c9f-8e03-c682d351af8c-registration-dir\") pod \"openebs-zfs-node-vk949\" (UID: \"74df464f-f315-4c9f-8e03-c682d351af8c\") " pod="kube-system/openebs-zfs-node-vk949" Jul 24 08:55:12 bluth k3s[11030]: E0724 08:55:12.953465 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-195f1a32-9c93-4737-983a-8dacee32e453 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:55:28.953430159 +0100 IST m=+255.441287099 (durationBeforeRetry 16s). Error: MountVolume.MountDevice failed for volume "pvc-195f1a32-9c93-4737-983a-8dacee32e453" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-195f1a32-9c93-4737-983a-8dacee32e453") pod "radarr-7486f78d7f-ntt44" (UID: "9ab9d8fe-e669-40bb-a0ce-7f7a4e5a69b5") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:55:13 bluth k3s[11030]: I0724 08:55:13.376081 11030 scope.go:110] "RemoveContainer" containerID="45dcd9b69c3fc1c416b4d575574763088bdacb840d4037acfad820d24bbe7f7b" Jul 24 08:55:13 bluth k3s[11030]: time="2022-07-24T08:55:13+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-sonarr\", Name:\"sonarr\", UID:\"b24571a2-a018-4716-ab7d-d6ab31e66e44\", APIVersion:\"v1\", ResourceVersion:\"3027945\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-sonarr/svclb-sonarr" Jul 24 08:55:13 bluth k3s[11030]: I0724 08:55:13.776719 11030 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=d312e744-054d-4a0d-8f39-46a83c98c808 path="/var/lib/kubelet/pods/d312e744-054d-4a0d-8f39-46a83c98c808/volumes" Jul 24 08:55:13 bluth k3s[11030]: I0724 08:55:13.780859 11030 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=ecd248e1-4900-4e80-ba0b-7233f4b438b4 path="/var/lib/kubelet/pods/ecd248e1-4900-4e80-ba0b-7233f4b438b4/volumes" Jul 24 08:55:13 bluth k3s[11030]: E0724 08:55:13.782142 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:55:13 bluth k3s[11030]: E0724 08:55:13.952587 11030 kuberuntime_gc.go:382] "Failed to remove container log dead symlink" err="remove /var/log/containers/svclb-qbittorrent-ldv82_ix-qbittorrent_lb-tcp-10095-45dcd9b69c3fc1c416b4d575574763088bdacb840d4037acfad820d24bbe7f7b.log: no such file or directory" path="/var/log/containers/svclb-qbittorrent-ldv82_ix-qbittorrent_lb-tcp-10095-45dcd9b69c3fc1c416b4d575574763088bdacb840d4037acfad820d24bbe7f7b.log" Jul 24 08:55:13 bluth k3s[11030]: E0724 08:55:13.960557 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:55:29.960525195 +0100 IST m=+256.448382136 (durationBeforeRetry 16s). Error: MountVolume.MountDevice failed for volume "pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4") pod "sonarr-867d7dd548-fw6z4" (UID: "1a1846b0-3361-4f4b-a1cd-d5f958d6a7ac") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:55:15 bluth k3s[11030]: E0724 08:55:15.754698 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:55:16 bluth k3s[11030]: E0724 08:55:16.361090 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: e7f852052783cf0f1158cbe52082e8494d9bec16200abb1bd86912d687bab28e" podSandboxID="e7f852052783cf0f1158cbe52082e8494d9bec16200abb1bd86912d687bab28e" pod="ix-prowlarr/svclb-prowlarr-jgkkl" Jul 24 08:55:17 bluth k3s[11030]: E0724 08:55:17.397519 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: e7f852052783cf0f1158cbe52082e8494d9bec16200abb1bd86912d687bab28e" podSandboxID="e7f852052783cf0f1158cbe52082e8494d9bec16200abb1bd86912d687bab28e" pod="ix-prowlarr/svclb-prowlarr-jgkkl" Jul 24 08:55:17 bluth k3s[11030]: E0724 08:55:17.401036 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: e7f852052783cf0f1158cbe52082e8494d9bec16200abb1bd86912d687bab28e" podSandboxID="e7f852052783cf0f1158cbe52082e8494d9bec16200abb1bd86912d687bab28e" pod="ix-prowlarr/svclb-prowlarr-jgkkl" Jul 24 08:55:17 bluth k3s[11030]: E0724 08:55:17.773567 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:55:18 bluth k3s[11030]: E0724 08:55:18.415505 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: e7f852052783cf0f1158cbe52082e8494d9bec16200abb1bd86912d687bab28e" podSandboxID="e7f852052783cf0f1158cbe52082e8494d9bec16200abb1bd86912d687bab28e" pod="ix-prowlarr/svclb-prowlarr-jgkkl" Jul 24 08:55:18 bluth k3s[11030]: E0724 08:55:18.416415 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: e7f852052783cf0f1158cbe52082e8494d9bec16200abb1bd86912d687bab28e" podSandboxID="e7f852052783cf0f1158cbe52082e8494d9bec16200abb1bd86912d687bab28e" pod="ix-prowlarr/svclb-prowlarr-jgkkl" Jul 24 08:55:19 bluth k3s[11030]: E0724 08:55:19.457515 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: b7eb2db6018786c3f85f9fc53965256556346e4a1fcce1606390c2a82e355732" podSandboxID="b7eb2db6018786c3f85f9fc53965256556346e4a1fcce1606390c2a82e355732" pod="ix-qbittorrent/svclb-qbittorrent-7jf9d" Jul 24 08:55:19 bluth k3s[11030]: E0724 08:55:19.459732 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: e7f852052783cf0f1158cbe52082e8494d9bec16200abb1bd86912d687bab28e" podSandboxID="e7f852052783cf0f1158cbe52082e8494d9bec16200abb1bd86912d687bab28e" pod="ix-prowlarr/svclb-prowlarr-jgkkl" Jul 24 08:55:19 bluth k3s[11030]: E0724 08:55:19.461789 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: e7f852052783cf0f1158cbe52082e8494d9bec16200abb1bd86912d687bab28e" podSandboxID="e7f852052783cf0f1158cbe52082e8494d9bec16200abb1bd86912d687bab28e" pod="ix-prowlarr/svclb-prowlarr-jgkkl" Jul 24 08:55:19 bluth k3s[11030]: I0724 08:55:19.576961 11030 scope.go:110] "RemoveContainer" containerID="45dcd9b69c3fc1c416b4d575574763088bdacb840d4037acfad820d24bbe7f7b" Jul 24 08:55:19 bluth k3s[11030]: E0724 08:55:19.578244 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 45dcd9b69c3fc1c416b4d575574763088bdacb840d4037acfad820d24bbe7f7b" containerID="45dcd9b69c3fc1c416b4d575574763088bdacb840d4037acfad820d24bbe7f7b" Jul 24 08:55:19 bluth k3s[11030]: I0724 08:55:19.578371 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:45dcd9b69c3fc1c416b4d575574763088bdacb840d4037acfad820d24bbe7f7b} err="failed to get container status \"45dcd9b69c3fc1c416b4d575574763088bdacb840d4037acfad820d24bbe7f7b\": rpc error: code = Unknown desc = Error: No such container: 45dcd9b69c3fc1c416b4d575574763088bdacb840d4037acfad820d24bbe7f7b" Jul 24 08:55:19 bluth k3s[11030]: I0724 08:55:19.578409 11030 scope.go:110] "RemoveContainer" containerID="5a2bde0c9fe43ee4870ac69ebca1c9f130c43dd8b4120fa0d3b147738a284b12" Jul 24 08:55:19 bluth k3s[11030]: E0724 08:55:19.749713 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:55:20 bluth k3s[11030]: E0724 08:55:20.499930 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: cf37e26d23606449781f8a77d7a38224c8969f154987bde48bca1566b536f094" podSandboxID="cf37e26d23606449781f8a77d7a38224c8969f154987bde48bca1566b536f094" pod="ix-radarr/svclb-radarr-s72px" Jul 24 08:55:20 bluth k3s[11030]: E0724 08:55:20.512053 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: b7eb2db6018786c3f85f9fc53965256556346e4a1fcce1606390c2a82e355732" podSandboxID="b7eb2db6018786c3f85f9fc53965256556346e4a1fcce1606390c2a82e355732" pod="ix-qbittorrent/svclb-qbittorrent-7jf9d" Jul 24 08:55:20 bluth k3s[11030]: E0724 08:55:20.530826 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: e7f852052783cf0f1158cbe52082e8494d9bec16200abb1bd86912d687bab28e" podSandboxID="e7f852052783cf0f1158cbe52082e8494d9bec16200abb1bd86912d687bab28e" pod="ix-prowlarr/svclb-prowlarr-jgkkl" Jul 24 08:55:20 bluth k3s[11030]: E0724 08:55:20.534299 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: b7eb2db6018786c3f85f9fc53965256556346e4a1fcce1606390c2a82e355732" podSandboxID="b7eb2db6018786c3f85f9fc53965256556346e4a1fcce1606390c2a82e355732" pod="ix-qbittorrent/svclb-qbittorrent-7jf9d" Jul 24 08:55:20 bluth k3s[11030]: E0724 08:55:20.537802 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: e7f852052783cf0f1158cbe52082e8494d9bec16200abb1bd86912d687bab28e" podSandboxID="e7f852052783cf0f1158cbe52082e8494d9bec16200abb1bd86912d687bab28e" pod="ix-prowlarr/svclb-prowlarr-jgkkl" Jul 24 08:55:21 bluth k3s[11030]: E0724 08:55:21.776889 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:55:22 bluth k3s[11030]: I0724 08:55:22.149167 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="e7f852052783cf0f1158cbe52082e8494d9bec16200abb1bd86912d687bab28e" Jul 24 08:55:22 bluth k3s[11030]: E0724 08:55:22.152739 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: cf37e26d23606449781f8a77d7a38224c8969f154987bde48bca1566b536f094" podSandboxID="cf37e26d23606449781f8a77d7a38224c8969f154987bde48bca1566b536f094" pod="ix-radarr/svclb-radarr-s72px" Jul 24 08:55:22 bluth k3s[11030]: E0724 08:55:22.157951 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: b7eb2db6018786c3f85f9fc53965256556346e4a1fcce1606390c2a82e355732" podSandboxID="b7eb2db6018786c3f85f9fc53965256556346e4a1fcce1606390c2a82e355732" pod="ix-qbittorrent/svclb-qbittorrent-7jf9d" Jul 24 08:55:22 bluth k3s[11030]: E0724 08:55:22.166991 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: cf37e26d23606449781f8a77d7a38224c8969f154987bde48bca1566b536f094" podSandboxID="cf37e26d23606449781f8a77d7a38224c8969f154987bde48bca1566b536f094" pod="ix-radarr/svclb-radarr-s72px" Jul 24 08:55:22 bluth k3s[11030]: E0724 08:55:22.170553 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: b7eb2db6018786c3f85f9fc53965256556346e4a1fcce1606390c2a82e355732" podSandboxID="b7eb2db6018786c3f85f9fc53965256556346e4a1fcce1606390c2a82e355732" pod="ix-qbittorrent/svclb-qbittorrent-7jf9d" Jul 24 08:55:23 bluth k3s[11030]: E0724 08:55:23.754164 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:55:24 bluth k3s[11030]: E0724 08:55:24.541379 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:55:56.541342882 +0100 IST m=+283.029199824 (durationBeforeRetry 32s). Error: MountVolume.MountDevice failed for volume "pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e") pod "esphome-658957fddc-6z2jm" (UID: "19254d20-7ad1-49a1-b408-f6e4213e5b4e") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:55:25 bluth k3s[11030]: I0724 08:55:25.071192 11030 scope.go:110] "RemoveContainer" containerID="353dd457b21028b066306f9214bf9a01a6e73071105259c1150aec1b0c3f2476" Jul 24 08:55:25 bluth k3s[11030]: E0724 08:55:25.349574 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-06521873-c179-499e-9079-194ec6618fdd podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:55:57.349538599 +0100 IST m=+283.837395542 (durationBeforeRetry 32s). Error: MountVolume.MountDevice failed for volume "pvc-06521873-c179-499e-9079-194ec6618fdd" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-06521873-c179-499e-9079-194ec6618fdd") pod "esphome-658957fddc-6z2jm" (UID: "19254d20-7ad1-49a1-b408-f6e4213e5b4e") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:55:25 bluth k3s[11030]: E0724 08:55:25.756106 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:55:26 bluth k3s[11030]: E0724 08:55:26.760398 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:55:58.760363941 +0100 IST m=+285.248220882 (durationBeforeRetry 32s). Error: MountVolume.MountDevice failed for volume "pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274") pod "prowlarr-5bcc99cc47-v86hm" (UID: "5d1f2955-de8a-4e25-b1f8-e607463ed369") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:55:27 bluth k3s[11030]: I0724 08:55:27.395345 31770 event.go:278] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ix-prowlarr", Name:"svclb-prowlarr-jgkkl", UID:"34feca2e-c4bb-4c0a-b5b1-fce6a0404faa", APIVersion:"v1", ResourceVersion:"3027710", FieldPath:""}): type: 'Normal' reason: 'AddedInterface' Add eth0 [172.16.0.195/16] from ix-net Jul 24 08:55:27 bluth k3s[11030]: E0724 08:55:27.769522 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-58562e12-4adc-4e83-8131-848f1b39bf00 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:55:59.769487581 +0100 IST m=+286.257344507 (durationBeforeRetry 32s). Error: MountVolume.MountDevice failed for volume "pvc-58562e12-4adc-4e83-8131-848f1b39bf00" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-58562e12-4adc-4e83-8131-848f1b39bf00") pod "qbittorrent-6d7cf9f7cd-xrqwj" (UID: "4d43a190-0b41-4f84-a84d-ce295694fcb5") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:55:27 bluth k3s[11030]: E0724 08:55:27.788891 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:55:28 bluth k3s[11030]: E0724 08:55:28.978550 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-195f1a32-9c93-4737-983a-8dacee32e453 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:56:00.978515254 +0100 IST m=+287.466372195 (durationBeforeRetry 32s). Error: MountVolume.MountDevice failed for volume "pvc-195f1a32-9c93-4737-983a-8dacee32e453" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-195f1a32-9c93-4737-983a-8dacee32e453") pod "radarr-7486f78d7f-ntt44" (UID: "9ab9d8fe-e669-40bb-a0ce-7f7a4e5a69b5") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:55:29 bluth k3s[11030]: E0724 08:55:29.753907 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:55:29 bluth k3s[11030]: E0724 08:55:29.988874 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:56:01.988838644 +0100 IST m=+288.476695587 (durationBeforeRetry 32s). Error: MountVolume.MountDevice failed for volume "pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4") pod "sonarr-867d7dd548-fw6z4" (UID: "1a1846b0-3361-4f4b-a1cd-d5f958d6a7ac") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:55:31 bluth k3s[11030]: E0724 08:55:31.756788 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:55:33 bluth k3s[11030]: E0724 08:55:33.773327 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:55:35 bluth k3s[11030]: E0724 08:55:35.763355 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:55:35 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T08:55:35.943+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 08:55:37 bluth k3s[11030]: E0724 08:55:37.756460 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:55:37 bluth k3s[11030]: I0724 08:55:37.788094 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="b7eb2db6018786c3f85f9fc53965256556346e4a1fcce1606390c2a82e355732" Jul 24 08:55:37 bluth k3s[11030]: time="2022-07-24T08:55:37+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-tdarr\", Name:\"tdarr-comm\", UID:\"c3b33312-b11b-4e16-915e-205202cc0a2d\", APIVersion:\"v1\", ResourceVersion:\"2504455\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-tdarr/svclb-tdarr-comm" Jul 24 08:55:37 bluth k3s[11030]: I0724 08:55:37.956418 32372 event.go:278] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ix-qbittorrent", Name:"svclb-qbittorrent-7jf9d", UID:"7e136cc6-03a9-494b-b3fd-aa8c2c1db745", APIVersion:"v1", ResourceVersion:"3027731", FieldPath:""}): type: 'Normal' reason: 'AddedInterface' Add eth0 [172.16.0.196/16] from ix-net Jul 24 08:55:38 bluth k3s[11030]: I0724 08:55:38.018796 32435 event.go:278] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ix-radarr", Name:"svclb-radarr-s72px", UID:"58948e04-14e5-460c-973b-b8767bd320f2", APIVersion:"v1", ResourceVersion:"3027766", FieldPath:""}): type: 'Normal' reason: 'AddedInterface' Add eth0 [172.16.0.197/16] from ix-net Jul 24 08:55:38 bluth k3s[11030]: I0724 08:55:38.070353 11030 scope.go:110] "RemoveContainer" containerID="5a2bde0c9fe43ee4870ac69ebca1c9f130c43dd8b4120fa0d3b147738a284b12" Jul 24 08:55:38 bluth k3s[11030]: E0724 08:55:38.070725 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 5a2bde0c9fe43ee4870ac69ebca1c9f130c43dd8b4120fa0d3b147738a284b12" containerID="5a2bde0c9fe43ee4870ac69ebca1c9f130c43dd8b4120fa0d3b147738a284b12" Jul 24 08:55:38 bluth k3s[11030]: I0724 08:55:38.070750 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:5a2bde0c9fe43ee4870ac69ebca1c9f130c43dd8b4120fa0d3b147738a284b12} err="failed to get container status \"5a2bde0c9fe43ee4870ac69ebca1c9f130c43dd8b4120fa0d3b147738a284b12\": rpc error: code = Unknown desc = Error: No such container: 5a2bde0c9fe43ee4870ac69ebca1c9f130c43dd8b4120fa0d3b147738a284b12" Jul 24 08:55:38 bluth k3s[11030]: I0724 08:55:38.070760 11030 scope.go:110] "RemoveContainer" containerID="353dd457b21028b066306f9214bf9a01a6e73071105259c1150aec1b0c3f2476" Jul 24 08:55:38 bluth k3s[11030]: E0724 08:55:38.070976 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 353dd457b21028b066306f9214bf9a01a6e73071105259c1150aec1b0c3f2476" containerID="353dd457b21028b066306f9214bf9a01a6e73071105259c1150aec1b0c3f2476" Jul 24 08:55:38 bluth k3s[11030]: I0724 08:55:38.071002 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:353dd457b21028b066306f9214bf9a01a6e73071105259c1150aec1b0c3f2476} err="failed to get container status \"353dd457b21028b066306f9214bf9a01a6e73071105259c1150aec1b0c3f2476\": rpc error: code = Unknown desc = Error: No such container: 353dd457b21028b066306f9214bf9a01a6e73071105259c1150aec1b0c3f2476" Jul 24 08:55:38 bluth k3s[11030]: I0724 08:55:38.071010 11030 scope.go:110] "RemoveContainer" containerID="8173768d1c65a73c0aaefa5256406ec639f08671ae7df7013210d6ce62e5a052" Jul 24 08:55:38 bluth k3s[11030]: E0724 08:55:38.071261 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 8173768d1c65a73c0aaefa5256406ec639f08671ae7df7013210d6ce62e5a052" containerID="8173768d1c65a73c0aaefa5256406ec639f08671ae7df7013210d6ce62e5a052" Jul 24 08:55:38 bluth k3s[11030]: I0724 08:55:38.071277 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:8173768d1c65a73c0aaefa5256406ec639f08671ae7df7013210d6ce62e5a052} err="failed to get container status \"8173768d1c65a73c0aaefa5256406ec639f08671ae7df7013210d6ce62e5a052\": rpc error: code = Unknown desc = Error: No such container: 8173768d1c65a73c0aaefa5256406ec639f08671ae7df7013210d6ce62e5a052" Jul 24 08:55:38 bluth k3s[11030]: I0724 08:55:38.071283 11030 scope.go:110] "RemoveContainer" containerID="996ab65b7608e8fb0d744ce68a7c7d875d830d4728c5bf75a0d8431536dc2e8f" Jul 24 08:55:38 bluth k3s[11030]: E0724 08:55:38.071463 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 996ab65b7608e8fb0d744ce68a7c7d875d830d4728c5bf75a0d8431536dc2e8f" containerID="996ab65b7608e8fb0d744ce68a7c7d875d830d4728c5bf75a0d8431536dc2e8f" Jul 24 08:55:38 bluth k3s[11030]: I0724 08:55:38.071479 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:996ab65b7608e8fb0d744ce68a7c7d875d830d4728c5bf75a0d8431536dc2e8f} err="failed to get container status \"996ab65b7608e8fb0d744ce68a7c7d875d830d4728c5bf75a0d8431536dc2e8f\": rpc error: code = Unknown desc = Error: No such container: 996ab65b7608e8fb0d744ce68a7c7d875d830d4728c5bf75a0d8431536dc2e8f" Jul 24 08:55:38 bluth k3s[11030]: I0724 08:55:38.071485 11030 scope.go:110] "RemoveContainer" containerID="782924fe93893e3f55e40b0f59f0d67d687452e1d614160a7b2b0975d992454a" Jul 24 08:55:38 bluth k3s[11030]: I0724 08:55:38.308328 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="a75a4bdb83aa20c871de3da8feaa5e06b858d3847dcdf9bf0162e619d0a0c93e" Jul 24 08:55:38 bluth k3s[11030]: I0724 08:55:38.319406 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="cf37e26d23606449781f8a77d7a38224c8969f154987bde48bca1566b536f094" Jul 24 08:55:38 bluth k3s[11030]: I0724 08:55:38.479830 32659 event.go:278] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ix-qbittorrent", Name:"svclb-qbittorrent-torrentudp-wjbmf", UID:"736bfe23-c554-4e02-a345-74a8edebfc9d", APIVersion:"v1", ResourceVersion:"3027771", FieldPath:""}): type: 'Normal' reason: 'AddedInterface' Add eth0 [172.16.0.198/16] from ix-net Jul 24 08:55:39 bluth k3s[11030]: I0724 08:55:39.442578 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="e3735e3cf238a6f88c209e7356c6641e88f9c04d7d5da0600b295c119539be4f" Jul 24 08:55:39 bluth k3s[11030]: E0724 08:55:39.446947 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" podSandboxID="8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" pod="ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-5rpjh" Jul 24 08:55:39 bluth k3s[11030]: E0724 08:55:39.473038 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: d2ab53a2378f9efb632444f6575b781d98bb6f9560da139689f2e33bffa5bfc0" podSandboxID="d2ab53a2378f9efb632444f6575b781d98bb6f9560da139689f2e33bffa5bfc0" pod="ix-status-html/status-html-ix-chart-765b966694-c4nwz" Jul 24 08:55:39 bluth k3s[11030]: E0724 08:55:39.790535 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:55:40 bluth k3s[11030]: E0724 08:55:40.533387 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: d2ab53a2378f9efb632444f6575b781d98bb6f9560da139689f2e33bffa5bfc0" podSandboxID="d2ab53a2378f9efb632444f6575b781d98bb6f9560da139689f2e33bffa5bfc0" pod="ix-status-html/status-html-ix-chart-765b966694-c4nwz" Jul 24 08:55:40 bluth k3s[11030]: E0724 08:55:40.534330 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" podSandboxID="8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" pod="ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-5rpjh" Jul 24 08:55:40 bluth k3s[11030]: E0724 08:55:40.535258 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" podSandboxID="7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" pod="kube-system/intel-gpu-plugin-9t629" Jul 24 08:55:40 bluth k3s[11030]: E0724 08:55:40.536167 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" podSandboxID="8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" pod="ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-5rpjh" Jul 24 08:55:40 bluth k3s[11030]: E0724 08:55:40.537039 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: d2ab53a2378f9efb632444f6575b781d98bb6f9560da139689f2e33bffa5bfc0" podSandboxID="d2ab53a2378f9efb632444f6575b781d98bb6f9560da139689f2e33bffa5bfc0" pod="ix-status-html/status-html-ix-chart-765b966694-c4nwz" Jul 24 08:55:41 bluth k3s[11030]: E0724 08:55:41.551951 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" podSandboxID="7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" pod="kube-system/intel-gpu-plugin-9t629" Jul 24 08:55:41 bluth k3s[11030]: E0724 08:55:41.552880 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: d2ab53a2378f9efb632444f6575b781d98bb6f9560da139689f2e33bffa5bfc0" podSandboxID="d2ab53a2378f9efb632444f6575b781d98bb6f9560da139689f2e33bffa5bfc0" pod="ix-status-html/status-html-ix-chart-765b966694-c4nwz" Jul 24 08:55:41 bluth k3s[11030]: E0724 08:55:41.553774 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" podSandboxID="8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" pod="ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-5rpjh" Jul 24 08:55:41 bluth k3s[11030]: E0724 08:55:41.554591 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: d2ab53a2378f9efb632444f6575b781d98bb6f9560da139689f2e33bffa5bfc0" podSandboxID="d2ab53a2378f9efb632444f6575b781d98bb6f9560da139689f2e33bffa5bfc0" pod="ix-status-html/status-html-ix-chart-765b966694-c4nwz" Jul 24 08:55:41 bluth k3s[11030]: E0724 08:55:41.555525 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" podSandboxID="8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" pod="ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-5rpjh" Jul 24 08:55:41 bluth k3s[11030]: E0724 08:55:41.556474 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" podSandboxID="7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" pod="kube-system/intel-gpu-plugin-9t629" Jul 24 08:55:41 bluth k3s[11030]: E0724 08:55:41.749878 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:55:43 bluth k3s[11030]: E0724 08:55:43.756582 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:55:45 bluth k3s[11030]: I0724 08:55:45.453179 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="d2ab53a2378f9efb632444f6575b781d98bb6f9560da139689f2e33bffa5bfc0" Jul 24 08:55:45 bluth k3s[11030]: E0724 08:55:45.455313 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" podSandboxID="8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" pod="ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-5rpjh" Jul 24 08:55:45 bluth k3s[11030]: E0724 08:55:45.459760 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" podSandboxID="7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" pod="kube-system/intel-gpu-plugin-9t629" Jul 24 08:55:45 bluth k3s[11030]: E0724 08:55:45.462337 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" podSandboxID="8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" pod="ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-5rpjh" Jul 24 08:55:45 bluth k3s[11030]: E0724 08:55:45.463499 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" podSandboxID="7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" pod="kube-system/intel-gpu-plugin-9t629" Jul 24 08:55:45 bluth k3s[11030]: I0724 08:55:45.744892 11030 scope.go:110] "RemoveContainer" containerID="cb842219d9539846ec5a654aaa10e3cdd7dc39127be57f7a670c77c09fb23b95" Jul 24 08:55:45 bluth k3s[11030]: E0724 08:55:45.745441 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: cb842219d9539846ec5a654aaa10e3cdd7dc39127be57f7a670c77c09fb23b95" containerID="cb842219d9539846ec5a654aaa10e3cdd7dc39127be57f7a670c77c09fb23b95" Jul 24 08:55:45 bluth k3s[11030]: I0724 08:55:45.745478 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:cb842219d9539846ec5a654aaa10e3cdd7dc39127be57f7a670c77c09fb23b95} err="failed to get container status \"cb842219d9539846ec5a654aaa10e3cdd7dc39127be57f7a670c77c09fb23b95\": rpc error: code = Unknown desc = Error: No such container: cb842219d9539846ec5a654aaa10e3cdd7dc39127be57f7a670c77c09fb23b95" Jul 24 08:55:45 bluth k3s[11030]: I0724 08:55:45.745492 11030 scope.go:110] "RemoveContainer" containerID="b36856ea81b4de326bf77efba4b7623b64e883bdfb64f26e1b89ece06ae9bced" Jul 24 08:55:45 bluth k3s[11030]: E0724 08:55:45.745958 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: b36856ea81b4de326bf77efba4b7623b64e883bdfb64f26e1b89ece06ae9bced" containerID="b36856ea81b4de326bf77efba4b7623b64e883bdfb64f26e1b89ece06ae9bced" Jul 24 08:55:45 bluth k3s[11030]: I0724 08:55:45.745989 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:b36856ea81b4de326bf77efba4b7623b64e883bdfb64f26e1b89ece06ae9bced} err="failed to get container status \"b36856ea81b4de326bf77efba4b7623b64e883bdfb64f26e1b89ece06ae9bced\": rpc error: code = Unknown desc = Error: No such container: b36856ea81b4de326bf77efba4b7623b64e883bdfb64f26e1b89ece06ae9bced" Jul 24 08:55:45 bluth k3s[11030]: I0724 08:55:45.746002 11030 scope.go:110] "RemoveContainer" containerID="6ad0b3c8059d50b878addcfca6272e5cb2118b8128b92b8010fcef833e38ee3d" Jul 24 08:55:45 bluth k3s[11030]: E0724 08:55:45.746456 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 6ad0b3c8059d50b878addcfca6272e5cb2118b8128b92b8010fcef833e38ee3d" containerID="6ad0b3c8059d50b878addcfca6272e5cb2118b8128b92b8010fcef833e38ee3d" Jul 24 08:55:45 bluth k3s[11030]: I0724 08:55:45.746492 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:6ad0b3c8059d50b878addcfca6272e5cb2118b8128b92b8010fcef833e38ee3d} err="failed to get container status \"6ad0b3c8059d50b878addcfca6272e5cb2118b8128b92b8010fcef833e38ee3d\": rpc error: code = Unknown desc = Error: No such container: 6ad0b3c8059d50b878addcfca6272e5cb2118b8128b92b8010fcef833e38ee3d" Jul 24 08:55:45 bluth k3s[11030]: I0724 08:55:45.746505 11030 scope.go:110] "RemoveContainer" containerID="0a003958e535623bbc52b6093b87d3b4ac93e8a62b678ead19f052c74598d985" Jul 24 08:55:45 bluth k3s[11030]: E0724 08:55:45.754751 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:55:46 bluth k3s[11030]: E0724 08:55:46.485212 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" podSandboxID="8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" pod="ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-5rpjh" Jul 24 08:55:46 bluth k3s[11030]: E0724 08:55:46.486244 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" podSandboxID="7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" pod="kube-system/intel-gpu-plugin-9t629" Jul 24 08:55:46 bluth k3s[11030]: E0724 08:55:46.487367 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 5a3ea206b07a61417d9e43e50eb218b546d21d59a97ceb9e01bf6d7dbf565975" podSandboxID="5a3ea206b07a61417d9e43e50eb218b546d21d59a97ceb9e01bf6d7dbf565975" pod="ix-tdarr/svclb-tdarr-wfk5l" Jul 24 08:55:46 bluth k3s[11030]: E0724 08:55:46.488406 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" podSandboxID="8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" pod="ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-5rpjh" Jul 24 08:55:46 bluth k3s[11030]: E0724 08:55:46.489421 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" podSandboxID="7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" pod="kube-system/intel-gpu-plugin-9t629" Jul 24 08:55:47 bluth k3s[11030]: E0724 08:55:47.505124 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" podSandboxID="8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" pod="ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-5rpjh" Jul 24 08:55:47 bluth k3s[11030]: E0724 08:55:47.506088 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" podSandboxID="7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" pod="kube-system/intel-gpu-plugin-9t629" Jul 24 08:55:47 bluth k3s[11030]: E0724 08:55:47.507071 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 5a3ea206b07a61417d9e43e50eb218b546d21d59a97ceb9e01bf6d7dbf565975" podSandboxID="5a3ea206b07a61417d9e43e50eb218b546d21d59a97ceb9e01bf6d7dbf565975" pod="ix-tdarr/svclb-tdarr-wfk5l" Jul 24 08:55:47 bluth k3s[11030]: E0724 08:55:47.508041 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" podSandboxID="8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" pod="ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-5rpjh" Jul 24 08:55:47 bluth k3s[11030]: E0724 08:55:47.508953 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" podSandboxID="7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" pod="kube-system/intel-gpu-plugin-9t629" Jul 24 08:55:47 bluth k3s[11030]: E0724 08:55:47.509868 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 5a3ea206b07a61417d9e43e50eb218b546d21d59a97ceb9e01bf6d7dbf565975" podSandboxID="5a3ea206b07a61417d9e43e50eb218b546d21d59a97ceb9e01bf6d7dbf565975" pod="ix-tdarr/svclb-tdarr-wfk5l" Jul 24 08:55:47 bluth k3s[11030]: E0724 08:55:47.770602 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:55:48 bluth k3s[11030]: E0724 08:55:48.525120 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 5a3ea206b07a61417d9e43e50eb218b546d21d59a97ceb9e01bf6d7dbf565975" podSandboxID="5a3ea206b07a61417d9e43e50eb218b546d21d59a97ceb9e01bf6d7dbf565975" pod="ix-tdarr/svclb-tdarr-wfk5l" Jul 24 08:55:48 bluth k3s[11030]: E0724 08:55:48.526095 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" podSandboxID="8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" pod="ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-5rpjh" Jul 24 08:55:48 bluth k3s[11030]: E0724 08:55:48.527024 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" podSandboxID="7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" pod="kube-system/intel-gpu-plugin-9t629" Jul 24 08:55:48 bluth k3s[11030]: E0724 08:55:48.527919 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" podSandboxID="8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" pod="ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-5rpjh" Jul 24 08:55:48 bluth k3s[11030]: E0724 08:55:48.528818 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" podSandboxID="7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" pod="kube-system/intel-gpu-plugin-9t629" Jul 24 08:55:48 bluth k3s[11030]: E0724 08:55:48.529706 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 5a3ea206b07a61417d9e43e50eb218b546d21d59a97ceb9e01bf6d7dbf565975" podSandboxID="5a3ea206b07a61417d9e43e50eb218b546d21d59a97ceb9e01bf6d7dbf565975" pod="ix-tdarr/svclb-tdarr-wfk5l" Jul 24 08:55:49 bluth k3s[11030]: E0724 08:55:49.558011 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" podSandboxID="8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" pod="ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-5rpjh" Jul 24 08:55:49 bluth k3s[11030]: E0724 08:55:49.559088 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" podSandboxID="7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" pod="kube-system/intel-gpu-plugin-9t629" Jul 24 08:55:49 bluth k3s[11030]: E0724 08:55:49.559984 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 5a3ea206b07a61417d9e43e50eb218b546d21d59a97ceb9e01bf6d7dbf565975" podSandboxID="5a3ea206b07a61417d9e43e50eb218b546d21d59a97ceb9e01bf6d7dbf565975" pod="ix-tdarr/svclb-tdarr-wfk5l" Jul 24 08:55:49 bluth k3s[11030]: E0724 08:55:49.560792 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" podSandboxID="7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" pod="kube-system/intel-gpu-plugin-9t629" Jul 24 08:55:49 bluth k3s[11030]: E0724 08:55:49.561626 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 5a3ea206b07a61417d9e43e50eb218b546d21d59a97ceb9e01bf6d7dbf565975" podSandboxID="5a3ea206b07a61417d9e43e50eb218b546d21d59a97ceb9e01bf6d7dbf565975" pod="ix-tdarr/svclb-tdarr-wfk5l" Jul 24 08:55:49 bluth k3s[11030]: E0724 08:55:49.562426 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" podSandboxID="8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" pod="ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-5rpjh" Jul 24 08:55:49 bluth k3s[11030]: E0724 08:55:49.780690 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:55:50 bluth k3s[11030]: E0724 08:55:50.576980 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" podSandboxID="8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" pod="ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-5rpjh" Jul 24 08:55:50 bluth k3s[11030]: E0724 08:55:50.577842 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" podSandboxID="7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" pod="kube-system/intel-gpu-plugin-9t629" Jul 24 08:55:50 bluth k3s[11030]: E0724 08:55:50.578756 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 5a3ea206b07a61417d9e43e50eb218b546d21d59a97ceb9e01bf6d7dbf565975" podSandboxID="5a3ea206b07a61417d9e43e50eb218b546d21d59a97ceb9e01bf6d7dbf565975" pod="ix-tdarr/svclb-tdarr-wfk5l" Jul 24 08:55:50 bluth k3s[11030]: E0724 08:55:50.579629 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 5a3ea206b07a61417d9e43e50eb218b546d21d59a97ceb9e01bf6d7dbf565975" podSandboxID="5a3ea206b07a61417d9e43e50eb218b546d21d59a97ceb9e01bf6d7dbf565975" pod="ix-tdarr/svclb-tdarr-wfk5l" Jul 24 08:55:50 bluth k3s[11030]: E0724 08:55:50.580562 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" podSandboxID="8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" pod="ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-5rpjh" Jul 24 08:55:50 bluth k3s[11030]: E0724 08:55:50.581458 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" podSandboxID="7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" pod="kube-system/intel-gpu-plugin-9t629" Jul 24 08:55:51 bluth k3s[11030]: E0724 08:55:51.597376 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 5a3ea206b07a61417d9e43e50eb218b546d21d59a97ceb9e01bf6d7dbf565975" podSandboxID="5a3ea206b07a61417d9e43e50eb218b546d21d59a97ceb9e01bf6d7dbf565975" pod="ix-tdarr/svclb-tdarr-wfk5l" Jul 24 08:55:51 bluth k3s[11030]: E0724 08:55:51.598388 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" podSandboxID="8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" pod="ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-5rpjh" Jul 24 08:55:51 bluth k3s[11030]: E0724 08:55:51.599262 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" podSandboxID="7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" pod="kube-system/intel-gpu-plugin-9t629" Jul 24 08:55:51 bluth k3s[11030]: E0724 08:55:51.600136 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" podSandboxID="7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" pod="kube-system/intel-gpu-plugin-9t629" Jul 24 08:55:51 bluth k3s[11030]: E0724 08:55:51.601066 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 5a3ea206b07a61417d9e43e50eb218b546d21d59a97ceb9e01bf6d7dbf565975" podSandboxID="5a3ea206b07a61417d9e43e50eb218b546d21d59a97ceb9e01bf6d7dbf565975" pod="ix-tdarr/svclb-tdarr-wfk5l" Jul 24 08:55:51 bluth k3s[11030]: E0724 08:55:51.601976 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" podSandboxID="8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" pod="ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-5rpjh" Jul 24 08:55:51 bluth k3s[11030]: E0724 08:55:51.775547 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:55:52 bluth k3s[11030]: E0724 08:55:52.616872 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" podSandboxID="8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" pod="ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-5rpjh" Jul 24 08:55:52 bluth k3s[11030]: E0724 08:55:52.617791 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" podSandboxID="7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" pod="kube-system/intel-gpu-plugin-9t629" Jul 24 08:55:52 bluth k3s[11030]: E0724 08:55:52.618682 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 5a3ea206b07a61417d9e43e50eb218b546d21d59a97ceb9e01bf6d7dbf565975" podSandboxID="5a3ea206b07a61417d9e43e50eb218b546d21d59a97ceb9e01bf6d7dbf565975" pod="ix-tdarr/svclb-tdarr-wfk5l" Jul 24 08:55:52 bluth k3s[11030]: E0724 08:55:52.619596 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" podSandboxID="8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" pod="ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-5rpjh" Jul 24 08:55:52 bluth k3s[11030]: E0724 08:55:52.620480 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" podSandboxID="7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" pod="kube-system/intel-gpu-plugin-9t629" Jul 24 08:55:52 bluth k3s[11030]: E0724 08:55:52.621395 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 5a3ea206b07a61417d9e43e50eb218b546d21d59a97ceb9e01bf6d7dbf565975" podSandboxID="5a3ea206b07a61417d9e43e50eb218b546d21d59a97ceb9e01bf6d7dbf565975" pod="ix-tdarr/svclb-tdarr-wfk5l" Jul 24 08:55:53 bluth k3s[11030]: E0724 08:55:53.635864 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 5a3ea206b07a61417d9e43e50eb218b546d21d59a97ceb9e01bf6d7dbf565975" podSandboxID="5a3ea206b07a61417d9e43e50eb218b546d21d59a97ceb9e01bf6d7dbf565975" pod="ix-tdarr/svclb-tdarr-wfk5l" Jul 24 08:55:53 bluth k3s[11030]: E0724 08:55:53.636756 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" podSandboxID="8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" pod="ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-5rpjh" Jul 24 08:55:53 bluth k3s[11030]: E0724 08:55:53.637587 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" podSandboxID="7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" pod="kube-system/intel-gpu-plugin-9t629" Jul 24 08:55:53 bluth k3s[11030]: E0724 08:55:53.638391 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" podSandboxID="8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" pod="ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-5rpjh" Jul 24 08:55:53 bluth k3s[11030]: E0724 08:55:53.639319 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" podSandboxID="7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" pod="kube-system/intel-gpu-plugin-9t629" Jul 24 08:55:53 bluth k3s[11030]: E0724 08:55:53.640221 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 5a3ea206b07a61417d9e43e50eb218b546d21d59a97ceb9e01bf6d7dbf565975" podSandboxID="5a3ea206b07a61417d9e43e50eb218b546d21d59a97ceb9e01bf6d7dbf565975" pod="ix-tdarr/svclb-tdarr-wfk5l" Jul 24 08:55:53 bluth k3s[11030]: E0724 08:55:53.753988 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:55:54 bluth k3s[11030]: E0724 08:55:54.667214 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" podSandboxID="7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" pod="kube-system/intel-gpu-plugin-9t629" Jul 24 08:55:54 bluth k3s[11030]: E0724 08:55:54.668146 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 5a3ea206b07a61417d9e43e50eb218b546d21d59a97ceb9e01bf6d7dbf565975" podSandboxID="5a3ea206b07a61417d9e43e50eb218b546d21d59a97ceb9e01bf6d7dbf565975" pod="ix-tdarr/svclb-tdarr-wfk5l" Jul 24 08:55:54 bluth k3s[11030]: E0724 08:55:54.668954 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" podSandboxID="8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" pod="ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-5rpjh" Jul 24 08:55:54 bluth k3s[11030]: E0724 08:55:54.671730 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 5a3ea206b07a61417d9e43e50eb218b546d21d59a97ceb9e01bf6d7dbf565975" podSandboxID="5a3ea206b07a61417d9e43e50eb218b546d21d59a97ceb9e01bf6d7dbf565975" pod="ix-tdarr/svclb-tdarr-wfk5l" Jul 24 08:55:54 bluth k3s[11030]: E0724 08:55:54.672728 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" podSandboxID="8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" pod="ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-5rpjh" Jul 24 08:55:54 bluth k3s[11030]: E0724 08:55:54.673558 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" podSandboxID="7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" pod="kube-system/intel-gpu-plugin-9t629" Jul 24 08:55:55 bluth k3s[11030]: E0724 08:55:55.689938 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" podSandboxID="8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" pod="ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-5rpjh" Jul 24 08:55:55 bluth k3s[11030]: E0724 08:55:55.691514 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" podSandboxID="7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" pod="kube-system/intel-gpu-plugin-9t629" Jul 24 08:55:55 bluth k3s[11030]: E0724 08:55:55.692742 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 5a3ea206b07a61417d9e43e50eb218b546d21d59a97ceb9e01bf6d7dbf565975" podSandboxID="5a3ea206b07a61417d9e43e50eb218b546d21d59a97ceb9e01bf6d7dbf565975" pod="ix-tdarr/svclb-tdarr-wfk5l" Jul 24 08:55:55 bluth k3s[11030]: E0724 08:55:55.693989 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" podSandboxID="7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" pod="kube-system/intel-gpu-plugin-9t629" Jul 24 08:55:55 bluth k3s[11030]: E0724 08:55:55.695175 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 5a3ea206b07a61417d9e43e50eb218b546d21d59a97ceb9e01bf6d7dbf565975" podSandboxID="5a3ea206b07a61417d9e43e50eb218b546d21d59a97ceb9e01bf6d7dbf565975" pod="ix-tdarr/svclb-tdarr-wfk5l" Jul 24 08:55:55 bluth k3s[11030]: E0724 08:55:55.696321 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" podSandboxID="8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" pod="ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-5rpjh" Jul 24 08:55:55 bluth k3s[11030]: E0724 08:55:55.751640 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:55:56 bluth k3s[11030]: E0724 08:55:56.616676 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:57:00.616642935 +0100 IST m=+347.104499862 (durationBeforeRetry 1m4s). Error: MountVolume.MountDevice failed for volume "pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e") pod "esphome-658957fddc-6z2jm" (UID: "19254d20-7ad1-49a1-b408-f6e4213e5b4e") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:55:56 bluth k3s[11030]: E0724 08:55:56.711882 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" podSandboxID="7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" pod="kube-system/intel-gpu-plugin-9t629" Jul 24 08:55:56 bluth k3s[11030]: E0724 08:55:56.712896 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 5a3ea206b07a61417d9e43e50eb218b546d21d59a97ceb9e01bf6d7dbf565975" podSandboxID="5a3ea206b07a61417d9e43e50eb218b546d21d59a97ceb9e01bf6d7dbf565975" pod="ix-tdarr/svclb-tdarr-wfk5l" Jul 24 08:55:57 bluth k3s[11030]: E0724 08:55:57.424705 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-06521873-c179-499e-9079-194ec6618fdd podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:57:01.42466441 +0100 IST m=+347.912521360 (durationBeforeRetry 1m4s). Error: MountVolume.MountDevice failed for volume "pvc-06521873-c179-499e-9079-194ec6618fdd" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-06521873-c179-499e-9079-194ec6618fdd") pod "esphome-658957fddc-6z2jm" (UID: "19254d20-7ad1-49a1-b408-f6e4213e5b4e") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:55:57 bluth k3s[11030]: E0724 08:55:57.754984 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:55:58 bluth k3s[11030]: I0724 08:55:58.640049 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="8e1b627681d5fa37e63dd1e7b20c15bfe83d908d9aeba2e453ea449b960b7105" Jul 24 08:55:58 bluth k3s[11030]: E0724 08:55:58.641152 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" podSandboxID="7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" pod="kube-system/intel-gpu-plugin-9t629" Jul 24 08:55:58 bluth k3s[11030]: E0724 08:55:58.642054 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 5a3ea206b07a61417d9e43e50eb218b546d21d59a97ceb9e01bf6d7dbf565975" podSandboxID="5a3ea206b07a61417d9e43e50eb218b546d21d59a97ceb9e01bf6d7dbf565975" pod="ix-tdarr/svclb-tdarr-wfk5l" Jul 24 08:55:58 bluth k3s[11030]: E0724 08:55:58.835980 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:57:02.835969513 +0100 IST m=+349.323826395 (durationBeforeRetry 1m4s). Error: MountVolume.MountDevice failed for volume "pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274") pod "prowlarr-5bcc99cc47-v86hm" (UID: "5d1f2955-de8a-4e25-b1f8-e607463ed369") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:55:58 bluth k3s[11030]: I0724 08:55:58.936869 33433 event.go:278] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ix-cloudflared", Name:"cloudflared-ix-chart-687fff6d55-vzff2", UID:"3da0644e-8d9b-4f63-b8dc-0199c4f76027", APIVersion:"v1", ResourceVersion:"3027807", FieldPath:""}): type: 'Normal' reason: 'AddedInterface' Add eth0 [172.16.0.199/16] from ix-net Jul 24 08:55:59 bluth k3s[11030]: I0724 08:55:59.420002 33591 event.go:278] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ix-status-html", Name:"status-html-ix-chart-765b966694-c4nwz", UID:"290c61bb-9f19-47b9-b548-312e738a0bf1", APIVersion:"v1", ResourceVersion:"3027820", FieldPath:""}): type: 'Normal' reason: 'AddedInterface' Add eth0 [172.16.0.200/16] from ix-net Jul 24 08:55:59 bluth k3s[11030]: E0724 08:55:59.672680 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: fea1a248d7ad018b21810bedfe925cf9142335f5f24ca14ecd4ddc57f86e10bc" podSandboxID="fea1a248d7ad018b21810bedfe925cf9142335f5f24ca14ecd4ddc57f86e10bc" pod="ix-qbittorrent/svclb-qbittorrent-torrent-dnxlw" Jul 24 08:55:59 bluth k3s[11030]: E0724 08:55:59.782264 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:55:59 bluth k3s[11030]: E0724 08:55:59.845391 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-58562e12-4adc-4e83-8131-848f1b39bf00 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:57:03.845357313 +0100 IST m=+350.333214234 (durationBeforeRetry 1m4s). Error: MountVolume.MountDevice failed for volume "pvc-58562e12-4adc-4e83-8131-848f1b39bf00" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-58562e12-4adc-4e83-8131-848f1b39bf00") pod "qbittorrent-6d7cf9f7cd-xrqwj" (UID: "4d43a190-0b41-4f84-a84d-ce295694fcb5") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:56:00 bluth k3s[11030]: I0724 08:56:00.759959 11030 scope.go:110] "RemoveContainer" containerID="593b08e726c64116817fc9358b157c421305965ed78d93958b8594decf870ad8" Jul 24 08:56:01 bluth k3s[11030]: E0724 08:56:01.054636 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-195f1a32-9c93-4737-983a-8dacee32e453 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:57:05.054601775 +0100 IST m=+351.542458699 (durationBeforeRetry 1m4s). Error: MountVolume.MountDevice failed for volume "pvc-195f1a32-9c93-4737-983a-8dacee32e453" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-195f1a32-9c93-4737-983a-8dacee32e453") pod "radarr-7486f78d7f-ntt44" (UID: "9ab9d8fe-e669-40bb-a0ce-7f7a4e5a69b5") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:56:01 bluth k3s[11030]: I0724 08:56:01.300890 33860 event.go:278] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ix-home-assistant", Name:"home-assistant-ix-chart-54bf5d89c8-5rpjh", UID:"3f2c328d-b981-469a-b3d6-2850e53c7d15", APIVersion:"v1", ResourceVersion:"3027827", FieldPath:""}): type: 'Normal' reason: 'AddedInterface' Add eth0 [172.16.0.201/16] from ix-net Jul 24 08:56:01 bluth k3s[11030]: I0724 08:56:01.661149 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="7a66335647d911a823727782c79872817c7549a1b206c4d4fdf995299bc3c285" Jul 24 08:56:01 bluth k3s[11030]: E0724 08:56:01.664989 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 8327e670f7dab96988235d261712e588550d6cbf333e5018ff3ed3a1350a66d9" podSandboxID="8327e670f7dab96988235d261712e588550d6cbf333e5018ff3ed3a1350a66d9" pod="ix-sonarr/svclb-sonarr-7tc6n" Jul 24 08:56:01 bluth k3s[11030]: E0724 08:56:01.756813 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:56:01 bluth k3s[11030]: I0724 08:56:01.783276 34069 event.go:278] Event(v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"intel-gpu-plugin-9t629", UID:"a93d958f-5cba-4480-8ca8-61ca64464a2b", APIVersion:"v1", ResourceVersion:"3027916", FieldPath:""}): type: 'Normal' reason: 'AddedInterface' Add eth0 [172.16.0.202/16] from ix-net Jul 24 08:56:02 bluth k3s[11030]: E0724 08:56:02.062272 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:57:06.062238976 +0100 IST m=+352.550095901 (durationBeforeRetry 1m4s). Error: MountVolume.MountDevice failed for volume "pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4") pod "sonarr-867d7dd548-fw6z4" (UID: "1a1846b0-3361-4f4b-a1cd-d5f958d6a7ac") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:56:03 bluth k3s[11030]: E0724 08:56:03.757179 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:56:05 bluth k3s[11030]: E0724 08:56:05.754372 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:56:06 bluth k3s[11030]: I0724 08:56:06.942823 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="5a3ea206b07a61417d9e43e50eb218b546d21d59a97ceb9e01bf6d7dbf565975" Jul 24 08:56:07 bluth k3s[11030]: I0724 08:56:07.087537 34263 event.go:278] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ix-tdarr", Name:"svclb-tdarr-wfk5l", UID:"2100e768-75ce-47f0-a8d1-dcf7d76214c1", APIVersion:"v1", ResourceVersion:"3027922", FieldPath:""}): type: 'Normal' reason: 'AddedInterface' Add eth0 [172.16.0.203/16] from ix-net Jul 24 08:56:07 bluth k3s[11030]: I0724 08:56:07.542983 11030 scope.go:110] "RemoveContainer" containerID="0a003958e535623bbc52b6093b87d3b4ac93e8a62b678ead19f052c74598d985" Jul 24 08:56:07 bluth k3s[11030]: E0724 08:56:07.543479 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 0a003958e535623bbc52b6093b87d3b4ac93e8a62b678ead19f052c74598d985" containerID="0a003958e535623bbc52b6093b87d3b4ac93e8a62b678ead19f052c74598d985" Jul 24 08:56:07 bluth k3s[11030]: I0724 08:56:07.543499 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:0a003958e535623bbc52b6093b87d3b4ac93e8a62b678ead19f052c74598d985} err="failed to get container status \"0a003958e535623bbc52b6093b87d3b4ac93e8a62b678ead19f052c74598d985\": rpc error: code = Unknown desc = Error: No such container: 0a003958e535623bbc52b6093b87d3b4ac93e8a62b678ead19f052c74598d985" Jul 24 08:56:07 bluth k3s[11030]: I0724 08:56:07.543529 11030 scope.go:110] "RemoveContainer" containerID="593b08e726c64116817fc9358b157c421305965ed78d93958b8594decf870ad8" Jul 24 08:56:07 bluth k3s[11030]: E0724 08:56:07.543802 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 593b08e726c64116817fc9358b157c421305965ed78d93958b8594decf870ad8" containerID="593b08e726c64116817fc9358b157c421305965ed78d93958b8594decf870ad8" Jul 24 08:56:07 bluth k3s[11030]: I0724 08:56:07.543825 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:593b08e726c64116817fc9358b157c421305965ed78d93958b8594decf870ad8} err="failed to get container status \"593b08e726c64116817fc9358b157c421305965ed78d93958b8594decf870ad8\": rpc error: code = Unknown desc = Error: No such container: 593b08e726c64116817fc9358b157c421305965ed78d93958b8594decf870ad8" Jul 24 08:56:07 bluth k3s[11030]: I0724 08:56:07.543834 11030 scope.go:110] "RemoveContainer" containerID="f0c517a8f8e10361e1a200c82e3452308e4a1d821ffd44d7f03d256c49220965" Jul 24 08:56:07 bluth k3s[11030]: E0724 08:56:07.785144 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:56:08 bluth k3s[11030]: E0724 08:56:08.024748 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 72ac180a3a68d798f24195ff9f7f801b1dbb04a1566caadc0e7e6cf4c30013a2" containerID="72ac180a3a68d798f24195ff9f7f801b1dbb04a1566caadc0e7e6cf4c30013a2" Jul 24 08:56:08 bluth k3s[11030]: E0724 08:56:08.024823 11030 kuberuntime_manager.go:1079] "getPodContainerStatuses for pod failed" err="rpc error: code = Unknown desc = Error: No such container: 72ac180a3a68d798f24195ff9f7f801b1dbb04a1566caadc0e7e6cf4c30013a2" pod="kube-system/openebs-zfs-controller-0" Jul 24 08:56:09 bluth k3s[11030]: I0724 08:56:09.201103 34600 event.go:278] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ix-qbittorrent", Name:"svclb-qbittorrent-torrent-dnxlw", UID:"7f1d5d70-8421-4009-8f25-961daaa8e3d5", APIVersion:"v1", ResourceVersion:"3027920", FieldPath:""}): type: 'Normal' reason: 'AddedInterface' Add eth0 [172.16.0.204/16] from ix-net Jul 24 08:56:09 bluth k3s[11030]: E0724 08:56:09.785608 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:56:11 bluth k3s[11030]: E0724 08:56:11.753949 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:56:13 bluth k3s[11030]: E0724 08:56:13.784807 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:56:14 bluth k3s[11030]: I0724 08:56:14.162780 11030 scope.go:110] "RemoveContainer" containerID="b5673a188b5939ae2275996f1ff184c56af3f2cf2d5355d01362cea1844f77d7" Jul 24 08:56:14 bluth k3s[11030]: I0724 08:56:14.421731 11030 scope.go:110] "RemoveContainer" containerID="a3f6bf7a95d792418db2f78fd5ce4e1cbfee8c7ae330cd5e9d986e067496a6ee" Jul 24 08:56:15 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T08:56:15.069+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 08:56:15 bluth k3s[11030]: I0724 08:56:15.365268 35080 event.go:278] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ix-sonarr", Name:"svclb-sonarr-7tc6n", UID:"2e654fc3-6d15-4a35-8d03-8cbafa8b8fed", APIVersion:"v1", ResourceVersion:"3027955", FieldPath:""}): type: 'Normal' reason: 'AddedInterface' Add eth0 [172.16.0.205/16] from ix-net Jul 24 08:56:15 bluth k3s[11030]: E0724 08:56:15.787006 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:56:16 bluth k3s[11030]: I0724 08:56:16.129365 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="eae41f6d8f993730dd20879891c4b79043e8319f34ad4cedfa2779a402fb15e8" Jul 24 08:56:16 bluth k3s[11030]: I0724 08:56:16.148012 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="e231470b1aefa57b927ed13dbbde3c4cfdd9c599a6f921c2667f980ecfbd5d11" Jul 24 08:56:16 bluth k3s[11030]: I0724 08:56:16.160193 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="fea1a248d7ad018b21810bedfe925cf9142335f5f24ca14ecd4ddc57f86e10bc" Jul 24 08:56:16 bluth k3s[11030]: I0724 08:56:16.204766 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="8327e670f7dab96988235d261712e588550d6cbf333e5018ff3ed3a1350a66d9" Jul 24 08:56:16 bluth k3s[11030]: I0724 08:56:16.264216 35266 event.go:278] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ix-esphome", Name:"svclb-esphome-5q48s", UID:"e1058b53-0fed-46bf-aaf5-9bf8634f0d0a", APIVersion:"v1", ResourceVersion:"3027918", FieldPath:""}): type: 'Normal' reason: 'AddedInterface' Add eth0 [172.16.0.206/16] from ix-net Jul 24 08:56:16 bluth k3s[11030]: E0724 08:56:16.470241 11030 authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, service account token has been invalidated]" Jul 24 08:56:16 bluth k3s[11030]: I0724 08:56:16.747553 11030 scope.go:110] "RemoveContainer" containerID="7bd09f76b5bfee02f08e22c7318308026bbabb9b61bb3152a523e129bae2c9e1" Jul 24 08:56:17 bluth k3s[11030]: E0724 08:56:17.237928 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 44921f08567c8c6af040f5c0687060ad8346dcf0b26e33af1ceff8496146986b" containerID="44921f08567c8c6af040f5c0687060ad8346dcf0b26e33af1ceff8496146986b" Jul 24 08:56:17 bluth k3s[11030]: E0724 08:56:17.237960 11030 kuberuntime_manager.go:1079] "getPodContainerStatuses for pod failed" err="rpc error: code = Unknown desc = Error: No such container: 44921f08567c8c6af040f5c0687060ad8346dcf0b26e33af1ceff8496146986b" pod="ix-qbittorrent/svclb-qbittorrent-7jf9d" Jul 24 08:56:17 bluth k3s[11030]: E0724 08:56:17.247248 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: d28a4b1d968b93a5e0080167e275a48923dbab461a0f8381213b1fb4b767838a" containerID="d28a4b1d968b93a5e0080167e275a48923dbab461a0f8381213b1fb4b767838a" Jul 24 08:56:17 bluth k3s[11030]: E0724 08:56:17.247282 11030 kuberuntime_manager.go:1079] "getPodContainerStatuses for pod failed" err="rpc error: code = Unknown desc = Error: No such container: d28a4b1d968b93a5e0080167e275a48923dbab461a0f8381213b1fb4b767838a" pod="ix-prowlarr/svclb-prowlarr-jgkkl" Jul 24 08:56:17 bluth k3s[11030]: E0724 08:56:17.765566 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:56:19 bluth k3s[11030]: I0724 08:56:19.411873 11030 scope.go:110] "RemoveContainer" containerID="f0c517a8f8e10361e1a200c82e3452308e4a1d821ffd44d7f03d256c49220965" Jul 24 08:56:19 bluth k3s[11030]: E0724 08:56:19.412920 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: f0c517a8f8e10361e1a200c82e3452308e4a1d821ffd44d7f03d256c49220965" containerID="f0c517a8f8e10361e1a200c82e3452308e4a1d821ffd44d7f03d256c49220965" Jul 24 08:56:19 bluth k3s[11030]: I0724 08:56:19.412996 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:f0c517a8f8e10361e1a200c82e3452308e4a1d821ffd44d7f03d256c49220965} err="failed to get container status \"f0c517a8f8e10361e1a200c82e3452308e4a1d821ffd44d7f03d256c49220965\": rpc error: code = Unknown desc = Error: No such container: f0c517a8f8e10361e1a200c82e3452308e4a1d821ffd44d7f03d256c49220965" Jul 24 08:56:19 bluth k3s[11030]: I0724 08:56:19.413029 11030 scope.go:110] "RemoveContainer" containerID="a3f6bf7a95d792418db2f78fd5ce4e1cbfee8c7ae330cd5e9d986e067496a6ee" Jul 24 08:56:19 bluth k3s[11030]: E0724 08:56:19.413948 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: a3f6bf7a95d792418db2f78fd5ce4e1cbfee8c7ae330cd5e9d986e067496a6ee" containerID="a3f6bf7a95d792418db2f78fd5ce4e1cbfee8c7ae330cd5e9d986e067496a6ee" Jul 24 08:56:19 bluth k3s[11030]: I0724 08:56:19.414021 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:a3f6bf7a95d792418db2f78fd5ce4e1cbfee8c7ae330cd5e9d986e067496a6ee} err="failed to get container status \"a3f6bf7a95d792418db2f78fd5ce4e1cbfee8c7ae330cd5e9d986e067496a6ee\": rpc error: code = Unknown desc = Error: No such container: a3f6bf7a95d792418db2f78fd5ce4e1cbfee8c7ae330cd5e9d986e067496a6ee" Jul 24 08:56:19 bluth k3s[11030]: I0724 08:56:19.414059 11030 scope.go:110] "RemoveContainer" containerID="d175307b98d3f21c64114da633d5e59ab4473eca8985300d01b7a1a7880b36b4" Jul 24 08:56:19 bluth k3s[11030]: E0724 08:56:19.775625 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:56:20 bluth k3s[11030]: I0724 08:56:20.360034 11030 trace.go:205] Trace[821339753]: "Create" url:/api/v1/namespaces/ix-prowlarr/events,user-agent:k3s/v1.23.5+k3s (linux/amd64) kubernetes/fbfa51e,audit-id:43abba1d-d66f-4d04-9177-f12639a30417,client:127.0.0.1,accept:application/vnd.kubernetes.protobuf,application/json,protocol:HTTP/2.0 (24-Jul-2022 08:56:19.791) (total time: 568ms): Jul 24 08:56:20 bluth k3s[11030]: Trace[821339753]: ---"Object stored in database" 568ms (08:56:20.359) Jul 24 08:56:20 bluth k3s[11030]: Trace[821339753]: [568.525846ms] [568.525846ms] END Jul 24 08:56:21 bluth k3s[11030]: E0724 08:56:21.758556 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:56:23 bluth k3s[11030]: E0724 08:56:23.754425 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:56:25 bluth k3s[11030]: E0724 08:56:25.102122 11030 authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, service account token has been invalidated]" Jul 24 08:56:25 bluth k3s[11030]: E0724 08:56:25.770370 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:56:25 bluth k3s[11030]: E0724 08:56:25.804328 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 523ceafc9b5164b15c0dbafaca8a3d52ccbc3c4d58313531826d61e47ff5fb31" containerID="523ceafc9b5164b15c0dbafaca8a3d52ccbc3c4d58313531826d61e47ff5fb31" Jul 24 08:56:25 bluth k3s[11030]: E0724 08:56:25.804353 11030 kuberuntime_manager.go:1079] "getPodContainerStatuses for pod failed" err="rpc error: code = Unknown desc = Error: No such container: 523ceafc9b5164b15c0dbafaca8a3d52ccbc3c4d58313531826d61e47ff5fb31" pod="ix-qbittorrent/svclb-qbittorrent-torrentudp-wjbmf" Jul 24 08:56:25 bluth k3s[11030]: time="2022-07-24T08:56:25+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-prowlarr\", Name:\"prowlarr\", UID:\"bd43faf9-73db-4bcd-a678-c673a7b54098\", APIVersion:\"v1\", ResourceVersion:\"3027702\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-prowlarr/svclb-prowlarr" Jul 24 08:56:25 bluth k3s[11030]: time="2022-07-24T08:56:25+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-prowlarr\", Name:\"prowlarr\", UID:\"bd43faf9-73db-4bcd-a678-c673a7b54098\", APIVersion:\"v1\", ResourceVersion:\"3027702\", FieldPath:\"\"}): type: 'Normal' reason: 'UpdatedIngressIP' LoadBalancer Ingress IP addresses updated: 192.168.0.143" Jul 24 08:56:25 bluth k3s[11030]: time="2022-07-24T08:56:25+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-prowlarr\", Name:\"prowlarr\", UID:\"bd43faf9-73db-4bcd-a678-c673a7b54098\", APIVersion:\"v1\", ResourceVersion:\"3028038\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-prowlarr/svclb-prowlarr" Jul 24 08:56:27 bluth k3s[11030]: E0724 08:56:27.765719 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:56:28 bluth k3s[11030]: E0724 08:56:28.428231 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 0cd5f49ae11febcd1414a0d6eb3a5d1f40fc8dcbfecc75d7ac508d564eda1ea5" containerID="0cd5f49ae11febcd1414a0d6eb3a5d1f40fc8dcbfecc75d7ac508d564eda1ea5" Jul 24 08:56:28 bluth k3s[11030]: E0724 08:56:28.428254 11030 kuberuntime_manager.go:1079] "getPodContainerStatuses for pod failed" err="rpc error: code = Unknown desc = Error: No such container: 0cd5f49ae11febcd1414a0d6eb3a5d1f40fc8dcbfecc75d7ac508d564eda1ea5" pod="ix-radarr/svclb-radarr-s72px" Jul 24 08:56:29 bluth k3s[11030]: I0724 08:56:29.740097 11030 scope.go:110] "RemoveContainer" containerID="8ccd98bfe3f8ac373925d4c0f4d05f2989800d69d62f7e4413e1c55054f6c9ad" Jul 24 08:56:29 bluth k3s[11030]: I0724 08:56:29.740904 11030 scope.go:110] "RemoveContainer" containerID="602dfb16f999ff127989af33a2fd17554b8c38fa5f34a84a168f19ab9f776d50" Jul 24 08:56:29 bluth k3s[11030]: E0724 08:56:29.778371 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:56:31 bluth k3s[11030]: E0724 08:56:31.378370 11030 manager.go:558] "Unexpected: unhealthyDevices and endpoints are out of sync" Jul 24 08:56:31 bluth k3s[11030]: E0724 08:56:31.750657 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:56:33 bluth k3s[11030]: I0724 08:56:33.451297 11030 docker_sandbox.go:402] "Failed to read pod IP from plugin/docker" err="networkPlugin cni failed on the status hook for pod \"esphome-658957fddc-bbcxl_ix-esphome\": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container \"6d2d1f41beed8c6cb0a510daf97a2b869656d3eb1f69580c3e40b7d603a6913d\"" Jul 24 08:56:33 bluth k3s[11030]: E0724 08:56:33.758537 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:56:34 bluth k3s[11030]: time="2022-07-24T08:56:34+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-qbittorrent\", Name:\"qbittorrent-torrentudp\", UID:\"1d92e74f-b912-46f1-b780-c740f80bb9ca\", APIVersion:\"v1\", ResourceVersion:\"3027754\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-qbittorrent/svclb-qbittorrent-torrentudp" Jul 24 08:56:34 bluth k3s[11030]: time="2022-07-24T08:56:34+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-qbittorrent\", Name:\"qbittorrent-torrentudp\", UID:\"1d92e74f-b912-46f1-b780-c740f80bb9ca\", APIVersion:\"v1\", ResourceVersion:\"3027754\", FieldPath:\"\"}): type: 'Normal' reason: 'UpdatedIngressIP' LoadBalancer Ingress IP addresses updated: 192.168.0.143" Jul 24 08:56:34 bluth k3s[11030]: time="2022-07-24T08:56:34+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-qbittorrent\", Name:\"qbittorrent-torrentudp\", UID:\"1d92e74f-b912-46f1-b780-c740f80bb9ca\", APIVersion:\"v1\", ResourceVersion:\"3028051\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-qbittorrent/svclb-qbittorrent-torrentudp" Jul 24 08:56:34 bluth k3s[11030]: time="2022-07-24T08:56:34+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-qbittorrent\", Name:\"qbittorrent\", UID:\"728a4977-c72e-46a0-96f2-a9279b009171\", APIVersion:\"v1\", ResourceVersion:\"3027645\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-qbittorrent/svclb-qbittorrent" Jul 24 08:56:35 bluth k3s[11030]: E0724 08:56:35.747749 11030 authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, service account token has been invalidated]" Jul 24 08:56:35 bluth k3s[11030]: E0724 08:56:35.758934 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:56:36 bluth k3s[11030]: E0724 08:56:36.548491 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: c599d1c98bbb1644aaf56dd0d47dee0e560bd9d280c55ba21f450e17641fcb15" containerID="c599d1c98bbb1644aaf56dd0d47dee0e560bd9d280c55ba21f450e17641fcb15" Jul 24 08:56:36 bluth k3s[11030]: E0724 08:56:36.548514 11030 kuberuntime_manager.go:1079] "getPodContainerStatuses for pod failed" err="rpc error: code = Unknown desc = Error: No such container: c599d1c98bbb1644aaf56dd0d47dee0e560bd9d280c55ba21f450e17641fcb15" pod="ix-cloudflared/cloudflared-ix-chart-687fff6d55-vzff2" Jul 24 08:56:36 bluth k3s[11030]: E0724 08:56:36.555956 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 108189c0f6504f4cd066364f56498fd77c6005de0cc77fb50ca5d11d1b2d05f8" containerID="108189c0f6504f4cd066364f56498fd77c6005de0cc77fb50ca5d11d1b2d05f8" Jul 24 08:56:36 bluth k3s[11030]: E0724 08:56:36.555983 11030 kuberuntime_manager.go:1079] "getPodContainerStatuses for pod failed" err="rpc error: code = Unknown desc = Error: No such container: 108189c0f6504f4cd066364f56498fd77c6005de0cc77fb50ca5d11d1b2d05f8" pod="ix-status-html/status-html-ix-chart-765b966694-c4nwz" Jul 24 08:56:37 bluth k3s[11030]: E0724 08:56:37.754796 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:56:39 bluth k3s[11030]: E0724 08:56:39.759718 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:56:41 bluth k3s[11030]: E0724 08:56:41.751573 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:56:43 bluth k3s[11030]: E0724 08:56:43.759142 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:56:44 bluth k3s[11030]: E0724 08:56:44.737196 11030 authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, service account token has been invalidated]" Jul 24 08:56:45 bluth k3s[11030]: E0724 08:56:45.759174 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:56:46 bluth k3s[11030]: I0724 08:56:46.731167 11030 scope.go:110] "RemoveContainer" containerID="d175307b98d3f21c64114da633d5e59ab4473eca8985300d01b7a1a7880b36b4" Jul 24 08:56:46 bluth k3s[11030]: E0724 08:56:46.732285 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: d175307b98d3f21c64114da633d5e59ab4473eca8985300d01b7a1a7880b36b4" containerID="d175307b98d3f21c64114da633d5e59ab4473eca8985300d01b7a1a7880b36b4" Jul 24 08:56:46 bluth k3s[11030]: I0724 08:56:46.732361 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:d175307b98d3f21c64114da633d5e59ab4473eca8985300d01b7a1a7880b36b4} err="failed to get container status \"d175307b98d3f21c64114da633d5e59ab4473eca8985300d01b7a1a7880b36b4\": rpc error: code = Unknown desc = Error: No such container: d175307b98d3f21c64114da633d5e59ab4473eca8985300d01b7a1a7880b36b4" Jul 24 08:56:46 bluth k3s[11030]: I0724 08:56:46.732391 11030 scope.go:110] "RemoveContainer" containerID="8ccd98bfe3f8ac373925d4c0f4d05f2989800d69d62f7e4413e1c55054f6c9ad" Jul 24 08:56:46 bluth k3s[11030]: E0724 08:56:46.733430 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 8ccd98bfe3f8ac373925d4c0f4d05f2989800d69d62f7e4413e1c55054f6c9ad" containerID="8ccd98bfe3f8ac373925d4c0f4d05f2989800d69d62f7e4413e1c55054f6c9ad" Jul 24 08:56:46 bluth k3s[11030]: I0724 08:56:46.733495 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:8ccd98bfe3f8ac373925d4c0f4d05f2989800d69d62f7e4413e1c55054f6c9ad} err="failed to get container status \"8ccd98bfe3f8ac373925d4c0f4d05f2989800d69d62f7e4413e1c55054f6c9ad\": rpc error: code = Unknown desc = Error: No such container: 8ccd98bfe3f8ac373925d4c0f4d05f2989800d69d62f7e4413e1c55054f6c9ad" Jul 24 08:56:46 bluth k3s[11030]: I0724 08:56:46.733521 11030 scope.go:110] "RemoveContainer" containerID="fe7bdbe3401af33e1d6685ff8499d4e2ac6f1cdf9c60e748dc590c5b44b791af" Jul 24 08:56:46 bluth k3s[11030]: I0724 08:56:46.741725 11030 scope.go:110] "RemoveContainer" containerID="44e4853c5f38ff2233bc667e7c45189d65e49a86f2eba2e69d6ff0dea9f690cf" Jul 24 08:56:46 bluth k3s[11030]: I0724 08:56:46.970436 11030 docker_sandbox.go:402] "Failed to read pod IP from plugin/docker" err="networkPlugin cni failed on the status hook for pod \"esphome-658957fddc-bbcxl_ix-esphome\": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container \"6d2d1f41beed8c6cb0a510daf97a2b869656d3eb1f69580c3e40b7d603a6913d\"" Jul 24 08:56:47 bluth k3s[11030]: E0724 08:56:47.754439 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:56:48 bluth k3s[11030]: E0724 08:56:48.018950 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 6747bde1f445bd7665e08586399d438f4aa4bac4d41d7edc4ab42862e7a2fbb6" containerID="6747bde1f445bd7665e08586399d438f4aa4bac4d41d7edc4ab42862e7a2fbb6" Jul 24 08:56:48 bluth k3s[11030]: E0724 08:56:48.019001 11030 kuberuntime_manager.go:1079] "getPodContainerStatuses for pod failed" err="rpc error: code = Unknown desc = Error: No such container: 6747bde1f445bd7665e08586399d438f4aa4bac4d41d7edc4ab42862e7a2fbb6" pod="kube-system/intel-gpu-plugin-9t629" Jul 24 08:56:48 bluth k3s[11030]: time="2022-07-24T08:56:48+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-radarr\", Name:\"radarr\", UID:\"2ceba47d-fa27-462f-9816-af4352781795\", APIVersion:\"v1\", ResourceVersion:\"3027740\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-radarr/svclb-radarr" Jul 24 08:56:48 bluth k3s[11030]: time="2022-07-24T08:56:48+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-radarr\", Name:\"radarr\", UID:\"2ceba47d-fa27-462f-9816-af4352781795\", APIVersion:\"v1\", ResourceVersion:\"3027740\", FieldPath:\"\"}): type: 'Normal' reason: 'UpdatedIngressIP' LoadBalancer Ingress IP addresses updated: 192.168.0.143" Jul 24 08:56:48 bluth k3s[11030]: time="2022-07-24T08:56:48+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-radarr\", Name:\"radarr\", UID:\"2ceba47d-fa27-462f-9816-af4352781795\", APIVersion:\"v1\", ResourceVersion:\"3028067\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-radarr/svclb-radarr" Jul 24 08:56:49 bluth k3s[11030]: E0724 08:56:49.758584 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:56:51 bluth k3s[11030]: E0724 08:56:51.759712 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:56:52 bluth k3s[11030]: E0724 08:56:52.366209 11030 authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, service account token has been invalidated]" Jul 24 08:56:53 bluth k3s[11030]: E0724 08:56:53.755179 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:56:54 bluth k3s[11030]: E0724 08:56:54.012305 11030 kubelet.go:1730] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[config], unattached volumes=[kube-api-access-m8kjp config shared temp varlogs list-0]: timed out waiting for the condition" pod="ix-sonarr/sonarr-867d7dd548-fw6z4" Jul 24 08:56:54 bluth k3s[11030]: E0724 08:56:54.012386 11030 pod_workers.go:949] "Error syncing pod, skipping" err="unmounted volumes=[config], unattached volumes=[kube-api-access-m8kjp config shared temp varlogs list-0]: timed out waiting for the condition" pod="ix-sonarr/sonarr-867d7dd548-fw6z4" podUID=1a1846b0-3361-4f4b-a1cd-d5f958d6a7ac Jul 24 08:56:54 bluth k3s[11030]: E0724 08:56:54.012971 11030 kubelet.go:1730] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[config], unattached volumes=[kube-api-access-4wqqq config shared temp varlogs list-0]: timed out waiting for the condition" pod="ix-radarr/radarr-7486f78d7f-ntt44" Jul 24 08:56:54 bluth k3s[11030]: E0724 08:56:54.013005 11030 kubelet.go:1730] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[config], unattached volumes=[kube-api-access-c8v9k config shared temp varlogs]: timed out waiting for the condition" pod="ix-prowlarr/prowlarr-5bcc99cc47-v86hm" Jul 24 08:56:54 bluth k3s[11030]: E0724 08:56:54.013032 11030 pod_workers.go:949] "Error syncing pod, skipping" err="unmounted volumes=[config], unattached volumes=[kube-api-access-4wqqq config shared temp varlogs list-0]: timed out waiting for the condition" pod="ix-radarr/radarr-7486f78d7f-ntt44" podUID=9ab9d8fe-e669-40bb-a0ce-7f7a4e5a69b5 Jul 24 08:56:54 bluth k3s[11030]: E0724 08:56:54.013069 11030 pod_workers.go:949] "Error syncing pod, skipping" err="unmounted volumes=[config], unattached volumes=[kube-api-access-c8v9k config shared temp varlogs]: timed out waiting for the condition" pod="ix-prowlarr/prowlarr-5bcc99cc47-v86hm" podUID=5d1f2955-de8a-4e25-b1f8-e607463ed369 Jul 24 08:56:54 bluth k3s[11030]: E0724 08:56:54.013245 11030 kubelet.go:1730] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[config platformio], unattached volumes=[varlogs kube-api-access-pq65c config platformio shared temp]: timed out waiting for the condition" pod="ix-esphome/esphome-658957fddc-6z2jm" Jul 24 08:56:54 bluth k3s[11030]: E0724 08:56:54.013300 11030 pod_workers.go:949] "Error syncing pod, skipping" err="unmounted volumes=[config platformio], unattached volumes=[varlogs kube-api-access-pq65c config platformio shared temp]: timed out waiting for the condition" pod="ix-esphome/esphome-658957fddc-6z2jm" podUID=19254d20-7ad1-49a1-b408-f6e4213e5b4e Jul 24 08:56:54 bluth k3s[11030]: E0724 08:56:54.014902 11030 kubelet.go:1730] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[config], unattached volumes=[list-0 kube-api-access-df98j config shared temp varlogs]: timed out waiting for the condition" pod="ix-qbittorrent/qbittorrent-6d7cf9f7cd-xrqwj" Jul 24 08:56:54 bluth k3s[11030]: E0724 08:56:54.014974 11030 pod_workers.go:949] "Error syncing pod, skipping" err="unmounted volumes=[config], unattached volumes=[list-0 kube-api-access-df98j config shared temp varlogs]: timed out waiting for the condition" pod="ix-qbittorrent/qbittorrent-6d7cf9f7cd-xrqwj" podUID=4d43a190-0b41-4f84-a84d-ce295694fcb5 Jul 24 08:56:55 bluth k3s[11030]: I0724 08:56:55.356603 11030 scope.go:110] "RemoveContainer" containerID="cfbb792152fd6498db6ad8ad88a72518ac6b5013489a020de4a2d04048f78e82" Jul 24 08:56:55 bluth k3s[11030]: I0724 08:56:55.575954 11030 scope.go:110] "RemoveContainer" containerID="bfef1aa6ad73cb573f4e507be48e8aef2f4475e7a8b4cda4662e785ea8b6e243" Jul 24 08:56:55 bluth k3s[11030]: E0724 08:56:55.787546 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:56:56 bluth k3s[11030]: E0724 08:56:56.615433 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: f82be99136f8c9e37c860463ea500e5c2b441f410181c81d70313993c2ebc875" containerID="f82be99136f8c9e37c860463ea500e5c2b441f410181c81d70313993c2ebc875" Jul 24 08:56:56 bluth k3s[11030]: E0724 08:56:56.615456 11030 kuberuntime_manager.go:1079] "getPodContainerStatuses for pod failed" err="rpc error: code = Unknown desc = Error: No such container: f82be99136f8c9e37c860463ea500e5c2b441f410181c81d70313993c2ebc875" pod="ix-tdarr/svclb-tdarr-wfk5l" Jul 24 08:56:56 bluth k3s[11030]: I0724 08:56:56.616950 11030 docker_sandbox.go:402] "Failed to read pod IP from plugin/docker" err="networkPlugin cni failed on the status hook for pod \"esphome-658957fddc-bbcxl_ix-esphome\": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container \"6d2d1f41beed8c6cb0a510daf97a2b869656d3eb1f69580c3e40b7d603a6913d\"" Jul 24 08:56:56 bluth k3s[11030]: E0724 08:56:56.625071 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: f8d6d2659381068c750ad76fb185454887ebb3b2b1ef2edfa953970d1554ccb9" containerID="f8d6d2659381068c750ad76fb185454887ebb3b2b1ef2edfa953970d1554ccb9" Jul 24 08:56:56 bluth k3s[11030]: E0724 08:56:56.625099 11030 kuberuntime_manager.go:1079] "getPodContainerStatuses for pod failed" err="rpc error: code = Unknown desc = Error: No such container: f8d6d2659381068c750ad76fb185454887ebb3b2b1ef2edfa953970d1554ccb9" pod="ix-home-assistant/home-assistant-ix-chart-54bf5d89c8-5rpjh" Jul 24 08:56:56 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T08:56:56.873+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 08:56:57 bluth k3s[11030]: E0724 08:56:57.755278 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:56:59 bluth k3s[11030]: E0724 08:56:59.753315 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:56:59 bluth k3s[11030]: E0724 08:56:59.916610 11030 authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, service account token has been invalidated]" Jul 24 08:57:00 bluth k3s[11030]: E0724 08:57:00.661514 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:59:02.661480375 +0100 IST m=+469.149337300 (durationBeforeRetry 2m2s). Error: MountVolume.MountDevice failed for volume "pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e") pod "esphome-658957fddc-6z2jm" (UID: "19254d20-7ad1-49a1-b408-f6e4213e5b4e") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:57:01 bluth k3s[11030]: E0724 08:57:01.468557 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-06521873-c179-499e-9079-194ec6618fdd podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:59:03.468513117 +0100 IST m=+469.956370039 (durationBeforeRetry 2m2s). Error: MountVolume.MountDevice failed for volume "pvc-06521873-c179-499e-9079-194ec6618fdd" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-06521873-c179-499e-9079-194ec6618fdd") pod "esphome-658957fddc-6z2jm" (UID: "19254d20-7ad1-49a1-b408-f6e4213e5b4e") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:57:01 bluth k3s[11030]: E0724 08:57:01.766880 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:57:02 bluth k3s[11030]: E0724 08:57:02.881302 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:59:04.88124315 +0100 IST m=+471.369100121 (durationBeforeRetry 2m2s). Error: MountVolume.MountDevice failed for volume "pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274") pod "prowlarr-5bcc99cc47-v86hm" (UID: "5d1f2955-de8a-4e25-b1f8-e607463ed369") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:57:03 bluth k3s[11030]: E0724 08:57:03.750676 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:57:03 bluth k3s[11030]: E0724 08:57:03.888826 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-58562e12-4adc-4e83-8131-848f1b39bf00 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:59:05.88881623 +0100 IST m=+472.376673110 (durationBeforeRetry 2m2s). Error: MountVolume.MountDevice failed for volume "pvc-58562e12-4adc-4e83-8131-848f1b39bf00" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-58562e12-4adc-4e83-8131-848f1b39bf00") pod "qbittorrent-6d7cf9f7cd-xrqwj" (UID: "4d43a190-0b41-4f84-a84d-ce295694fcb5") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:57:05 bluth k3s[11030]: I0724 08:57:05.091344 11030 scope.go:110] "RemoveContainer" containerID="fe7bdbe3401af33e1d6685ff8499d4e2ac6f1cdf9c60e748dc590c5b44b791af" Jul 24 08:57:05 bluth k3s[11030]: E0724 08:57:05.092525 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: fe7bdbe3401af33e1d6685ff8499d4e2ac6f1cdf9c60e748dc590c5b44b791af" containerID="fe7bdbe3401af33e1d6685ff8499d4e2ac6f1cdf9c60e748dc590c5b44b791af" Jul 24 08:57:05 bluth k3s[11030]: I0724 08:57:05.092606 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:fe7bdbe3401af33e1d6685ff8499d4e2ac6f1cdf9c60e748dc590c5b44b791af} err="failed to get container status \"fe7bdbe3401af33e1d6685ff8499d4e2ac6f1cdf9c60e748dc590c5b44b791af\": rpc error: code = Unknown desc = Error: No such container: fe7bdbe3401af33e1d6685ff8499d4e2ac6f1cdf9c60e748dc590c5b44b791af" Jul 24 08:57:05 bluth k3s[11030]: I0724 08:57:05.092636 11030 scope.go:110] "RemoveContainer" containerID="cfbb792152fd6498db6ad8ad88a72518ac6b5013489a020de4a2d04048f78e82" Jul 24 08:57:05 bluth k3s[11030]: E0724 08:57:05.093763 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: cfbb792152fd6498db6ad8ad88a72518ac6b5013489a020de4a2d04048f78e82" containerID="cfbb792152fd6498db6ad8ad88a72518ac6b5013489a020de4a2d04048f78e82" Jul 24 08:57:05 bluth k3s[11030]: I0724 08:57:05.093838 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:cfbb792152fd6498db6ad8ad88a72518ac6b5013489a020de4a2d04048f78e82} err="failed to get container status \"cfbb792152fd6498db6ad8ad88a72518ac6b5013489a020de4a2d04048f78e82\": rpc error: code = Unknown desc = Error: No such container: cfbb792152fd6498db6ad8ad88a72518ac6b5013489a020de4a2d04048f78e82" Jul 24 08:57:05 bluth k3s[11030]: I0724 08:57:05.093875 11030 scope.go:110] "RemoveContainer" containerID="eb0e10edd2f063bdae9a352c791deb2b10c9e3653067c197556c1bfce551a197" Jul 24 08:57:05 bluth k3s[11030]: E0724 08:57:05.098702 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-195f1a32-9c93-4737-983a-8dacee32e453 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:59:07.098669129 +0100 IST m=+473.586526053 (durationBeforeRetry 2m2s). Error: MountVolume.MountDevice failed for volume "pvc-195f1a32-9c93-4737-983a-8dacee32e453" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-195f1a32-9c93-4737-983a-8dacee32e453") pod "radarr-7486f78d7f-ntt44" (UID: "9ab9d8fe-e669-40bb-a0ce-7f7a4e5a69b5") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:57:05 bluth k3s[11030]: I0724 08:57:05.110894 11030 scope.go:110] "RemoveContainer" containerID="e5f26b185535168955288354fe20116be589fa2e5035f142ac660d7918c98bfa" Jul 24 08:57:05 bluth k3s[11030]: E0724 08:57:05.771685 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:57:06 bluth k3s[11030]: E0724 08:57:06.107646 11030 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4 podName: nodeName:}" failed. No retries permitted until 2022-07-24 08:59:08.107636099 +0100 IST m=+474.595492985 (durationBeforeRetry 2m2s). Error: MountVolume.MountDevice failed for volume "pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4") pod "sonarr-867d7dd548-fw6z4" (UID: "1a1846b0-3361-4f4b-a1cd-d5f958d6a7ac") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers Jul 24 08:57:06 bluth k3s[11030]: time="2022-07-24T08:57:06+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-tdarr\", Name:\"tdarr\", UID:\"90709eb9-3d84-4b91-b6ab-73ece8ffcb86\", APIVersion:\"v1\", ResourceVersion:\"3027898\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-tdarr/svclb-tdarr" Jul 24 08:57:06 bluth k3s[11030]: time="2022-07-24T08:57:06+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-tdarr\", Name:\"tdarr\", UID:\"90709eb9-3d84-4b91-b6ab-73ece8ffcb86\", APIVersion:\"v1\", ResourceVersion:\"3027898\", FieldPath:\"\"}): type: 'Normal' reason: 'UpdatedIngressIP' LoadBalancer Ingress IP addresses updated: 192.168.0.143" Jul 24 08:57:06 bluth k3s[11030]: time="2022-07-24T08:57:06+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-tdarr\", Name:\"tdarr\", UID:\"90709eb9-3d84-4b91-b6ab-73ece8ffcb86\", APIVersion:\"v1\", ResourceVersion:\"3028106\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-tdarr/svclb-tdarr" Jul 24 08:57:07 bluth k3s[11030]: E0724 08:57:07.755954 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:57:09 bluth k3s[11030]: E0724 08:57:09.040178 11030 authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, service account token has been invalidated]" Jul 24 08:57:09 bluth k3s[11030]: E0724 08:57:09.400677 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: df942c6cff83c663f6e3bc873bf3bf76e26ab9a76a11361760505b06f3dda6fd" containerID="df942c6cff83c663f6e3bc873bf3bf76e26ab9a76a11361760505b06f3dda6fd" Jul 24 08:57:09 bluth k3s[11030]: E0724 08:57:09.400701 11030 kuberuntime_manager.go:1079] "getPodContainerStatuses for pod failed" err="rpc error: code = Unknown desc = Error: No such container: df942c6cff83c663f6e3bc873bf3bf76e26ab9a76a11361760505b06f3dda6fd" pod="kube-system/openebs-zfs-node-vk949" Jul 24 08:57:09 bluth k3s[11030]: E0724 08:57:09.403964 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 8a35d5a908bdcf58ea1fbf4b79868f70b54c7299b90b01c99b1d2cecd22b0f44" containerID="8a35d5a908bdcf58ea1fbf4b79868f70b54c7299b90b01c99b1d2cecd22b0f44" Jul 24 08:57:09 bluth k3s[11030]: E0724 08:57:09.403989 11030 kuberuntime_manager.go:1079] "getPodContainerStatuses for pod failed" err="rpc error: code = Unknown desc = Error: No such container: 8a35d5a908bdcf58ea1fbf4b79868f70b54c7299b90b01c99b1d2cecd22b0f44" pod="ix-sonarr/svclb-sonarr-7tc6n" Jul 24 08:57:09 bluth k3s[11030]: E0724 08:57:09.410241 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 9132e315e0d5bc7b9e04b040529b2d2a7d22802b8015cee1d336243b41077fbd" containerID="9132e315e0d5bc7b9e04b040529b2d2a7d22802b8015cee1d336243b41077fbd" Jul 24 08:57:09 bluth k3s[11030]: E0724 08:57:09.410267 11030 kuberuntime_manager.go:1079] "getPodContainerStatuses for pod failed" err="rpc error: code = Unknown desc = Error: No such container: 9132e315e0d5bc7b9e04b040529b2d2a7d22802b8015cee1d336243b41077fbd" pod="kube-system/openebs-zfs-controller-0" Jul 24 08:57:09 bluth k3s[11030]: E0724 08:57:09.779136 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:57:10 bluth k3s[11030]: E0724 08:57:10.435012 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: c4969e3d944da0b6164eb9fd6b2fa0878a5671fd644ca187da10f5daa46d1635" containerID="c4969e3d944da0b6164eb9fd6b2fa0878a5671fd644ca187da10f5daa46d1635" Jul 24 08:57:10 bluth k3s[11030]: E0724 08:57:10.435038 11030 kuberuntime_manager.go:1079] "getPodContainerStatuses for pod failed" err="rpc error: code = Unknown desc = Error: No such container: c4969e3d944da0b6164eb9fd6b2fa0878a5671fd644ca187da10f5daa46d1635" pod="ix-esphome/svclb-esphome-5q48s" Jul 24 08:57:11 bluth k3s[11030]: E0724 08:57:11.751143 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:57:12 bluth k3s[11030]: I0724 08:57:12.124659 11030 scope.go:110] "RemoveContainer" containerID="782924fe93893e3f55e40b0f59f0d67d687452e1d614160a7b2b0975d992454a" Jul 24 08:57:12 bluth k3s[11030]: E0724 08:57:12.125138 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 782924fe93893e3f55e40b0f59f0d67d687452e1d614160a7b2b0975d992454a" containerID="782924fe93893e3f55e40b0f59f0d67d687452e1d614160a7b2b0975d992454a" Jul 24 08:57:12 bluth k3s[11030]: I0724 08:57:12.125173 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:782924fe93893e3f55e40b0f59f0d67d687452e1d614160a7b2b0975d992454a} err="failed to get container status \"782924fe93893e3f55e40b0f59f0d67d687452e1d614160a7b2b0975d992454a\": rpc error: code = Unknown desc = Error: No such container: 782924fe93893e3f55e40b0f59f0d67d687452e1d614160a7b2b0975d992454a" Jul 24 08:57:12 bluth k3s[11030]: I0724 08:57:12.125184 11030 scope.go:110] "RemoveContainer" containerID="b196460233e5a5b9fa676e7e35bea9ca195cf40970bce9956975f9feeb1866c6" Jul 24 08:57:13 bluth k3s[11030]: I0724 08:57:13.558467 11030 scope.go:110] "RemoveContainer" containerID="cfbb792152fd6498db6ad8ad88a72518ac6b5013489a020de4a2d04048f78e82" Jul 24 08:57:13 bluth k3s[11030]: E0724 08:57:13.560132 11030 kuberuntime_gc.go:146] "Failed to remove container" err="failed to get container status \"cfbb792152fd6498db6ad8ad88a72518ac6b5013489a020de4a2d04048f78e82\": rpc error: code = Unknown desc = Error: No such container: cfbb792152fd6498db6ad8ad88a72518ac6b5013489a020de4a2d04048f78e82" containerID="cfbb792152fd6498db6ad8ad88a72518ac6b5013489a020de4a2d04048f78e82" Jul 24 08:57:13 bluth k3s[11030]: I0724 08:57:13.560449 11030 scope.go:110] "RemoveContainer" containerID="fe7bdbe3401af33e1d6685ff8499d4e2ac6f1cdf9c60e748dc590c5b44b791af" Jul 24 08:57:13 bluth k3s[11030]: E0724 08:57:13.561883 11030 kuberuntime_gc.go:146] "Failed to remove container" err="failed to get container status \"fe7bdbe3401af33e1d6685ff8499d4e2ac6f1cdf9c60e748dc590c5b44b791af\": rpc error: code = Unknown desc = Error: No such container: fe7bdbe3401af33e1d6685ff8499d4e2ac6f1cdf9c60e748dc590c5b44b791af" containerID="fe7bdbe3401af33e1d6685ff8499d4e2ac6f1cdf9c60e748dc590c5b44b791af" Jul 24 08:57:13 bluth k3s[11030]: I0724 08:57:13.561939 11030 scope.go:110] "RemoveContainer" containerID="a3f6bf7a95d792418db2f78fd5ce4e1cbfee8c7ae330cd5e9d986e067496a6ee" Jul 24 08:57:13 bluth k3s[11030]: E0724 08:57:13.565340 11030 kuberuntime_gc.go:146] "Failed to remove container" err="failed to get container status \"a3f6bf7a95d792418db2f78fd5ce4e1cbfee8c7ae330cd5e9d986e067496a6ee\": rpc error: code = Unknown desc = Error: No such container: a3f6bf7a95d792418db2f78fd5ce4e1cbfee8c7ae330cd5e9d986e067496a6ee" containerID="a3f6bf7a95d792418db2f78fd5ce4e1cbfee8c7ae330cd5e9d986e067496a6ee" Jul 24 08:57:13 bluth k3s[11030]: E0724 08:57:13.565685 11030 kuberuntime_gc.go:141] "Failed to stop container" err="rpc error: code = Unknown desc = Error: No such container: f0c517a8f8e10361e1a200c82e3452308e4a1d821ffd44d7f03d256c49220965" containerID="f0c517a8f8e10361e1a200c82e3452308e4a1d821ffd44d7f03d256c49220965" Jul 24 08:57:13 bluth k3s[11030]: I0724 08:57:13.565702 11030 scope.go:110] "RemoveContainer" containerID="f2e0aaba8fe98fbfc891167786dd45c087333c3a081a5002abe08f129933f4b3" Jul 24 08:57:13 bluth k3s[11030]: time="2022-07-24T08:57:13+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-qbittorrent\", Name:\"qbittorrent-torrent\", UID:\"29b172f8-d799-4e0c-aac7-78ec5af5c01b\", APIVersion:\"v1\", ResourceVersion:\"3027879\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-qbittorrent/svclb-qbittorrent-torrent" Jul 24 08:57:13 bluth k3s[11030]: time="2022-07-24T08:57:13+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-qbittorrent\", Name:\"qbittorrent-torrent\", UID:\"29b172f8-d799-4e0c-aac7-78ec5af5c01b\", APIVersion:\"v1\", ResourceVersion:\"3027879\", FieldPath:\"\"}): type: 'Normal' reason: 'UpdatedIngressIP' LoadBalancer Ingress IP addresses updated: 192.168.0.143" Jul 24 08:57:13 bluth k3s[11030]: time="2022-07-24T08:57:13+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-qbittorrent\", Name:\"qbittorrent-torrent\", UID:\"29b172f8-d799-4e0c-aac7-78ec5af5c01b\", APIVersion:\"v1\", ResourceVersion:\"3028124\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-qbittorrent/svclb-qbittorrent-torrent" Jul 24 08:57:13 bluth k3s[11030]: E0724 08:57:13.790945 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:57:14 bluth k3s[11030]: E0724 08:57:14.436453 11030 authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, service account token has been invalidated]" Jul 24 08:57:15 bluth k3s[11030]: E0724 08:57:15.755124 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:57:17 bluth k3s[11030]: E0724 08:57:17.754859 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:57:18 bluth k3s[11030]: I0724 08:57:18.724534 11030 trace.go:205] Trace[698097178]: "GuaranteedUpdate etcd3" type:*coordination.Lease (24-Jul-2022 08:57:16.513) (total time: 2210ms): Jul 24 08:57:18 bluth k3s[11030]: Trace[698097178]: ---"Transaction committed" 2210ms (08:57:18.724) Jul 24 08:57:18 bluth k3s[11030]: Trace[698097178]: [2.210828976s] [2.210828976s] END Jul 24 08:57:18 bluth k3s[11030]: I0724 08:57:18.724817 11030 trace.go:205] Trace[1530578401]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ix-truenas,user-agent:Go-http-client/2.0,audit-id:e53fa9a6-e2ad-40db-92e9-2e8ff2d33816,client:127.0.0.1,accept:application/vnd.kubernetes.protobuf,application/json,protocol:HTTP/2.0 (24-Jul-2022 08:57:16.513) (total time: 2211ms): Jul 24 08:57:18 bluth k3s[11030]: Trace[1530578401]: ---"Object stored in database" 2210ms (08:57:18.724) Jul 24 08:57:18 bluth k3s[11030]: Trace[1530578401]: [2.211167744s] [2.211167744s] END Jul 24 08:57:19 bluth k3s[11030]: E0724 08:57:19.755150 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:57:20 bluth k3s[11030]: E0724 08:57:20.377648 11030 authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, service account token has been invalidated]" Jul 24 08:57:21 bluth k3s[11030]: I0724 08:57:21.701880 11030 trace.go:205] Trace[1106438293]: "Create" url:/api/v1/namespaces/kube-system/events,user-agent:k3s/v1.23.5+k3s (linux/amd64) kubernetes/fbfa51e,audit-id:2ab509e2-0a01-4bce-9831-87b4645be166,client:127.0.0.1,accept:application/vnd.kubernetes.protobuf,application/json,protocol:HTTP/2.0 (24-Jul-2022 08:57:19.350) (total time: 2351ms): Jul 24 08:57:21 bluth k3s[11030]: Trace[1106438293]: ---"Object stored in database" 2351ms (08:57:21.701) Jul 24 08:57:21 bluth k3s[11030]: Trace[1106438293]: [2.351527807s] [2.351527807s] END Jul 24 08:57:21 bluth k3s[11030]: E0724 08:57:21.755285 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:57:23 bluth k3s[11030]: E0724 08:57:23.766638 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:57:25 bluth k3s[11030]: E0724 08:57:25.755268 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:57:25 bluth k3s[11030]: E0724 08:57:25.961412 11030 authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, service account token has been invalidated]" Jul 24 08:57:27 bluth k3s[11030]: E0724 08:57:27.769773 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:57:28 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T08:57:28.210+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 08:57:29 bluth k3s[11030]: E0724 08:57:29.781277 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:57:29 bluth k3s[11030]: I0724 08:57:29.965945 11030 scope.go:110] "RemoveContainer" containerID="6437addedb00f04d2200af9df94d2c46ad0c84b94ab5f9cd01ede9763cc37a31" Jul 24 08:57:30 bluth k3s[11030]: I0724 08:57:30.029835 11030 scope.go:110] "RemoveContainer" containerID="2f05cc589edb55ee416116c44f237391d0ebd2ad73aa69221f61c1ae98f3ebf5" Jul 24 08:57:31 bluth k3s[11030]: E0724 08:57:31.756295 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:57:32 bluth k3s[11030]: E0724 08:57:32.769337 11030 authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, service account token has been invalidated]" Jul 24 08:57:33 bluth k3s[11030]: E0724 08:57:33.759360 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:57:34 bluth k3s[11030]: time="2022-07-24T08:57:34+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-sonarr\", Name:\"sonarr\", UID:\"b24571a2-a018-4716-ab7d-d6ab31e66e44\", APIVersion:\"v1\", ResourceVersion:\"3027945\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-sonarr/svclb-sonarr" Jul 24 08:57:34 bluth k3s[11030]: time="2022-07-24T08:57:34+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-sonarr\", Name:\"sonarr\", UID:\"b24571a2-a018-4716-ab7d-d6ab31e66e44\", APIVersion:\"v1\", ResourceVersion:\"3027945\", FieldPath:\"\"}): type: 'Normal' reason: 'UpdatedIngressIP' LoadBalancer Ingress IP addresses updated: 192.168.0.143" Jul 24 08:57:34 bluth k3s[11030]: time="2022-07-24T08:57:34+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-esphome\", Name:\"esphome\", UID:\"9d7f07bb-0ed3-4a9b-b9b3-ce707e119669\", APIVersion:\"v1\", ResourceVersion:\"3027860\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-esphome/svclb-esphome" Jul 24 08:57:34 bluth k3s[11030]: time="2022-07-24T08:57:34+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-sonarr\", Name:\"sonarr\", UID:\"b24571a2-a018-4716-ab7d-d6ab31e66e44\", APIVersion:\"v1\", ResourceVersion:\"3028146\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-sonarr/svclb-sonarr" Jul 24 08:57:34 bluth k3s[11030]: time="2022-07-24T08:57:34+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-esphome\", Name:\"esphome\", UID:\"9d7f07bb-0ed3-4a9b-b9b3-ce707e119669\", APIVersion:\"v1\", ResourceVersion:\"3027860\", FieldPath:\"\"}): type: 'Normal' reason: 'UpdatedIngressIP' LoadBalancer Ingress IP addresses updated: 192.168.0.143" Jul 24 08:57:34 bluth k3s[11030]: time="2022-07-24T08:57:34+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-esphome\", Name:\"esphome\", UID:\"9d7f07bb-0ed3-4a9b-b9b3-ce707e119669\", APIVersion:\"v1\", ResourceVersion:\"3028148\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-esphome/svclb-esphome" Jul 24 08:57:35 bluth k3s[11030]: E0724 08:57:35.759515 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:57:37 bluth k3s[11030]: E0724 08:57:37.759586 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:57:38 bluth k3s[11030]: I0724 08:57:38.029551 11030 scope.go:110] "RemoveContainer" containerID="bf5b2add1424bf252e033ed11b9b8edcb0933249a169e744c00925e65d6dd0b1" Jul 24 08:57:38 bluth k3s[11030]: I0724 08:57:38.029635 11030 scope.go:110] "RemoveContainer" containerID="9c386c1dc976ec8ca37d2565a3ca1712d3fa3e3558a77eaecb5dbaca90741687" Jul 24 08:57:39 bluth k3s[11030]: E0724 08:57:39.793272 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:57:40 bluth k3s[11030]: E0724 08:57:40.863177 11030 authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, service account token has been invalidated]" Jul 24 08:57:41 bluth k3s[11030]: E0724 08:57:41.768427 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:57:42 bluth k3s[11030]: I0724 08:57:42.778964 11030 scope.go:110] "RemoveContainer" containerID="b196460233e5a5b9fa676e7e35bea9ca195cf40970bce9956975f9feeb1866c6" Jul 24 08:57:42 bluth k3s[11030]: I0724 08:57:42.779420 11030 scope.go:110] "RemoveContainer" containerID="42b902fbe947e992f34bf8a08c63ee2db5d7ad5ba83866bafb058bd7f2f1377e" Jul 24 08:57:42 bluth k3s[11030]: E0724 08:57:42.780274 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: b196460233e5a5b9fa676e7e35bea9ca195cf40970bce9956975f9feeb1866c6" containerID="b196460233e5a5b9fa676e7e35bea9ca195cf40970bce9956975f9feeb1866c6" Jul 24 08:57:42 bluth k3s[11030]: E0724 08:57:42.780363 11030 kuberuntime_gc.go:146] "Failed to remove container" err="failed to get container status \"b196460233e5a5b9fa676e7e35bea9ca195cf40970bce9956975f9feeb1866c6\": rpc error: code = Unknown desc = Error: No such container: b196460233e5a5b9fa676e7e35bea9ca195cf40970bce9956975f9feeb1866c6" containerID="b196460233e5a5b9fa676e7e35bea9ca195cf40970bce9956975f9feeb1866c6" Jul 24 08:57:42 bluth k3s[11030]: I0724 08:57:42.780396 11030 scope.go:110] "RemoveContainer" containerID="42b902fbe947e992f34bf8a08c63ee2db5d7ad5ba83866bafb058bd7f2f1377e" Jul 24 08:57:42 bluth k3s[11030]: E0724 08:57:42.979136 11030 remote_runtime.go:505] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to remove container \"42b902fbe947e992f34bf8a08c63ee2db5d7ad5ba83866bafb058bd7f2f1377e\": Error response from daemon: removal of container 42b902fbe947e992f34bf8a08c63ee2db5d7ad5ba83866bafb058bd7f2f1377e is already in progress" containerID="42b902fbe947e992f34bf8a08c63ee2db5d7ad5ba83866bafb058bd7f2f1377e" Jul 24 08:57:42 bluth k3s[11030]: E0724 08:57:42.979218 11030 kuberuntime_gc.go:146] "Failed to remove container" err="rpc error: code = Unknown desc = failed to remove container \"42b902fbe947e992f34bf8a08c63ee2db5d7ad5ba83866bafb058bd7f2f1377e\": Error response from daemon: removal of container 42b902fbe947e992f34bf8a08c63ee2db5d7ad5ba83866bafb058bd7f2f1377e is already in progress" containerID="42b902fbe947e992f34bf8a08c63ee2db5d7ad5ba83866bafb058bd7f2f1377e" Jul 24 08:57:42 bluth k3s[11030]: I0724 08:57:42.979261 11030 scope.go:110] "RemoveContainer" containerID="bf5b2add1424bf252e033ed11b9b8edcb0933249a169e744c00925e65d6dd0b1" Jul 24 08:57:42 bluth k3s[11030]: E0724 08:57:42.980410 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: bf5b2add1424bf252e033ed11b9b8edcb0933249a169e744c00925e65d6dd0b1" containerID="bf5b2add1424bf252e033ed11b9b8edcb0933249a169e744c00925e65d6dd0b1" Jul 24 08:57:42 bluth k3s[11030]: E0724 08:57:42.980481 11030 kuberuntime_gc.go:146] "Failed to remove container" err="failed to get container status \"bf5b2add1424bf252e033ed11b9b8edcb0933249a169e744c00925e65d6dd0b1\": rpc error: code = Unknown desc = Error: No such container: bf5b2add1424bf252e033ed11b9b8edcb0933249a169e744c00925e65d6dd0b1" containerID="bf5b2add1424bf252e033ed11b9b8edcb0933249a169e744c00925e65d6dd0b1" Jul 24 08:57:42 bluth k3s[11030]: I0724 08:57:42.980510 11030 scope.go:110] "RemoveContainer" containerID="2c03ead7d0c9acbaa5c5287c0fc7c095b28c281ee20ff47d5f657ccb9db92cbb" Jul 24 08:57:43 bluth k3s[11030]: E0724 08:57:43.778310 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:57:45 bluth k3s[11030]: E0724 08:57:45.793883 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:57:47 bluth k3s[11030]: I0724 08:57:47.674859 11030 scope.go:110] "RemoveContainer" containerID="caecd739834fb496e4f85acb4bbf6596b6d5f784a17abf9b90bd230218f227f3" Jul 24 08:57:47 bluth k3s[11030]: E0724 08:57:47.762809 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:57:49 bluth k3s[11030]: E0724 08:57:49.760202 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:57:50 bluth k3s[11030]: E0724 08:57:50.747366 11030 authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, service account token has been invalidated]" Jul 24 08:57:51 bluth k3s[11030]: E0724 08:57:51.755741 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:57:52 bluth k3s[11030]: I0724 08:57:52.672907 11030 scope.go:110] "RemoveContainer" containerID="5663eacfde3830db553aca2b541f0f3063848ad7aa0633ecc515c50ab4033f6c" Jul 24 08:57:53 bluth k3s[11030]: E0724 08:57:53.768053 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:57:55 bluth k3s[11030]: E0724 08:57:55.756509 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:57:57 bluth k3s[11030]: E0724 08:57:57.035630 11030 authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, service account token has been invalidated]" Jul 24 08:57:57 bluth k3s[11030]: E0724 08:57:57.759685 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:57:59 bluth k3s[11030]: E0724 08:57:59.759664 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:58:01 bluth k3s[11030]: E0724 08:58:01.755856 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:58:03 bluth k3s[11030]: E0724 08:58:03.175475 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: a8f16966a954bc0a1732cf0088535fcf200f4f3bd0942166d7c7f8b024a4f79e" containerID="a8f16966a954bc0a1732cf0088535fcf200f4f3bd0942166d7c7f8b024a4f79e" Jul 24 08:58:03 bluth k3s[11030]: E0724 08:58:03.175553 11030 kuberuntime_manager.go:1079] "getPodContainerStatuses for pod failed" err="rpc error: code = Unknown desc = Error: No such container: a8f16966a954bc0a1732cf0088535fcf200f4f3bd0942166d7c7f8b024a4f79e" pod="kube-system/intel-gpu-plugin-9t629" Jul 24 08:58:03 bluth k3s[11030]: I0724 08:58:03.213211 11030 scope.go:110] "RemoveContainer" containerID="b196460233e5a5b9fa676e7e35bea9ca195cf40970bce9956975f9feeb1866c6" Jul 24 08:58:03 bluth k3s[11030]: I0724 08:58:03.214311 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:b196460233e5a5b9fa676e7e35bea9ca195cf40970bce9956975f9feeb1866c6} err="failed to get container status \"b196460233e5a5b9fa676e7e35bea9ca195cf40970bce9956975f9feeb1866c6\": rpc error: code = Unknown desc = Error: No such container: b196460233e5a5b9fa676e7e35bea9ca195cf40970bce9956975f9feeb1866c6" Jul 24 08:58:03 bluth k3s[11030]: I0724 08:58:03.214384 11030 scope.go:110] "RemoveContainer" containerID="2f05cc589edb55ee416116c44f237391d0ebd2ad73aa69221f61c1ae98f3ebf5" Jul 24 08:58:03 bluth k3s[11030]: E0724 08:58:03.215561 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 2f05cc589edb55ee416116c44f237391d0ebd2ad73aa69221f61c1ae98f3ebf5" containerID="2f05cc589edb55ee416116c44f237391d0ebd2ad73aa69221f61c1ae98f3ebf5" Jul 24 08:58:03 bluth k3s[11030]: I0724 08:58:03.215651 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:2f05cc589edb55ee416116c44f237391d0ebd2ad73aa69221f61c1ae98f3ebf5} err="failed to get container status \"2f05cc589edb55ee416116c44f237391d0ebd2ad73aa69221f61c1ae98f3ebf5\": rpc error: code = Unknown desc = Error: No such container: 2f05cc589edb55ee416116c44f237391d0ebd2ad73aa69221f61c1ae98f3ebf5" Jul 24 08:58:03 bluth k3s[11030]: I0724 08:58:03.215685 11030 scope.go:110] "RemoveContainer" containerID="2c03ead7d0c9acbaa5c5287c0fc7c095b28c281ee20ff47d5f657ccb9db92cbb" Jul 24 08:58:03 bluth k3s[11030]: E0724 08:58:03.216685 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 2c03ead7d0c9acbaa5c5287c0fc7c095b28c281ee20ff47d5f657ccb9db92cbb" containerID="2c03ead7d0c9acbaa5c5287c0fc7c095b28c281ee20ff47d5f657ccb9db92cbb" Jul 24 08:58:03 bluth k3s[11030]: I0724 08:58:03.216761 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:2c03ead7d0c9acbaa5c5287c0fc7c095b28c281ee20ff47d5f657ccb9db92cbb} err="failed to get container status \"2c03ead7d0c9acbaa5c5287c0fc7c095b28c281ee20ff47d5f657ccb9db92cbb\": rpc error: code = Unknown desc = Error: No such container: 2c03ead7d0c9acbaa5c5287c0fc7c095b28c281ee20ff47d5f657ccb9db92cbb" Jul 24 08:58:03 bluth k3s[11030]: I0724 08:58:03.216790 11030 scope.go:110] "RemoveContainer" containerID="bf5b2add1424bf252e033ed11b9b8edcb0933249a169e744c00925e65d6dd0b1" Jul 24 08:58:03 bluth k3s[11030]: I0724 08:58:03.217765 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:bf5b2add1424bf252e033ed11b9b8edcb0933249a169e744c00925e65d6dd0b1} err="failed to get container status \"bf5b2add1424bf252e033ed11b9b8edcb0933249a169e744c00925e65d6dd0b1\": rpc error: code = Unknown desc = Error: No such container: bf5b2add1424bf252e033ed11b9b8edcb0933249a169e744c00925e65d6dd0b1" Jul 24 08:58:03 bluth k3s[11030]: I0724 08:58:03.217806 11030 scope.go:110] "RemoveContainer" containerID="6ad0b3c8059d50b878addcfca6272e5cb2118b8128b92b8010fcef833e38ee3d" Jul 24 08:58:03 bluth k3s[11030]: E0724 08:58:03.218640 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 6ad0b3c8059d50b878addcfca6272e5cb2118b8128b92b8010fcef833e38ee3d" containerID="6ad0b3c8059d50b878addcfca6272e5cb2118b8128b92b8010fcef833e38ee3d" Jul 24 08:58:03 bluth k3s[11030]: I0724 08:58:03.218705 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:6ad0b3c8059d50b878addcfca6272e5cb2118b8128b92b8010fcef833e38ee3d} err="failed to get container status \"6ad0b3c8059d50b878addcfca6272e5cb2118b8128b92b8010fcef833e38ee3d\": rpc error: code = Unknown desc = Error: No such container: 6ad0b3c8059d50b878addcfca6272e5cb2118b8128b92b8010fcef833e38ee3d" Jul 24 08:58:03 bluth k3s[11030]: I0724 08:58:03.218733 11030 scope.go:110] "RemoveContainer" containerID="a3f6bf7a95d792418db2f78fd5ce4e1cbfee8c7ae330cd5e9d986e067496a6ee" Jul 24 08:58:03 bluth k3s[11030]: E0724 08:58:03.219599 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: a3f6bf7a95d792418db2f78fd5ce4e1cbfee8c7ae330cd5e9d986e067496a6ee" containerID="a3f6bf7a95d792418db2f78fd5ce4e1cbfee8c7ae330cd5e9d986e067496a6ee" Jul 24 08:58:03 bluth k3s[11030]: I0724 08:58:03.219670 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:a3f6bf7a95d792418db2f78fd5ce4e1cbfee8c7ae330cd5e9d986e067496a6ee} err="failed to get container status \"a3f6bf7a95d792418db2f78fd5ce4e1cbfee8c7ae330cd5e9d986e067496a6ee\": rpc error: code = Unknown desc = Error: No such container: a3f6bf7a95d792418db2f78fd5ce4e1cbfee8c7ae330cd5e9d986e067496a6ee" Jul 24 08:58:03 bluth k3s[11030]: I0724 08:58:03.219699 11030 scope.go:110] "RemoveContainer" containerID="da222fbe1dab51f95aac3d1db119f0e73a8724c1ee0f062fbee44543a382f10a" Jul 24 08:58:03 bluth k3s[11030]: E0724 08:58:03.760116 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:58:04 bluth k3s[11030]: E0724 08:58:04.322273 11030 authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, service account token has been invalidated]" Jul 24 08:58:05 bluth k3s[11030]: E0724 08:58:05.757064 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:58:05 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T08:58:05.913+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 08:58:07 bluth k3s[11030]: E0724 08:58:07.760074 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:58:07 bluth k3s[11030]: I0724 08:58:07.954115 11030 scope.go:110] "RemoveContainer" containerID="2f05cc589edb55ee416116c44f237391d0ebd2ad73aa69221f61c1ae98f3ebf5" Jul 24 08:58:07 bluth k3s[11030]: E0724 08:58:07.955563 11030 kuberuntime_gc.go:146] "Failed to remove container" err="failed to get container status \"2f05cc589edb55ee416116c44f237391d0ebd2ad73aa69221f61c1ae98f3ebf5\": rpc error: code = Unknown desc = Error: No such container: 2f05cc589edb55ee416116c44f237391d0ebd2ad73aa69221f61c1ae98f3ebf5" containerID="2f05cc589edb55ee416116c44f237391d0ebd2ad73aa69221f61c1ae98f3ebf5" Jul 24 08:58:07 bluth k3s[11030]: I0724 08:58:07.955613 11030 scope.go:110] "RemoveContainer" containerID="caecd739834fb496e4f85acb4bbf6596b6d5f784a17abf9b90bd230218f227f3" Jul 24 08:58:07 bluth k3s[11030]: E0724 08:58:07.956909 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: caecd739834fb496e4f85acb4bbf6596b6d5f784a17abf9b90bd230218f227f3" containerID="caecd739834fb496e4f85acb4bbf6596b6d5f784a17abf9b90bd230218f227f3" Jul 24 08:58:07 bluth k3s[11030]: E0724 08:58:07.957004 11030 kuberuntime_gc.go:146] "Failed to remove container" err="failed to get container status \"caecd739834fb496e4f85acb4bbf6596b6d5f784a17abf9b90bd230218f227f3\": rpc error: code = Unknown desc = Error: No such container: caecd739834fb496e4f85acb4bbf6596b6d5f784a17abf9b90bd230218f227f3" containerID="caecd739834fb496e4f85acb4bbf6596b6d5f784a17abf9b90bd230218f227f3" Jul 24 08:58:07 bluth k3s[11030]: I0724 08:58:07.957038 11030 scope.go:110] "RemoveContainer" containerID="756fdda552578aff50e29f2e417620f8f541349b9c30860194d255f85855e431" Jul 24 08:58:09 bluth k3s[11030]: E0724 08:58:09.795025 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:58:10 bluth k3s[11030]: E0724 08:58:10.327041 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 06eefc12a0bf11c5a58f4cabc3c155642c0efb73cb044d8a293548d860b65c1d" containerID="06eefc12a0bf11c5a58f4cabc3c155642c0efb73cb044d8a293548d860b65c1d" Jul 24 08:58:10 bluth k3s[11030]: E0724 08:58:10.327066 11030 kuberuntime_manager.go:1079] "getPodContainerStatuses for pod failed" err="rpc error: code = Unknown desc = Error: No such container: 06eefc12a0bf11c5a58f4cabc3c155642c0efb73cb044d8a293548d860b65c1d" pod="kube-system/openebs-zfs-controller-0" Jul 24 08:58:11 bluth k3s[11030]: E0724 08:58:11.233147 11030 authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, service account token has been invalidated]" Jul 24 08:58:11 bluth k3s[11030]: E0724 08:58:11.751308 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:58:13 bluth k3s[11030]: E0724 08:58:13.755266 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:58:15 bluth k3s[11030]: E0724 08:58:15.759912 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:58:16 bluth k3s[11030]: I0724 08:58:16.218861 11030 scope.go:110] "RemoveContainer" containerID="8ccd98bfe3f8ac373925d4c0f4d05f2989800d69d62f7e4413e1c55054f6c9ad" Jul 24 08:58:16 bluth k3s[11030]: E0724 08:58:16.220040 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 8ccd98bfe3f8ac373925d4c0f4d05f2989800d69d62f7e4413e1c55054f6c9ad" containerID="8ccd98bfe3f8ac373925d4c0f4d05f2989800d69d62f7e4413e1c55054f6c9ad" Jul 24 08:58:16 bluth k3s[11030]: I0724 08:58:16.220122 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:8ccd98bfe3f8ac373925d4c0f4d05f2989800d69d62f7e4413e1c55054f6c9ad} err="failed to get container status \"8ccd98bfe3f8ac373925d4c0f4d05f2989800d69d62f7e4413e1c55054f6c9ad\": rpc error: code = Unknown desc = Error: No such container: 8ccd98bfe3f8ac373925d4c0f4d05f2989800d69d62f7e4413e1c55054f6c9ad" Jul 24 08:58:16 bluth k3s[11030]: I0724 08:58:16.220156 11030 scope.go:110] "RemoveContainer" containerID="44e4853c5f38ff2233bc667e7c45189d65e49a86f2eba2e69d6ff0dea9f690cf" Jul 24 08:58:16 bluth k3s[11030]: E0724 08:58:16.221193 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 44e4853c5f38ff2233bc667e7c45189d65e49a86f2eba2e69d6ff0dea9f690cf" containerID="44e4853c5f38ff2233bc667e7c45189d65e49a86f2eba2e69d6ff0dea9f690cf" Jul 24 08:58:16 bluth k3s[11030]: I0724 08:58:16.221277 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:44e4853c5f38ff2233bc667e7c45189d65e49a86f2eba2e69d6ff0dea9f690cf} err="failed to get container status \"44e4853c5f38ff2233bc667e7c45189d65e49a86f2eba2e69d6ff0dea9f690cf\": rpc error: code = Unknown desc = Error: No such container: 44e4853c5f38ff2233bc667e7c45189d65e49a86f2eba2e69d6ff0dea9f690cf" Jul 24 08:58:16 bluth k3s[11030]: I0724 08:58:16.221309 11030 scope.go:110] "RemoveContainer" containerID="cfbb792152fd6498db6ad8ad88a72518ac6b5013489a020de4a2d04048f78e82" Jul 24 08:58:16 bluth k3s[11030]: E0724 08:58:16.222351 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: cfbb792152fd6498db6ad8ad88a72518ac6b5013489a020de4a2d04048f78e82" containerID="cfbb792152fd6498db6ad8ad88a72518ac6b5013489a020de4a2d04048f78e82" Jul 24 08:58:16 bluth k3s[11030]: I0724 08:58:16.222426 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:cfbb792152fd6498db6ad8ad88a72518ac6b5013489a020de4a2d04048f78e82} err="failed to get container status \"cfbb792152fd6498db6ad8ad88a72518ac6b5013489a020de4a2d04048f78e82\": rpc error: code = Unknown desc = Error: No such container: cfbb792152fd6498db6ad8ad88a72518ac6b5013489a020de4a2d04048f78e82" Jul 24 08:58:16 bluth k3s[11030]: I0724 08:58:16.222458 11030 scope.go:110] "RemoveContainer" containerID="5663eacfde3830db553aca2b541f0f3063848ad7aa0633ecc515c50ab4033f6c" Jul 24 08:58:16 bluth k3s[11030]: E0724 08:58:16.223385 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 5663eacfde3830db553aca2b541f0f3063848ad7aa0633ecc515c50ab4033f6c" containerID="5663eacfde3830db553aca2b541f0f3063848ad7aa0633ecc515c50ab4033f6c" Jul 24 08:58:16 bluth k3s[11030]: I0724 08:58:16.223450 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:5663eacfde3830db553aca2b541f0f3063848ad7aa0633ecc515c50ab4033f6c} err="failed to get container status \"5663eacfde3830db553aca2b541f0f3063848ad7aa0633ecc515c50ab4033f6c\": rpc error: code = Unknown desc = Error: No such container: 5663eacfde3830db553aca2b541f0f3063848ad7aa0633ecc515c50ab4033f6c" Jul 24 08:58:16 bluth k3s[11030]: I0724 08:58:16.223478 11030 scope.go:110] "RemoveContainer" containerID="0116727ef517122669f9ac99183cd270fe069d99a1a6d6be05567380ffdfd0af" Jul 24 08:58:16 bluth k3s[11030]: I0724 08:58:16.726542 11030 scope.go:110] "RemoveContainer" containerID="da222fbe1dab51f95aac3d1db119f0e73a8724c1ee0f062fbee44543a382f10a" Jul 24 08:58:16 bluth k3s[11030]: E0724 08:58:16.727177 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: da222fbe1dab51f95aac3d1db119f0e73a8724c1ee0f062fbee44543a382f10a" containerID="da222fbe1dab51f95aac3d1db119f0e73a8724c1ee0f062fbee44543a382f10a" Jul 24 08:58:16 bluth k3s[11030]: E0724 08:58:16.727199 11030 kuberuntime_gc.go:146] "Failed to remove container" err="failed to get container status \"da222fbe1dab51f95aac3d1db119f0e73a8724c1ee0f062fbee44543a382f10a\": rpc error: code = Unknown desc = Error: No such container: da222fbe1dab51f95aac3d1db119f0e73a8724c1ee0f062fbee44543a382f10a" containerID="da222fbe1dab51f95aac3d1db119f0e73a8724c1ee0f062fbee44543a382f10a" Jul 24 08:58:16 bluth k3s[11030]: I0724 08:58:16.727209 11030 scope.go:110] "RemoveContainer" containerID="eb0e10edd2f063bdae9a352c791deb2b10c9e3653067c197556c1bfce551a197" Jul 24 08:58:16 bluth k3s[11030]: E0724 08:58:16.727479 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: eb0e10edd2f063bdae9a352c791deb2b10c9e3653067c197556c1bfce551a197" containerID="eb0e10edd2f063bdae9a352c791deb2b10c9e3653067c197556c1bfce551a197" Jul 24 08:58:16 bluth k3s[11030]: E0724 08:58:16.727497 11030 kuberuntime_gc.go:146] "Failed to remove container" err="failed to get container status \"eb0e10edd2f063bdae9a352c791deb2b10c9e3653067c197556c1bfce551a197\": rpc error: code = Unknown desc = Error: No such container: eb0e10edd2f063bdae9a352c791deb2b10c9e3653067c197556c1bfce551a197" containerID="eb0e10edd2f063bdae9a352c791deb2b10c9e3653067c197556c1bfce551a197" Jul 24 08:58:16 bluth k3s[11030]: I0724 08:58:16.727505 11030 scope.go:110] "RemoveContainer" containerID="0116727ef517122669f9ac99183cd270fe069d99a1a6d6be05567380ffdfd0af" Jul 24 08:58:16 bluth k3s[11030]: E0724 08:58:16.729263 11030 remote_runtime.go:505] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to remove container \"0116727ef517122669f9ac99183cd270fe069d99a1a6d6be05567380ffdfd0af\": Error response from daemon: removal of container 0116727ef517122669f9ac99183cd270fe069d99a1a6d6be05567380ffdfd0af is already in progress" containerID="0116727ef517122669f9ac99183cd270fe069d99a1a6d6be05567380ffdfd0af" Jul 24 08:58:16 bluth k3s[11030]: E0724 08:58:16.729279 11030 kuberuntime_gc.go:146] "Failed to remove container" err="rpc error: code = Unknown desc = failed to remove container \"0116727ef517122669f9ac99183cd270fe069d99a1a6d6be05567380ffdfd0af\": Error response from daemon: removal of container 0116727ef517122669f9ac99183cd270fe069d99a1a6d6be05567380ffdfd0af is already in progress" containerID="0116727ef517122669f9ac99183cd270fe069d99a1a6d6be05567380ffdfd0af" Jul 24 08:58:16 bluth k3s[11030]: I0724 08:58:16.729287 11030 scope.go:110] "RemoveContainer" containerID="686b0d91412ae30adb501e5ec1920243d7e14a8b1ee45f2c43c9e4365f372201" Jul 24 08:58:17 bluth k3s[11030]: E0724 08:58:17.759890 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:58:19 bluth k3s[11030]: E0724 08:58:19.053955 11030 authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, service account token has been invalidated]" Jul 24 08:58:19 bluth k3s[11030]: E0724 08:58:19.756061 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:58:21 bluth k3s[11030]: E0724 08:58:21.759687 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:58:23 bluth k3s[11030]: E0724 08:58:23.755771 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:58:24 bluth k3s[11030]: I0724 08:58:24.499494 11030 scope.go:110] "RemoveContainer" containerID="42b902fbe947e992f34bf8a08c63ee2db5d7ad5ba83866bafb058bd7f2f1377e" Jul 24 08:58:24 bluth k3s[11030]: E0724 08:58:24.504106 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 42b902fbe947e992f34bf8a08c63ee2db5d7ad5ba83866bafb058bd7f2f1377e" containerID="42b902fbe947e992f34bf8a08c63ee2db5d7ad5ba83866bafb058bd7f2f1377e" Jul 24 08:58:24 bluth k3s[11030]: I0724 08:58:24.504202 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:42b902fbe947e992f34bf8a08c63ee2db5d7ad5ba83866bafb058bd7f2f1377e} err="failed to get container status \"42b902fbe947e992f34bf8a08c63ee2db5d7ad5ba83866bafb058bd7f2f1377e\": rpc error: code = Unknown desc = Error: No such container: 42b902fbe947e992f34bf8a08c63ee2db5d7ad5ba83866bafb058bd7f2f1377e" Jul 24 08:58:24 bluth k3s[11030]: I0724 08:58:24.504237 11030 scope.go:110] "RemoveContainer" containerID="caecd739834fb496e4f85acb4bbf6596b6d5f784a17abf9b90bd230218f227f3" Jul 24 08:58:24 bluth k3s[11030]: I0724 08:58:24.506232 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:caecd739834fb496e4f85acb4bbf6596b6d5f784a17abf9b90bd230218f227f3} err="failed to get container status \"caecd739834fb496e4f85acb4bbf6596b6d5f784a17abf9b90bd230218f227f3\": rpc error: code = Unknown desc = Error: No such container: caecd739834fb496e4f85acb4bbf6596b6d5f784a17abf9b90bd230218f227f3" Jul 24 08:58:24 bluth k3s[11030]: I0724 08:58:24.506641 11030 scope.go:110] "RemoveContainer" containerID="2f05cc589edb55ee416116c44f237391d0ebd2ad73aa69221f61c1ae98f3ebf5" Jul 24 08:58:24 bluth k3s[11030]: I0724 08:58:24.507968 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:2f05cc589edb55ee416116c44f237391d0ebd2ad73aa69221f61c1ae98f3ebf5} err="failed to get container status \"2f05cc589edb55ee416116c44f237391d0ebd2ad73aa69221f61c1ae98f3ebf5\": rpc error: code = Unknown desc = Error: No such container: 2f05cc589edb55ee416116c44f237391d0ebd2ad73aa69221f61c1ae98f3ebf5" Jul 24 08:58:24 bluth k3s[11030]: I0724 08:58:24.508017 11030 scope.go:110] "RemoveContainer" containerID="42b902fbe947e992f34bf8a08c63ee2db5d7ad5ba83866bafb058bd7f2f1377e" Jul 24 08:58:24 bluth k3s[11030]: I0724 08:58:24.509083 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:42b902fbe947e992f34bf8a08c63ee2db5d7ad5ba83866bafb058bd7f2f1377e} err="failed to get container status \"42b902fbe947e992f34bf8a08c63ee2db5d7ad5ba83866bafb058bd7f2f1377e\": rpc error: code = Unknown desc = Error: No such container: 42b902fbe947e992f34bf8a08c63ee2db5d7ad5ba83866bafb058bd7f2f1377e" Jul 24 08:58:24 bluth k3s[11030]: I0724 08:58:24.509152 11030 scope.go:110] "RemoveContainer" containerID="caecd739834fb496e4f85acb4bbf6596b6d5f784a17abf9b90bd230218f227f3" Jul 24 08:58:24 bluth k3s[11030]: I0724 08:58:24.510284 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:caecd739834fb496e4f85acb4bbf6596b6d5f784a17abf9b90bd230218f227f3} err="failed to get container status \"caecd739834fb496e4f85acb4bbf6596b6d5f784a17abf9b90bd230218f227f3\": rpc error: code = Unknown desc = Error: No such container: caecd739834fb496e4f85acb4bbf6596b6d5f784a17abf9b90bd230218f227f3" Jul 24 08:58:24 bluth k3s[11030]: I0724 08:58:24.510361 11030 scope.go:110] "RemoveContainer" containerID="2c03ead7d0c9acbaa5c5287c0fc7c095b28c281ee20ff47d5f657ccb9db92cbb" Jul 24 08:58:24 bluth k3s[11030]: I0724 08:58:24.511559 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:2c03ead7d0c9acbaa5c5287c0fc7c095b28c281ee20ff47d5f657ccb9db92cbb} err="failed to get container status \"2c03ead7d0c9acbaa5c5287c0fc7c095b28c281ee20ff47d5f657ccb9db92cbb\": rpc error: code = Unknown desc = Error: No such container: 2c03ead7d0c9acbaa5c5287c0fc7c095b28c281ee20ff47d5f657ccb9db92cbb" Jul 24 08:58:24 bluth k3s[11030]: I0724 08:58:24.511607 11030 scope.go:110] "RemoveContainer" containerID="caecd739834fb496e4f85acb4bbf6596b6d5f784a17abf9b90bd230218f227f3" Jul 24 08:58:24 bluth k3s[11030]: I0724 08:58:24.512705 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:caecd739834fb496e4f85acb4bbf6596b6d5f784a17abf9b90bd230218f227f3} err="failed to get container status \"caecd739834fb496e4f85acb4bbf6596b6d5f784a17abf9b90bd230218f227f3\": rpc error: code = Unknown desc = Error: No such container: caecd739834fb496e4f85acb4bbf6596b6d5f784a17abf9b90bd230218f227f3" Jul 24 08:58:24 bluth k3s[11030]: E0724 08:58:24.537537 11030 authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, service account token has been invalidated]" Jul 24 08:58:24 bluth k3s[11030]: E0724 08:58:24.537555 11030 authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, service account token has been invalidated]" Jul 24 08:58:24 bluth k3s[11030]: E0724 08:58:24.537541 11030 authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, service account token has been invalidated]" Jul 24 08:58:24 bluth k3s[11030]: E0724 08:58:24.542946 11030 authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, service account token has been invalidated]" Jul 24 08:58:24 bluth k3s[11030]: E0724 08:58:24.545364 11030 authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, service account token has been invalidated]" Jul 24 08:58:25 bluth k3s[11030]: I0724 08:58:25.164550 11030 scope.go:110] "RemoveContainer" containerID="b91ebd694b1c600c8c5bed572133ceb681e2887880d43188147754a231f0893d" Jul 24 08:58:25 bluth k3s[11030]: I0724 08:58:25.363989 11030 manager.go:422] "Got registration request from device plugin with resource" resourceName="gpu.intel.com/i915" Jul 24 08:58:25 bluth k3s[11030]: E0724 08:58:25.753540 11030 authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, service account token has been invalidated]" Jul 24 08:58:25 bluth k3s[11030]: E0724 08:58:25.755543 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:58:25 bluth k3s[11030]: E0724 08:58:25.828205 11030 authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, service account token has been invalidated]" Jul 24 08:58:26 bluth k3s[11030]: I0724 08:58:26.073647 11030 csi_plugin.go:99] kubernetes.io/csi: Trying to validate a new CSI Driver with name: zfs.csi.openebs.io endpoint: /var/lib/kubelet/plugins/zfs-localpv/csi.sock versions: 1.0.0 Jul 24 08:58:26 bluth k3s[11030]: I0724 08:58:26.073668 11030 csi_plugin.go:112] kubernetes.io/csi: Register new plugin with name: zfs.csi.openebs.io at endpoint: /var/lib/kubelet/plugins/zfs-localpv/csi.sock Jul 24 08:58:26 bluth k3s[11030]: E0724 08:58:26.095654 11030 authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, service account token has been invalidated]" Jul 24 08:58:26 bluth k3s[11030]: I0724 08:58:26.921162 11030 controller.go:611] quota admission added evaluator for: zfsnodes.zfs.openebs.io Jul 24 08:58:27 bluth k3s[11030]: E0724 08:58:27.759919 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:58:28 bluth k3s[11030]: E0724 08:58:28.255950 11030 authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, service account token has been invalidated]" Jul 24 08:58:29 bluth k3s[11030]: E0724 08:58:29.755433 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:58:31 bluth k3s[11030]: E0724 08:58:31.782435 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:58:33 bluth k3s[11030]: I0724 08:58:33.455810 11030 scope.go:110] "RemoveContainer" containerID="71136f6fb0ee2636e0473695ae9068ded11a0bc00c0071e8799bc37fa1bbaa7b" Jul 24 08:58:33 bluth k3s[11030]: I0724 08:58:33.592611 11030 scope.go:110] "RemoveContainer" containerID="fd4570ef9df9bcf5b972fe2c82b83f8100d6df535f7dc9742f803f496b0a4252" Jul 24 08:58:33 bluth k3s[11030]: E0724 08:58:33.756379 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:58:35 bluth k3s[11030]: I0724 08:58:35.199917 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:58:35 bluth k3s[11030]: I0724 08:58:35.339216 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:58:35 bluth k3s[11030]: I0724 08:58:35.462476 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"shared\" (UniqueName: \"kubernetes.io/empty-dir/01a1ac7e-39b1-4370-bf89-2be6180ecda9-shared\") pod \"plex-bd956645c-wxsbd\" (UID: \"01a1ac7e-39b1-4370-bf89-2be6180ecda9\") " pod="ix-plex/plex-bd956645c-wxsbd" Jul 24 08:58:35 bluth k3s[11030]: I0724 08:58:35.462517 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/host-path/01a1ac7e-39b1-4370-bf89-2be6180ecda9-data\") pod \"plex-bd956645c-wxsbd\" (UID: \"01a1ac7e-39b1-4370-bf89-2be6180ecda9\") " pod="ix-plex/plex-bd956645c-wxsbd" Jul 24 08:58:35 bluth k3s[11030]: I0724 08:58:35.462543 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w28nn\" (UniqueName: \"kubernetes.io/projected/01a1ac7e-39b1-4370-bf89-2be6180ecda9-kube-api-access-w28nn\") pod \"plex-bd956645c-wxsbd\" (UID: \"01a1ac7e-39b1-4370-bf89-2be6180ecda9\") " pod="ix-plex/plex-bd956645c-wxsbd" Jul 24 08:58:35 bluth k3s[11030]: I0724 08:58:35.462566 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/01a1ac7e-39b1-4370-bf89-2be6180ecda9-config\") pod \"plex-bd956645c-wxsbd\" (UID: \"01a1ac7e-39b1-4370-bf89-2be6180ecda9\") " pod="ix-plex/plex-bd956645c-wxsbd" Jul 24 08:58:35 bluth k3s[11030]: I0724 08:58:35.462587 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"transcode\" (UniqueName: \"kubernetes.io/host-path/01a1ac7e-39b1-4370-bf89-2be6180ecda9-transcode\") pod \"plex-bd956645c-wxsbd\" (UID: \"01a1ac7e-39b1-4370-bf89-2be6180ecda9\") " pod="ix-plex/plex-bd956645c-wxsbd" Jul 24 08:58:35 bluth k3s[11030]: I0724 08:58:35.462614 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"shared-logs\" (UniqueName: \"kubernetes.io/empty-dir/01a1ac7e-39b1-4370-bf89-2be6180ecda9-shared-logs\") pod \"plex-bd956645c-wxsbd\" (UID: \"01a1ac7e-39b1-4370-bf89-2be6180ecda9\") " pod="ix-plex/plex-bd956645c-wxsbd" Jul 24 08:58:35 bluth k3s[11030]: I0724 08:58:35.462652 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plex-probe-check\" (UniqueName: \"kubernetes.io/configmap/01a1ac7e-39b1-4370-bf89-2be6180ecda9-plex-probe-check\") pod \"plex-bd956645c-wxsbd\" (UID: \"01a1ac7e-39b1-4370-bf89-2be6180ecda9\") " pod="ix-plex/plex-bd956645c-wxsbd" Jul 24 08:58:35 bluth k3s[11030]: I0724 08:58:35.563743 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-2e6b3d10-435c-4ce5-806d-61d551dcef05\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-2e6b3d10-435c-4ce5-806d-61d551dcef05\") pod \"tdarr-746d65b466-vdnf9\" (UID: \"33f86e11-0114-4a11-806f-1adb1743ea52\") " pod="ix-tdarr/tdarr-746d65b466-vdnf9" Jul 24 08:58:35 bluth k3s[11030]: I0724 08:58:35.564195 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-028dc34a-8c2e-4a0a-b507-c33a02541407\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-028dc34a-8c2e-4a0a-b507-c33a02541407\") pod \"tdarr-746d65b466-vdnf9\" (UID: \"33f86e11-0114-4a11-806f-1adb1743ea52\") " pod="ix-tdarr/tdarr-746d65b466-vdnf9" Jul 24 08:58:35 bluth k3s[11030]: I0724 08:58:35.564279 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"list-0\" (UniqueName: \"kubernetes.io/host-path/33f86e11-0114-4a11-806f-1adb1743ea52-list-0\") pod \"tdarr-746d65b466-vdnf9\" (UID: \"33f86e11-0114-4a11-806f-1adb1743ea52\") " pod="ix-tdarr/tdarr-746d65b466-vdnf9" Jul 24 08:58:35 bluth k3s[11030]: I0724 08:58:35.686485 11030 csi_attacher.go:354] kubernetes.io/csi: attacher.MountDevice STAGE_UNSTAGE_VOLUME capability not set. Skipping MountDevice... Jul 24 08:58:35 bluth k3s[11030]: I0724 08:58:35.686527 11030 csi_attacher.go:354] kubernetes.io/csi: attacher.MountDevice STAGE_UNSTAGE_VOLUME capability not set. Skipping MountDevice... Jul 24 08:58:35 bluth k3s[11030]: I0724 08:58:35.687432 11030 operation_generator.go:674] "MountVolume.MountDevice succeeded for volume \"pvc-028dc34a-8c2e-4a0a-b507-c33a02541407\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-028dc34a-8c2e-4a0a-b507-c33a02541407\") pod \"tdarr-746d65b466-vdnf9\" (UID: \"33f86e11-0114-4a11-806f-1adb1743ea52\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-028dc34a-8c2e-4a0a-b507-c33a02541407/globalmount\"" pod="ix-tdarr/tdarr-746d65b466-vdnf9" Jul 24 08:58:35 bluth k3s[11030]: I0724 08:58:35.687442 11030 operation_generator.go:674] "MountVolume.MountDevice succeeded for volume \"pvc-2e6b3d10-435c-4ce5-806d-61d551dcef05\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-2e6b3d10-435c-4ce5-806d-61d551dcef05\") pod \"tdarr-746d65b466-vdnf9\" (UID: \"33f86e11-0114-4a11-806f-1adb1743ea52\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-2e6b3d10-435c-4ce5-806d-61d551dcef05/globalmount\"" pod="ix-tdarr/tdarr-746d65b466-vdnf9" Jul 24 08:58:35 bluth k3s[11030]: E0724 08:58:35.801297 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:58:36 bluth k3s[11030]: I0724 08:58:36.067652 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-3706f211-9694-4ec9-88b9-55f66b73b802\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-3706f211-9694-4ec9-88b9-55f66b73b802\") pod \"tdarr-746d65b466-vdnf9\" (UID: \"33f86e11-0114-4a11-806f-1adb1743ea52\") " pod="ix-tdarr/tdarr-746d65b466-vdnf9" Jul 24 08:58:36 bluth k3s[11030]: I0724 08:58:36.173832 11030 csi_attacher.go:354] kubernetes.io/csi: attacher.MountDevice STAGE_UNSTAGE_VOLUME capability not set. Skipping MountDevice... Jul 24 08:58:36 bluth k3s[11030]: I0724 08:58:36.173937 11030 operation_generator.go:674] "MountVolume.MountDevice succeeded for volume \"pvc-3706f211-9694-4ec9-88b9-55f66b73b802\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-3706f211-9694-4ec9-88b9-55f66b73b802\") pod \"tdarr-746d65b466-vdnf9\" (UID: \"33f86e11-0114-4a11-806f-1adb1743ea52\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3706f211-9694-4ec9-88b9-55f66b73b802/globalmount\"" pod="ix-tdarr/tdarr-746d65b466-vdnf9" Jul 24 08:58:36 bluth k3s[11030]: I0724 08:58:36.184212 11030 scope.go:110] "RemoveContainer" containerID="8ccd98bfe3f8ac373925d4c0f4d05f2989800d69d62f7e4413e1c55054f6c9ad" Jul 24 08:58:36 bluth k3s[11030]: E0724 08:58:36.185429 11030 kuberuntime_gc.go:146] "Failed to remove container" err="failed to get container status \"8ccd98bfe3f8ac373925d4c0f4d05f2989800d69d62f7e4413e1c55054f6c9ad\": rpc error: code = Unknown desc = Error: No such container: 8ccd98bfe3f8ac373925d4c0f4d05f2989800d69d62f7e4413e1c55054f6c9ad" containerID="8ccd98bfe3f8ac373925d4c0f4d05f2989800d69d62f7e4413e1c55054f6c9ad" Jul 24 08:58:36 bluth k3s[11030]: I0724 08:58:36.185481 11030 scope.go:110] "RemoveContainer" containerID="d175307b98d3f21c64114da633d5e59ab4473eca8985300d01b7a1a7880b36b4" Jul 24 08:58:36 bluth k3s[11030]: E0724 08:58:36.186564 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: d175307b98d3f21c64114da633d5e59ab4473eca8985300d01b7a1a7880b36b4" containerID="d175307b98d3f21c64114da633d5e59ab4473eca8985300d01b7a1a7880b36b4" Jul 24 08:58:36 bluth k3s[11030]: E0724 08:58:36.186635 11030 kuberuntime_gc.go:146] "Failed to remove container" err="failed to get container status \"d175307b98d3f21c64114da633d5e59ab4473eca8985300d01b7a1a7880b36b4\": rpc error: code = Unknown desc = Error: No such container: d175307b98d3f21c64114da633d5e59ab4473eca8985300d01b7a1a7880b36b4" containerID="d175307b98d3f21c64114da633d5e59ab4473eca8985300d01b7a1a7880b36b4" Jul 24 08:58:36 bluth k3s[11030]: I0724 08:58:36.186669 11030 scope.go:110] "RemoveContainer" containerID="7673b83d4f747be04e8b0f40d7c8595fa93a8d385addce36ab0ea5dbbbbd9c5d" Jul 24 08:58:36 bluth k3s[11030]: I0724 08:58:36.374222 11030 scope.go:110] "RemoveContainer" containerID="fb1930eb1ef400fea43e204a0e2614accadb77edf3d228fab933866a17b73cbe" Jul 24 08:58:37 bluth k3s[11030]: I0724 08:58:37.078718 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp\" (UniqueName: \"kubernetes.io/empty-dir/33f86e11-0114-4a11-806f-1adb1743ea52-temp\") pod \"tdarr-746d65b466-vdnf9\" (UID: \"33f86e11-0114-4a11-806f-1adb1743ea52\") " pod="ix-tdarr/tdarr-746d65b466-vdnf9" Jul 24 08:58:37 bluth k3s[11030]: I0724 08:58:37.078833 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-4e6fcc6b-6890-4751-a09a-75bb6cde146f\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-4e6fcc6b-6890-4751-a09a-75bb6cde146f\") pod \"tdarr-746d65b466-vdnf9\" (UID: \"33f86e11-0114-4a11-806f-1adb1743ea52\") " pod="ix-tdarr/tdarr-746d65b466-vdnf9" Jul 24 08:58:37 bluth k3s[11030]: I0724 08:58:37.079058 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"shared\" (UniqueName: \"kubernetes.io/empty-dir/33f86e11-0114-4a11-806f-1adb1743ea52-shared\") pod \"tdarr-746d65b466-vdnf9\" (UID: \"33f86e11-0114-4a11-806f-1adb1743ea52\") " pod="ix-tdarr/tdarr-746d65b466-vdnf9" Jul 24 08:58:37 bluth k3s[11030]: I0724 08:58:37.183317 11030 csi_attacher.go:354] kubernetes.io/csi: attacher.MountDevice STAGE_UNSTAGE_VOLUME capability not set. Skipping MountDevice... Jul 24 08:58:37 bluth k3s[11030]: I0724 08:58:37.183416 11030 operation_generator.go:674] "MountVolume.MountDevice succeeded for volume \"pvc-4e6fcc6b-6890-4751-a09a-75bb6cde146f\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-4e6fcc6b-6890-4751-a09a-75bb6cde146f\") pod \"tdarr-746d65b466-vdnf9\" (UID: \"33f86e11-0114-4a11-806f-1adb1743ea52\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-4e6fcc6b-6890-4751-a09a-75bb6cde146f/globalmount\"" pod="ix-tdarr/tdarr-746d65b466-vdnf9" Jul 24 08:58:37 bluth k3s[11030]: I0724 08:58:37.685427 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2bdf\" (UniqueName: \"kubernetes.io/projected/33f86e11-0114-4a11-806f-1adb1743ea52-kube-api-access-r2bdf\") pod \"tdarr-746d65b466-vdnf9\" (UID: \"33f86e11-0114-4a11-806f-1adb1743ea52\") " pod="ix-tdarr/tdarr-746d65b466-vdnf9" Jul 24 08:58:37 bluth k3s[11030]: I0724 08:58:37.685689 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-c41ca0c8-d532-4ca2-b6ca-fe3e9ddf87d2\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c41ca0c8-d532-4ca2-b6ca-fe3e9ddf87d2\") pod \"tdarr-746d65b466-vdnf9\" (UID: \"33f86e11-0114-4a11-806f-1adb1743ea52\") " pod="ix-tdarr/tdarr-746d65b466-vdnf9" Jul 24 08:58:37 bluth k3s[11030]: I0724 08:58:37.685755 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"varrun\" (UniqueName: \"kubernetes.io/empty-dir/33f86e11-0114-4a11-806f-1adb1743ea52-varrun\") pod \"tdarr-746d65b466-vdnf9\" (UID: \"33f86e11-0114-4a11-806f-1adb1743ea52\") " pod="ix-tdarr/tdarr-746d65b466-vdnf9" Jul 24 08:58:37 bluth k3s[11030]: I0724 08:58:37.686015 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"varlogs\" (UniqueName: \"kubernetes.io/empty-dir/33f86e11-0114-4a11-806f-1adb1743ea52-varlogs\") pod \"tdarr-746d65b466-vdnf9\" (UID: \"33f86e11-0114-4a11-806f-1adb1743ea52\") " pod="ix-tdarr/tdarr-746d65b466-vdnf9" Jul 24 08:58:37 bluth k3s[11030]: E0724 08:58:37.755209 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:58:37 bluth k3s[11030]: I0724 08:58:37.789992 11030 csi_attacher.go:354] kubernetes.io/csi: attacher.MountDevice STAGE_UNSTAGE_VOLUME capability not set. Skipping MountDevice... Jul 24 08:58:37 bluth k3s[11030]: I0724 08:58:37.790085 11030 operation_generator.go:674] "MountVolume.MountDevice succeeded for volume \"pvc-c41ca0c8-d532-4ca2-b6ca-fe3e9ddf87d2\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c41ca0c8-d532-4ca2-b6ca-fe3e9ddf87d2\") pod \"tdarr-746d65b466-vdnf9\" (UID: \"33f86e11-0114-4a11-806f-1adb1743ea52\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-c41ca0c8-d532-4ca2-b6ca-fe3e9ddf87d2/globalmount\"" pod="ix-tdarr/tdarr-746d65b466-vdnf9" Jul 24 08:58:37 bluth k3s[11030]: I0724 08:58:37.992074 11030 trace.go:205] Trace[775108213]: "Get" url:/api/v1/namespaces/ix-home-assistant/pods/home-assistant-ix-chart-54bf5d89c8-5rpjh/log,user-agent:OpenAPI-Generator/22.6.0-snapshot/python,audit-id:20ee518f-11b1-46ac-84a9-2d0b6ee4ba38,client:127.0.0.1,accept:application/json,protocol:HTTP/1.1 (24-Jul-2022 08:58:15.007) (total time: 22984ms): Jul 24 08:58:37 bluth k3s[11030]: Trace[775108213]: ---"Transformed response object" 22982ms (08:58:37.991) Jul 24 08:58:37 bluth k3s[11030]: Trace[775108213]: [22.984201535s] [22.984201535s] END Jul 24 08:58:37 bluth k3s[11030]: I0724 08:58:37.993563 11030 log.go:184] http: superfluous response.WriteHeader call from github.com/emicklei/go-restful.(*Response).WriteHeader (response.go:220) Jul 24 08:58:39 bluth k3s[11030]: E0724 08:58:39.760531 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:58:40 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T08:58:40.339+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 08:58:41 bluth k3s[11030]: E0724 08:58:41.767291 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:58:43 bluth k3s[11030]: I0724 08:58:43.502874 11030 scope.go:110] "RemoveContainer" containerID="9aadd1ea349c7b532f80a04fba909e332f1ac11dfb6e42442c62c52b076d7d8a" Jul 24 08:58:43 bluth k3s[11030]: I0724 08:58:43.540935 11030 scope.go:110] "RemoveContainer" containerID="cef632ebbb8791e11361d4dccb48f0aa8230796c41ed25a7b2bb1b2aa069c375" Jul 24 08:58:43 bluth k3s[11030]: E0724 08:58:43.756007 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:58:43 bluth k3s[11030]: I0724 08:58:43.895142 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="fb1930eb1ef400fea43e204a0e2614accadb77edf3d228fab933866a17b73cbe" Jul 24 08:58:45 bluth k3s[11030]: E0724 08:58:45.759305 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:58:47 bluth k3s[11030]: E0724 08:58:47.767365 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:58:48 bluth k3s[11030]: I0724 08:58:48.609921 11030 scope.go:110] "RemoveContainer" containerID="126b17c0eeded4927a8fbba1f626b3c39d327df91e5e9f0c24666d4e27cb1583" Jul 24 08:58:48 bluth k3s[11030]: I0724 08:58:48.845163 11030 scope.go:110] "RemoveContainer" containerID="fd4570ef9df9bcf5b972fe2c82b83f8100d6df535f7dc9742f803f496b0a4252" Jul 24 08:58:48 bluth k3s[11030]: E0724 08:58:48.846485 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: fd4570ef9df9bcf5b972fe2c82b83f8100d6df535f7dc9742f803f496b0a4252" containerID="fd4570ef9df9bcf5b972fe2c82b83f8100d6df535f7dc9742f803f496b0a4252" Jul 24 08:58:48 bluth k3s[11030]: E0724 08:58:48.846566 11030 kuberuntime_gc.go:146] "Failed to remove container" err="failed to get container status \"fd4570ef9df9bcf5b972fe2c82b83f8100d6df535f7dc9742f803f496b0a4252\": rpc error: code = Unknown desc = Error: No such container: fd4570ef9df9bcf5b972fe2c82b83f8100d6df535f7dc9742f803f496b0a4252" containerID="fd4570ef9df9bcf5b972fe2c82b83f8100d6df535f7dc9742f803f496b0a4252" Jul 24 08:58:48 bluth k3s[11030]: I0724 08:58:48.846606 11030 scope.go:110] "RemoveContainer" containerID="4aa1fa5008080c3b2af38749471d5c7de7ba967a09ee62310a859c0d7f9ec5f0" Jul 24 08:58:49 bluth k3s[11030]: I0724 08:58:49.039569 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="cef632ebbb8791e11361d4dccb48f0aa8230796c41ed25a7b2bb1b2aa069c375" Jul 24 08:58:49 bluth k3s[11030]: E0724 08:58:49.755105 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:58:50 bluth k3s[11030]: I0724 08:58:50.089542 11030 docker_sandbox.go:402] "Failed to read pod IP from plugin/docker" err="networkPlugin cni failed on the status hook for pod \"cloudflared-ix-chart-687fff6d55-t247w_ix-cloudflared\": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container \"a6327cc849b9115cec3b175ecb675c69b212d724d426e29b9149fcfcbe004693\"" Jul 24 08:58:51 bluth k3s[11030]: E0724 08:58:51.760242 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:58:53 bluth k3s[11030]: E0724 08:58:53.755237 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:58:53 bluth k3s[11030]: I0724 08:58:53.962647 11030 scope.go:110] "RemoveContainer" containerID="7673b83d4f747be04e8b0f40d7c8595fa93a8d385addce36ab0ea5dbbbbd9c5d" Jul 24 08:58:53 bluth k3s[11030]: I0724 08:58:53.962779 11030 scope.go:110] "RemoveContainer" containerID="cef632ebbb8791e11361d4dccb48f0aa8230796c41ed25a7b2bb1b2aa069c375" Jul 24 08:58:53 bluth k3s[11030]: E0724 08:58:53.963157 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 7673b83d4f747be04e8b0f40d7c8595fa93a8d385addce36ab0ea5dbbbbd9c5d" containerID="7673b83d4f747be04e8b0f40d7c8595fa93a8d385addce36ab0ea5dbbbbd9c5d" Jul 24 08:58:53 bluth k3s[11030]: I0724 08:58:53.963179 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:7673b83d4f747be04e8b0f40d7c8595fa93a8d385addce36ab0ea5dbbbbd9c5d} err="failed to get container status \"7673b83d4f747be04e8b0f40d7c8595fa93a8d385addce36ab0ea5dbbbbd9c5d\": rpc error: code = Unknown desc = Error: No such container: 7673b83d4f747be04e8b0f40d7c8595fa93a8d385addce36ab0ea5dbbbbd9c5d" Jul 24 08:58:53 bluth k3s[11030]: I0724 08:58:53.963187 11030 scope.go:110] "RemoveContainer" containerID="b29dd1db4c679a9a1cb59df49aea5454679e29f63d3a960523293d9f0df17e93" Jul 24 08:58:53 bluth k3s[11030]: E0724 08:58:53.963160 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: cef632ebbb8791e11361d4dccb48f0aa8230796c41ed25a7b2bb1b2aa069c375" containerID="cef632ebbb8791e11361d4dccb48f0aa8230796c41ed25a7b2bb1b2aa069c375" Jul 24 08:58:53 bluth k3s[11030]: E0724 08:58:53.963224 11030 kuberuntime_gc.go:146] "Failed to remove container" err="failed to get container status \"cef632ebbb8791e11361d4dccb48f0aa8230796c41ed25a7b2bb1b2aa069c375\": rpc error: code = Unknown desc = Error: No such container: cef632ebbb8791e11361d4dccb48f0aa8230796c41ed25a7b2bb1b2aa069c375" containerID="cef632ebbb8791e11361d4dccb48f0aa8230796c41ed25a7b2bb1b2aa069c375" Jul 24 08:58:55 bluth k3s[11030]: E0724 08:58:55.759325 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:58:56 bluth k3s[11030]: I0724 08:58:56.456926 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="126b17c0eeded4927a8fbba1f626b3c39d327df91e5e9f0c24666d4e27cb1583" Jul 24 08:58:56 bluth k3s[11030]: I0724 08:58:56.458339 11030 docker_sandbox.go:402] "Failed to read pod IP from plugin/docker" err="networkPlugin cni failed on the status hook for pod \"cloudflared-ix-chart-687fff6d55-t247w_ix-cloudflared\": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container \"a6327cc849b9115cec3b175ecb675c69b212d724d426e29b9149fcfcbe004693\"" Jul 24 08:58:57 bluth k3s[11030]: E0724 08:58:57.795305 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:58:58 bluth k3s[11030]: I0724 08:58:58.552537 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="b29dd1db4c679a9a1cb59df49aea5454679e29f63d3a960523293d9f0df17e93" Jul 24 08:58:59 bluth k3s[11030]: E0724 08:58:59.757314 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:59:00 bluth k3s[11030]: E0724 08:59:00.601495 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 83f7233eb32fd8f443286246267cc4c944b29e9248c22e92f48dcf8fe41694a6" podSandboxID="83f7233eb32fd8f443286246267cc4c944b29e9248c22e92f48dcf8fe41694a6" pod="ix-plex/plex-bd956645c-wxsbd" Jul 24 08:59:01 bluth k3s[11030]: E0724 08:59:01.615518 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 83f7233eb32fd8f443286246267cc4c944b29e9248c22e92f48dcf8fe41694a6" podSandboxID="83f7233eb32fd8f443286246267cc4c944b29e9248c22e92f48dcf8fe41694a6" pod="ix-plex/plex-bd956645c-wxsbd" Jul 24 08:59:01 bluth k3s[11030]: E0724 08:59:01.767483 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:59:02 bluth k3s[11030]: I0724 08:59:02.712271 11030 csi_attacher.go:354] kubernetes.io/csi: attacher.MountDevice STAGE_UNSTAGE_VOLUME capability not set. Skipping MountDevice... Jul 24 08:59:02 bluth k3s[11030]: I0724 08:59:02.712380 11030 operation_generator.go:674] "MountVolume.MountDevice succeeded for volume \"pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e\") pod \"esphome-658957fddc-6z2jm\" (UID: \"19254d20-7ad1-49a1-b408-f6e4213e5b4e\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-d5fec104-f302-4f41-b48f-bb42ab0f154e/globalmount\"" pod="ix-esphome/esphome-658957fddc-6z2jm" Jul 24 08:59:03 bluth k3s[11030]: I0724 08:59:03.520720 11030 csi_attacher.go:354] kubernetes.io/csi: attacher.MountDevice STAGE_UNSTAGE_VOLUME capability not set. Skipping MountDevice... Jul 24 08:59:03 bluth k3s[11030]: I0724 08:59:03.520816 11030 operation_generator.go:674] "MountVolume.MountDevice succeeded for volume \"pvc-06521873-c179-499e-9079-194ec6618fdd\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-06521873-c179-499e-9079-194ec6618fdd\") pod \"esphome-658957fddc-6z2jm\" (UID: \"19254d20-7ad1-49a1-b408-f6e4213e5b4e\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-06521873-c179-499e-9079-194ec6618fdd/globalmount\"" pod="ix-esphome/esphome-658957fddc-6z2jm" Jul 24 08:59:03 bluth k3s[11030]: E0724 08:59:03.762687 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:59:04 bluth k3s[11030]: I0724 08:59:04.931356 11030 csi_attacher.go:354] kubernetes.io/csi: attacher.MountDevice STAGE_UNSTAGE_VOLUME capability not set. Skipping MountDevice... Jul 24 08:59:04 bluth k3s[11030]: I0724 08:59:04.931384 11030 operation_generator.go:674] "MountVolume.MountDevice succeeded for volume \"pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274\") pod \"prowlarr-5bcc99cc47-v86hm\" (UID: \"5d1f2955-de8a-4e25-b1f8-e607463ed369\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-232229d0-a8ca-4b2e-9e92-8ad4c895b274/globalmount\"" pod="ix-prowlarr/prowlarr-5bcc99cc47-v86hm" Jul 24 08:59:05 bluth k3s[11030]: I0724 08:59:05.268467 11030 scope.go:110] "RemoveContainer" containerID="0ffa7770fac2800c05da6625137d616f559d5c73c8816fc7d97b1f5b06727d14" Jul 24 08:59:05 bluth k3s[11030]: I0724 08:59:05.434506 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cr6l5\" (UniqueName: \"kubernetes.io/projected/508c7225-82e1-4311-9b86-6a6ad8bf3740-kube-api-access-cr6l5\") pod \"508c7225-82e1-4311-9b86-6a6ad8bf3740\" (UID: \"508c7225-82e1-4311-9b86-6a6ad8bf3740\") " Jul 24 08:59:05 bluth k3s[11030]: I0724 08:59:05.434621 11030 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/empty-dir/508c7225-82e1-4311-9b86-6a6ad8bf3740-socket-dir\") pod \"508c7225-82e1-4311-9b86-6a6ad8bf3740\" (UID: \"508c7225-82e1-4311-9b86-6a6ad8bf3740\") " Jul 24 08:59:05 bluth k3s[11030]: W0724 08:59:05.435634 11030 empty_dir.go:517] Warning: Failed to clear quota on /var/lib/kubelet/pods/508c7225-82e1-4311-9b86-6a6ad8bf3740/volumes/kubernetes.io~empty-dir/socket-dir: clearQuota called, but quotas disabled Jul 24 08:59:05 bluth k3s[11030]: I0724 08:59:05.436204 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/508c7225-82e1-4311-9b86-6a6ad8bf3740-socket-dir" (OuterVolumeSpecName: "socket-dir") pod "508c7225-82e1-4311-9b86-6a6ad8bf3740" (UID: "508c7225-82e1-4311-9b86-6a6ad8bf3740"). InnerVolumeSpecName "socket-dir". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Jul 24 08:59:05 bluth k3s[11030]: I0724 08:59:05.439178 11030 operation_generator.go:910] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/508c7225-82e1-4311-9b86-6a6ad8bf3740-kube-api-access-cr6l5" (OuterVolumeSpecName: "kube-api-access-cr6l5") pod "508c7225-82e1-4311-9b86-6a6ad8bf3740" (UID: "508c7225-82e1-4311-9b86-6a6ad8bf3740"). InnerVolumeSpecName "kube-api-access-cr6l5". PluginName "kubernetes.io/projected", VolumeGidValue "" Jul 24 08:59:05 bluth k3s[11030]: I0724 08:59:05.536112 11030 reconciler.go:300] "Volume detached for volume \"kube-api-access-cr6l5\" (UniqueName: \"kubernetes.io/projected/508c7225-82e1-4311-9b86-6a6ad8bf3740-kube-api-access-cr6l5\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:59:05 bluth k3s[11030]: I0724 08:59:05.536180 11030 reconciler.go:300] "Volume detached for volume \"socket-dir\" (UniqueName: \"kubernetes.io/empty-dir/508c7225-82e1-4311-9b86-6a6ad8bf3740-socket-dir\") on node \"ix-truenas\" DevicePath \"\"" Jul 24 08:59:05 bluth k3s[11030]: E0724 08:59:05.765036 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:59:05 bluth k3s[11030]: I0724 08:59:05.942106 11030 csi_attacher.go:354] kubernetes.io/csi: attacher.MountDevice STAGE_UNSTAGE_VOLUME capability not set. Skipping MountDevice... Jul 24 08:59:05 bluth k3s[11030]: I0724 08:59:05.942200 11030 operation_generator.go:674] "MountVolume.MountDevice succeeded for volume \"pvc-58562e12-4adc-4e83-8131-848f1b39bf00\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-58562e12-4adc-4e83-8131-848f1b39bf00\") pod \"qbittorrent-6d7cf9f7cd-xrqwj\" (UID: \"4d43a190-0b41-4f84-a84d-ce295694fcb5\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-58562e12-4adc-4e83-8131-848f1b39bf00/globalmount\"" pod="ix-qbittorrent/qbittorrent-6d7cf9f7cd-xrqwj" Jul 24 08:59:07 bluth k3s[11030]: I0724 08:59:07.153822 11030 csi_attacher.go:354] kubernetes.io/csi: attacher.MountDevice STAGE_UNSTAGE_VOLUME capability not set. Skipping MountDevice... Jul 24 08:59:07 bluth k3s[11030]: I0724 08:59:07.153922 11030 operation_generator.go:674] "MountVolume.MountDevice succeeded for volume \"pvc-195f1a32-9c93-4737-983a-8dacee32e453\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-195f1a32-9c93-4737-983a-8dacee32e453\") pod \"radarr-7486f78d7f-ntt44\" (UID: \"9ab9d8fe-e669-40bb-a0ce-7f7a4e5a69b5\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-195f1a32-9c93-4737-983a-8dacee32e453/globalmount\"" pod="ix-radarr/radarr-7486f78d7f-ntt44" Jul 24 08:59:07 bluth k3s[11030]: E0724 08:59:07.754963 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:59:07 bluth k3s[11030]: E0724 08:59:07.761975 11030 kubelet.go:1730] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[config], unattached volumes=[shared temp varlogs list-0 kube-api-access-m8kjp config]: timed out waiting for the condition" pod="ix-sonarr/sonarr-867d7dd548-fw6z4" Jul 24 08:59:07 bluth k3s[11030]: E0724 08:59:07.762044 11030 pod_workers.go:949] "Error syncing pod, skipping" err="unmounted volumes=[config], unattached volumes=[shared temp varlogs list-0 kube-api-access-m8kjp config]: timed out waiting for the condition" pod="ix-sonarr/sonarr-867d7dd548-fw6z4" podUID=1a1846b0-3361-4f4b-a1cd-d5f958d6a7ac Jul 24 08:59:08 bluth k3s[11030]: I0724 08:59:08.160581 11030 csi_attacher.go:354] kubernetes.io/csi: attacher.MountDevice STAGE_UNSTAGE_VOLUME capability not set. Skipping MountDevice... Jul 24 08:59:08 bluth k3s[11030]: I0724 08:59:08.160610 11030 operation_generator.go:674] "MountVolume.MountDevice succeeded for volume \"pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4\") pod \"sonarr-867d7dd548-fw6z4\" (UID: \"1a1846b0-3361-4f4b-a1cd-d5f958d6a7ac\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-ee85c0d5-50c7-4011-b9f6-dd70968655c4/globalmount\"" pod="ix-sonarr/sonarr-867d7dd548-fw6z4" Jul 24 08:59:09 bluth k3s[11030]: E0724 08:59:09.754715 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:59:10 bluth k3s[11030]: E0724 08:59:10.592237 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 250ac9919c437ad0c19523b9bf8bd790820d43ad3466a8ecfb362746d58334de" podSandboxID="250ac9919c437ad0c19523b9bf8bd790820d43ad3466a8ecfb362746d58334de" pod="ix-tdarr/tdarr-746d65b466-vdnf9" Jul 24 08:59:10 bluth k3s[11030]: I0724 08:59:10.615627 11030 event.go:294] "Event occurred" object="kube-system/openebs-zfs-controller" kind="StatefulSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulCreate" message="create Pod openebs-zfs-controller-0 in StatefulSet openebs-zfs-controller successful" Jul 24 08:59:10 bluth k3s[11030]: I0724 08:59:10.619413 11030 topology_manager.go:200] "Topology Admit Handler" Jul 24 08:59:10 bluth k3s[11030]: I0724 08:59:10.781071 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2vgw\" (UniqueName: \"kubernetes.io/projected/18ad2548-e271-464c-b16a-22db78a782a4-kube-api-access-p2vgw\") pod \"openebs-zfs-controller-0\" (UID: \"18ad2548-e271-464c-b16a-22db78a782a4\") " pod="kube-system/openebs-zfs-controller-0" Jul 24 08:59:10 bluth k3s[11030]: I0724 08:59:10.781143 11030 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/empty-dir/18ad2548-e271-464c-b16a-22db78a782a4-socket-dir\") pod \"openebs-zfs-controller-0\" (UID: \"18ad2548-e271-464c-b16a-22db78a782a4\") " pod="kube-system/openebs-zfs-controller-0" Jul 24 08:59:11 bluth k3s[11030]: E0724 08:59:11.621613 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 250ac9919c437ad0c19523b9bf8bd790820d43ad3466a8ecfb362746d58334de" podSandboxID="250ac9919c437ad0c19523b9bf8bd790820d43ad3466a8ecfb362746d58334de" pod="ix-tdarr/tdarr-746d65b466-vdnf9" Jul 24 08:59:11 bluth k3s[11030]: E0724 08:59:11.622495 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 250ac9919c437ad0c19523b9bf8bd790820d43ad3466a8ecfb362746d58334de" podSandboxID="250ac9919c437ad0c19523b9bf8bd790820d43ad3466a8ecfb362746d58334de" pod="ix-tdarr/tdarr-746d65b466-vdnf9" Jul 24 08:59:11 bluth k3s[11030]: I0724 08:59:11.757293 11030 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=508c7225-82e1-4311-9b86-6a6ad8bf3740 path="/var/lib/kubelet/pods/508c7225-82e1-4311-9b86-6a6ad8bf3740/volumes" Jul 24 08:59:12 bluth k3s[11030]: E0724 08:59:12.314137 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:59:12 bluth k3s[11030]: E0724 08:59:12.648539 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 250ac9919c437ad0c19523b9bf8bd790820d43ad3466a8ecfb362746d58334de" podSandboxID="250ac9919c437ad0c19523b9bf8bd790820d43ad3466a8ecfb362746d58334de" pod="ix-tdarr/tdarr-746d65b466-vdnf9" Jul 24 08:59:12 bluth k3s[11030]: E0724 08:59:12.649446 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 250ac9919c437ad0c19523b9bf8bd790820d43ad3466a8ecfb362746d58334de" podSandboxID="250ac9919c437ad0c19523b9bf8bd790820d43ad3466a8ecfb362746d58334de" pod="ix-tdarr/tdarr-746d65b466-vdnf9" Jul 24 08:59:13 bluth k3s[11030]: E0724 08:59:13.790194 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:59:13 bluth k3s[11030]: I0724 08:59:13.790289 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="250ac9919c437ad0c19523b9bf8bd790820d43ad3466a8ecfb362746d58334de" Jul 24 08:59:15 bluth k3s[11030]: E0724 08:59:15.755976 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:59:16 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T08:59:16.309+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 08:59:17 bluth k3s[11030]: E0724 08:59:17.759692 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:59:19 bluth k3s[11030]: E0724 08:59:19.757098 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:59:20 bluth k3s[11030]: I0724 08:59:20.993548 11030 scope.go:110] "RemoveContainer" containerID="06eefc12a0bf11c5a58f4cabc3c155642c0efb73cb044d8a293548d860b65c1d" Jul 24 08:59:21 bluth k3s[11030]: E0724 08:59:21.760062 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:59:23 bluth k3s[11030]: E0724 08:59:23.755392 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:59:25 bluth k3s[11030]: E0724 08:59:25.754738 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:59:27 bluth k3s[11030]: I0724 08:59:27.552823 45908 event.go:278] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ix-tdarr", Name:"tdarr-746d65b466-vdnf9", UID:"33f86e11-0114-4a11-806f-1adb1743ea52", APIVersion:"v1", ResourceVersion:"3028195", FieldPath:""}): type: 'Normal' reason: 'AddedInterface' Add eth0 [172.16.0.207/16] from ix-net Jul 24 08:59:27 bluth k3s[11030]: E0724 08:59:27.751223 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:59:29 bluth k3s[11030]: E0724 08:59:29.759188 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:59:31 bluth k3s[11030]: E0724 08:59:31.761002 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:59:33 bluth k3s[11030]: I0724 08:59:33.082911 11030 scope.go:110] "RemoveContainer" containerID="9132e315e0d5bc7b9e04b040529b2d2a7d22802b8015cee1d336243b41077fbd" Jul 24 08:59:33 bluth k3s[11030]: E0724 08:59:33.776514 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:59:35 bluth k3s[11030]: E0724 08:59:35.755512 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:59:37 bluth k3s[11030]: E0724 08:59:37.791828 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:59:39 bluth k3s[11030]: E0724 08:59:39.755407 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:59:41 bluth k3s[11030]: E0724 08:59:41.784009 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:59:42 bluth k3s[11030]: I0724 08:59:42.885911 11030 scope.go:110] "RemoveContainer" containerID="72ac180a3a68d798f24195ff9f7f801b1dbb04a1566caadc0e7e6cf4c30013a2" Jul 24 08:59:43 bluth k3s[11030]: I0724 08:59:43.684674 11030 trace.go:205] Trace[836728675]: "GuaranteedUpdate etcd3" type:*coordination.Lease (24-Jul-2022 08:59:41.948) (total time: 1736ms): Jul 24 08:59:43 bluth k3s[11030]: Trace[836728675]: ---"Transaction committed" 1735ms (08:59:43.684) Jul 24 08:59:43 bluth k3s[11030]: Trace[836728675]: [1.736259647s] [1.736259647s] END Jul 24 08:59:43 bluth k3s[11030]: I0724 08:59:43.684770 11030 trace.go:205] Trace[652666978]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ix-truenas,user-agent:Go-http-client/2.0,audit-id:a4cd7954-cf48-4f49-a3ea-6cae1b118e26,client:127.0.0.1,accept:application/vnd.kubernetes.protobuf,application/json,protocol:HTTP/2.0 (24-Jul-2022 08:59:41.948) (total time: 1736ms): Jul 24 08:59:43 bluth k3s[11030]: Trace[652666978]: ---"Object stored in database" 1736ms (08:59:43.684) Jul 24 08:59:43 bluth k3s[11030]: Trace[652666978]: [1.73641146s] [1.73641146s] END Jul 24 08:59:43 bluth k3s[11030]: E0724 08:59:43.755635 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:59:45 bluth k3s[11030]: E0724 08:59:45.761195 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:59:47 bluth k3s[11030]: E0724 08:59:47.765546 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:59:49 bluth k3s[11030]: E0724 08:59:49.755454 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:59:51 bluth k3s[11030]: E0724 08:59:51.773918 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:59:52 bluth k3s[11030]: I0724 08:59:52.200774 11030 scope.go:110] "RemoveContainer" containerID="d4aed5c62635e97bfd4e98215c786fc4419297c23985754ae4f57fc76cc6c21c" Jul 24 08:59:53 bluth k3s[11030]: E0724 08:59:53.755587 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:59:55 bluth k3s[11030]: I0724 08:59:55.165476 11030 trace.go:205] Trace[1503052995]: "GuaranteedUpdate etcd3" type:*coordination.Lease (24-Jul-2022 08:59:53.781) (total time: 1383ms): Jul 24 08:59:55 bluth k3s[11030]: Trace[1503052995]: ---"Transaction committed" 1383ms (08:59:55.165) Jul 24 08:59:55 bluth k3s[11030]: Trace[1503052995]: [1.383693761s] [1.383693761s] END Jul 24 08:59:55 bluth k3s[11030]: I0724 08:59:55.165749 11030 trace.go:205] Trace[976097078]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ix-truenas,user-agent:Go-http-client/2.0,audit-id:c1112598-0921-49b3-a05e-2f94e76347ea,client:127.0.0.1,accept:application/vnd.kubernetes.protobuf,application/json,protocol:HTTP/2.0 (24-Jul-2022 08:59:53.781) (total time: 1384ms): Jul 24 08:59:55 bluth k3s[11030]: Trace[976097078]: ---"Object stored in database" 1383ms (08:59:55.165) Jul 24 08:59:55 bluth k3s[11030]: Trace[976097078]: [1.384043646s] [1.384043646s] END Jul 24 08:59:55 bluth k3s[11030]: E0724 08:59:55.759319 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:59:57 bluth k3s[11030]: E0724 08:59:57.786524 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:59:57 bluth k3s[11030]: E0724 08:59:57.960133 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" podSandboxID="14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" pod="ix-esphome/esphome-658957fddc-6z2jm" Jul 24 08:59:58 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T08:59:58.600+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 08:59:58 bluth k3s[11030]: E0724 08:59:58.974488 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" podSandboxID="14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" pod="ix-esphome/esphome-658957fddc-6z2jm" Jul 24 08:59:58 bluth k3s[11030]: E0724 08:59:58.975334 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" podSandboxID="14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" pod="ix-esphome/esphome-658957fddc-6z2jm" Jul 24 08:59:59 bluth k3s[11030]: E0724 08:59:59.755777 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 08:59:59 bluth k3s[11030]: E0724 08:59:59.989244 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" podSandboxID="14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" pod="ix-esphome/esphome-658957fddc-6z2jm" Jul 24 08:59:59 bluth k3s[11030]: E0724 08:59:59.990095 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" podSandboxID="14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" pod="ix-esphome/esphome-658957fddc-6z2jm" Jul 24 09:00:01 bluth k3s[11030]: E0724 09:00:01.004237 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" podSandboxID="14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" pod="ix-esphome/esphome-658957fddc-6z2jm" Jul 24 09:00:01 bluth k3s[11030]: E0724 09:00:01.005090 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" podSandboxID="14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" pod="ix-esphome/esphome-658957fddc-6z2jm" Jul 24 09:00:01 bluth k3s[11030]: E0724 09:00:01.759528 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:00:02 bluth k3s[11030]: E0724 09:00:02.019562 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" podSandboxID="14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" pod="ix-esphome/esphome-658957fddc-6z2jm" Jul 24 09:00:02 bluth k3s[11030]: E0724 09:00:02.020547 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" podSandboxID="14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" pod="ix-esphome/esphome-658957fddc-6z2jm" Jul 24 09:00:03 bluth k3s[11030]: E0724 09:00:03.034456 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" podSandboxID="14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" pod="ix-esphome/esphome-658957fddc-6z2jm" Jul 24 09:00:03 bluth k3s[11030]: E0724 09:00:03.035290 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" podSandboxID="14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" pod="ix-esphome/esphome-658957fddc-6z2jm" Jul 24 09:00:03 bluth k3s[11030]: E0724 09:00:03.761798 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:00:04 bluth k3s[11030]: E0724 09:00:04.048512 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" podSandboxID="14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" pod="ix-esphome/esphome-658957fddc-6z2jm" Jul 24 09:00:04 bluth k3s[11030]: E0724 09:00:04.049311 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" podSandboxID="14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" pod="ix-esphome/esphome-658957fddc-6z2jm" Jul 24 09:00:05 bluth k3s[11030]: E0724 09:00:05.074760 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" podSandboxID="14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" pod="ix-esphome/esphome-658957fddc-6z2jm" Jul 24 09:00:05 bluth k3s[11030]: E0724 09:00:05.075694 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: df80f0d931640bcaca997a47eefbd3ea29f10222b73cec2ec6561d8b91890780" podSandboxID="df80f0d931640bcaca997a47eefbd3ea29f10222b73cec2ec6561d8b91890780" pod="ix-prowlarr/prowlarr-5bcc99cc47-v86hm" Jul 24 09:00:05 bluth k3s[11030]: E0724 09:00:05.076480 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" podSandboxID="14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" pod="ix-esphome/esphome-658957fddc-6z2jm" Jul 24 09:00:05 bluth k3s[11030]: E0724 09:00:05.755503 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:00:06 bluth k3s[11030]: E0724 09:00:06.109969 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: df80f0d931640bcaca997a47eefbd3ea29f10222b73cec2ec6561d8b91890780" podSandboxID="df80f0d931640bcaca997a47eefbd3ea29f10222b73cec2ec6561d8b91890780" pod="ix-prowlarr/prowlarr-5bcc99cc47-v86hm" Jul 24 09:00:06 bluth k3s[11030]: E0724 09:00:06.113792 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" podSandboxID="14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" pod="ix-esphome/esphome-658957fddc-6z2jm" Jul 24 09:00:06 bluth k3s[11030]: E0724 09:00:06.116932 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" podSandboxID="14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" pod="ix-esphome/esphome-658957fddc-6z2jm" Jul 24 09:00:06 bluth k3s[11030]: E0724 09:00:06.120127 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: df80f0d931640bcaca997a47eefbd3ea29f10222b73cec2ec6561d8b91890780" podSandboxID="df80f0d931640bcaca997a47eefbd3ea29f10222b73cec2ec6561d8b91890780" pod="ix-prowlarr/prowlarr-5bcc99cc47-v86hm" Jul 24 09:00:07 bluth k3s[11030]: E0724 09:00:07.129198 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" podSandboxID="14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" pod="ix-esphome/esphome-658957fddc-6z2jm" Jul 24 09:00:07 bluth k3s[11030]: E0724 09:00:07.130061 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: df80f0d931640bcaca997a47eefbd3ea29f10222b73cec2ec6561d8b91890780" podSandboxID="df80f0d931640bcaca997a47eefbd3ea29f10222b73cec2ec6561d8b91890780" pod="ix-prowlarr/prowlarr-5bcc99cc47-v86hm" Jul 24 09:00:07 bluth k3s[11030]: E0724 09:00:07.130921 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: df80f0d931640bcaca997a47eefbd3ea29f10222b73cec2ec6561d8b91890780" podSandboxID="df80f0d931640bcaca997a47eefbd3ea29f10222b73cec2ec6561d8b91890780" pod="ix-prowlarr/prowlarr-5bcc99cc47-v86hm" Jul 24 09:00:07 bluth k3s[11030]: E0724 09:00:07.131748 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" podSandboxID="14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" pod="ix-esphome/esphome-658957fddc-6z2jm" Jul 24 09:00:07 bluth k3s[11030]: E0724 09:00:07.791421 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:00:08 bluth k3s[11030]: E0724 09:00:08.146144 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" podSandboxID="14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" pod="ix-esphome/esphome-658957fddc-6z2jm" Jul 24 09:00:08 bluth k3s[11030]: E0724 09:00:08.147059 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: df80f0d931640bcaca997a47eefbd3ea29f10222b73cec2ec6561d8b91890780" podSandboxID="df80f0d931640bcaca997a47eefbd3ea29f10222b73cec2ec6561d8b91890780" pod="ix-prowlarr/prowlarr-5bcc99cc47-v86hm" Jul 24 09:00:08 bluth k3s[11030]: E0724 09:00:08.147864 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" podSandboxID="14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" pod="ix-esphome/esphome-658957fddc-6z2jm" Jul 24 09:00:08 bluth k3s[11030]: E0724 09:00:08.148667 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: df80f0d931640bcaca997a47eefbd3ea29f10222b73cec2ec6561d8b91890780" podSandboxID="df80f0d931640bcaca997a47eefbd3ea29f10222b73cec2ec6561d8b91890780" pod="ix-prowlarr/prowlarr-5bcc99cc47-v86hm" Jul 24 09:00:09 bluth k3s[11030]: E0724 09:00:09.161648 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: df80f0d931640bcaca997a47eefbd3ea29f10222b73cec2ec6561d8b91890780" podSandboxID="df80f0d931640bcaca997a47eefbd3ea29f10222b73cec2ec6561d8b91890780" pod="ix-prowlarr/prowlarr-5bcc99cc47-v86hm" Jul 24 09:00:09 bluth k3s[11030]: E0724 09:00:09.162477 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" podSandboxID="14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" pod="ix-esphome/esphome-658957fddc-6z2jm" Jul 24 09:00:09 bluth k3s[11030]: E0724 09:00:09.163280 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" podSandboxID="14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" pod="ix-esphome/esphome-658957fddc-6z2jm" Jul 24 09:00:09 bluth k3s[11030]: E0724 09:00:09.164058 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: df80f0d931640bcaca997a47eefbd3ea29f10222b73cec2ec6561d8b91890780" podSandboxID="df80f0d931640bcaca997a47eefbd3ea29f10222b73cec2ec6561d8b91890780" pod="ix-prowlarr/prowlarr-5bcc99cc47-v86hm" Jul 24 09:00:09 bluth k3s[11030]: E0724 09:00:09.755691 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:00:10 bluth k3s[11030]: I0724 09:00:10.109686 11030 scope.go:110] "RemoveContainer" containerID="e0a015a83f4f225c152c833960e5c4fcee97f9db0f8aa22d1b75e978b477cf6d" Jul 24 09:00:10 bluth k3s[11030]: E0724 09:00:10.198847 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: df80f0d931640bcaca997a47eefbd3ea29f10222b73cec2ec6561d8b91890780" podSandboxID="df80f0d931640bcaca997a47eefbd3ea29f10222b73cec2ec6561d8b91890780" pod="ix-prowlarr/prowlarr-5bcc99cc47-v86hm" Jul 24 09:00:10 bluth k3s[11030]: E0724 09:00:10.202165 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: a72851790b0e29c44549e9fb06bb300c5bdd4e0a13d04e4dd95cfb600b736d56" podSandboxID="a72851790b0e29c44549e9fb06bb300c5bdd4e0a13d04e4dd95cfb600b736d56" pod="ix-radarr/radarr-7486f78d7f-ntt44" Jul 24 09:00:10 bluth k3s[11030]: E0724 09:00:10.348511 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" podSandboxID="14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" pod="ix-esphome/esphome-658957fddc-6z2jm" Jul 24 09:00:10 bluth k3s[11030]: E0724 09:00:10.350256 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 146634363c46a581c47b394440ec1456ddf446e18b5c19e09c5108d2dc1cc242" containerID="146634363c46a581c47b394440ec1456ddf446e18b5c19e09c5108d2dc1cc242" Jul 24 09:00:10 bluth k3s[11030]: E0724 09:00:10.350275 11030 kuberuntime_manager.go:1079] "getPodContainerStatuses for pod failed" err="rpc error: code = Unknown desc = Error: No such container: 146634363c46a581c47b394440ec1456ddf446e18b5c19e09c5108d2dc1cc242" pod="ix-plex/plex-bd956645c-wxsbd" Jul 24 09:00:10 bluth k3s[11030]: E0724 09:00:10.351080 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: df80f0d931640bcaca997a47eefbd3ea29f10222b73cec2ec6561d8b91890780" podSandboxID="df80f0d931640bcaca997a47eefbd3ea29f10222b73cec2ec6561d8b91890780" pod="ix-prowlarr/prowlarr-5bcc99cc47-v86hm" Jul 24 09:00:10 bluth k3s[11030]: E0724 09:00:10.351883 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" podSandboxID="14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" pod="ix-esphome/esphome-658957fddc-6z2jm" Jul 24 09:00:11 bluth k3s[11030]: E0724 09:00:11.367624 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: df80f0d931640bcaca997a47eefbd3ea29f10222b73cec2ec6561d8b91890780" podSandboxID="df80f0d931640bcaca997a47eefbd3ea29f10222b73cec2ec6561d8b91890780" pod="ix-prowlarr/prowlarr-5bcc99cc47-v86hm" Jul 24 09:00:11 bluth k3s[11030]: E0724 09:00:11.368408 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: de3bfadf58b135d1a7e346197ecaf8a849fd799016c8fd257375485dabd08778" podSandboxID="de3bfadf58b135d1a7e346197ecaf8a849fd799016c8fd257375485dabd08778" pod="kube-system/openebs-zfs-controller-0" Jul 24 09:00:11 bluth k3s[11030]: E0724 09:00:11.369184 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: a72851790b0e29c44549e9fb06bb300c5bdd4e0a13d04e4dd95cfb600b736d56" podSandboxID="a72851790b0e29c44549e9fb06bb300c5bdd4e0a13d04e4dd95cfb600b736d56" pod="ix-radarr/radarr-7486f78d7f-ntt44" Jul 24 09:00:11 bluth k3s[11030]: E0724 09:00:11.376382 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" podSandboxID="14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" pod="ix-esphome/esphome-658957fddc-6z2jm" Jul 24 09:00:11 bluth k3s[11030]: E0724 09:00:11.377280 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: df80f0d931640bcaca997a47eefbd3ea29f10222b73cec2ec6561d8b91890780" podSandboxID="df80f0d931640bcaca997a47eefbd3ea29f10222b73cec2ec6561d8b91890780" pod="ix-prowlarr/prowlarr-5bcc99cc47-v86hm" Jul 24 09:00:11 bluth k3s[11030]: E0724 09:00:11.378127 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: a72851790b0e29c44549e9fb06bb300c5bdd4e0a13d04e4dd95cfb600b736d56" podSandboxID="a72851790b0e29c44549e9fb06bb300c5bdd4e0a13d04e4dd95cfb600b736d56" pod="ix-radarr/radarr-7486f78d7f-ntt44" Jul 24 09:00:11 bluth k3s[11030]: E0724 09:00:11.379057 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" podSandboxID="14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" pod="ix-esphome/esphome-658957fddc-6z2jm" Jul 24 09:00:11 bluth k3s[11030]: E0724 09:00:11.777081 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:00:12 bluth k3s[11030]: E0724 09:00:12.394768 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: df80f0d931640bcaca997a47eefbd3ea29f10222b73cec2ec6561d8b91890780" podSandboxID="df80f0d931640bcaca997a47eefbd3ea29f10222b73cec2ec6561d8b91890780" pod="ix-prowlarr/prowlarr-5bcc99cc47-v86hm" Jul 24 09:00:12 bluth k3s[11030]: E0724 09:00:12.395640 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: de3bfadf58b135d1a7e346197ecaf8a849fd799016c8fd257375485dabd08778" podSandboxID="de3bfadf58b135d1a7e346197ecaf8a849fd799016c8fd257375485dabd08778" pod="kube-system/openebs-zfs-controller-0" Jul 24 09:00:12 bluth k3s[11030]: E0724 09:00:12.396550 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: a72851790b0e29c44549e9fb06bb300c5bdd4e0a13d04e4dd95cfb600b736d56" podSandboxID="a72851790b0e29c44549e9fb06bb300c5bdd4e0a13d04e4dd95cfb600b736d56" pod="ix-radarr/radarr-7486f78d7f-ntt44" Jul 24 09:00:12 bluth k3s[11030]: E0724 09:00:12.397375 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" podSandboxID="14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" pod="ix-esphome/esphome-658957fddc-6z2jm" Jul 24 09:00:12 bluth k3s[11030]: E0724 09:00:12.401981 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: df80f0d931640bcaca997a47eefbd3ea29f10222b73cec2ec6561d8b91890780" podSandboxID="df80f0d931640bcaca997a47eefbd3ea29f10222b73cec2ec6561d8b91890780" pod="ix-prowlarr/prowlarr-5bcc99cc47-v86hm" Jul 24 09:00:12 bluth k3s[11030]: E0724 09:00:12.402942 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: de3bfadf58b135d1a7e346197ecaf8a849fd799016c8fd257375485dabd08778" podSandboxID="de3bfadf58b135d1a7e346197ecaf8a849fd799016c8fd257375485dabd08778" pod="kube-system/openebs-zfs-controller-0" Jul 24 09:00:12 bluth k3s[11030]: E0724 09:00:12.403879 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: a72851790b0e29c44549e9fb06bb300c5bdd4e0a13d04e4dd95cfb600b736d56" podSandboxID="a72851790b0e29c44549e9fb06bb300c5bdd4e0a13d04e4dd95cfb600b736d56" pod="ix-radarr/radarr-7486f78d7f-ntt44" Jul 24 09:00:12 bluth k3s[11030]: E0724 09:00:12.404714 11030 kuberuntime_manager.go:1065] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = Error: No such container: 14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" podSandboxID="14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" pod="ix-esphome/esphome-658957fddc-6z2jm" Jul 24 09:00:13 bluth k3s[11030]: E0724 09:00:13.762269 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:00:15 bluth k3s[11030]: E0724 09:00:15.778841 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:00:17 bluth k3s[11030]: E0724 09:00:17.757808 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:00:18 bluth k3s[11030]: I0724 09:00:18.381327 11030 scope.go:110] "RemoveContainer" containerID="0ffa7770fac2800c05da6625137d616f559d5c73c8816fc7d97b1f5b06727d14" Jul 24 09:00:18 bluth k3s[11030]: E0724 09:00:18.381741 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 0ffa7770fac2800c05da6625137d616f559d5c73c8816fc7d97b1f5b06727d14" containerID="0ffa7770fac2800c05da6625137d616f559d5c73c8816fc7d97b1f5b06727d14" Jul 24 09:00:18 bluth k3s[11030]: I0724 09:00:18.381767 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:0ffa7770fac2800c05da6625137d616f559d5c73c8816fc7d97b1f5b06727d14} err="failed to get container status \"0ffa7770fac2800c05da6625137d616f559d5c73c8816fc7d97b1f5b06727d14\": rpc error: code = Unknown desc = Error: No such container: 0ffa7770fac2800c05da6625137d616f559d5c73c8816fc7d97b1f5b06727d14" Jul 24 09:00:18 bluth k3s[11030]: I0724 09:00:18.381776 11030 scope.go:110] "RemoveContainer" containerID="06eefc12a0bf11c5a58f4cabc3c155642c0efb73cb044d8a293548d860b65c1d" Jul 24 09:00:18 bluth k3s[11030]: E0724 09:00:18.381999 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 06eefc12a0bf11c5a58f4cabc3c155642c0efb73cb044d8a293548d860b65c1d" containerID="06eefc12a0bf11c5a58f4cabc3c155642c0efb73cb044d8a293548d860b65c1d" Jul 24 09:00:18 bluth k3s[11030]: I0724 09:00:18.382015 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:06eefc12a0bf11c5a58f4cabc3c155642c0efb73cb044d8a293548d860b65c1d} err="failed to get container status \"06eefc12a0bf11c5a58f4cabc3c155642c0efb73cb044d8a293548d860b65c1d\": rpc error: code = Unknown desc = Error: No such container: 06eefc12a0bf11c5a58f4cabc3c155642c0efb73cb044d8a293548d860b65c1d" Jul 24 09:00:18 bluth k3s[11030]: I0724 09:00:18.382022 11030 scope.go:110] "RemoveContainer" containerID="9132e315e0d5bc7b9e04b040529b2d2a7d22802b8015cee1d336243b41077fbd" Jul 24 09:00:18 bluth k3s[11030]: E0724 09:00:18.382241 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 9132e315e0d5bc7b9e04b040529b2d2a7d22802b8015cee1d336243b41077fbd" containerID="9132e315e0d5bc7b9e04b040529b2d2a7d22802b8015cee1d336243b41077fbd" Jul 24 09:00:18 bluth k3s[11030]: I0724 09:00:18.382313 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:9132e315e0d5bc7b9e04b040529b2d2a7d22802b8015cee1d336243b41077fbd} err="failed to get container status \"9132e315e0d5bc7b9e04b040529b2d2a7d22802b8015cee1d336243b41077fbd\": rpc error: code = Unknown desc = Error: No such container: 9132e315e0d5bc7b9e04b040529b2d2a7d22802b8015cee1d336243b41077fbd" Jul 24 09:00:18 bluth k3s[11030]: I0724 09:00:18.382377 11030 scope.go:110] "RemoveContainer" containerID="72ac180a3a68d798f24195ff9f7f801b1dbb04a1566caadc0e7e6cf4c30013a2" Jul 24 09:00:18 bluth k3s[11030]: E0724 09:00:18.382647 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 72ac180a3a68d798f24195ff9f7f801b1dbb04a1566caadc0e7e6cf4c30013a2" containerID="72ac180a3a68d798f24195ff9f7f801b1dbb04a1566caadc0e7e6cf4c30013a2" Jul 24 09:00:18 bluth k3s[11030]: I0724 09:00:18.382668 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:72ac180a3a68d798f24195ff9f7f801b1dbb04a1566caadc0e7e6cf4c30013a2} err="failed to get container status \"72ac180a3a68d798f24195ff9f7f801b1dbb04a1566caadc0e7e6cf4c30013a2\": rpc error: code = Unknown desc = Error: No such container: 72ac180a3a68d798f24195ff9f7f801b1dbb04a1566caadc0e7e6cf4c30013a2" Jul 24 09:00:18 bluth k3s[11030]: I0724 09:00:18.382675 11030 scope.go:110] "RemoveContainer" containerID="d4aed5c62635e97bfd4e98215c786fc4419297c23985754ae4f57fc76cc6c21c" Jul 24 09:00:18 bluth k3s[11030]: E0724 09:00:18.382899 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: d4aed5c62635e97bfd4e98215c786fc4419297c23985754ae4f57fc76cc6c21c" containerID="d4aed5c62635e97bfd4e98215c786fc4419297c23985754ae4f57fc76cc6c21c" Jul 24 09:00:18 bluth k3s[11030]: I0724 09:00:18.382964 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:d4aed5c62635e97bfd4e98215c786fc4419297c23985754ae4f57fc76cc6c21c} err="failed to get container status \"d4aed5c62635e97bfd4e98215c786fc4419297c23985754ae4f57fc76cc6c21c\": rpc error: code = Unknown desc = Error: No such container: d4aed5c62635e97bfd4e98215c786fc4419297c23985754ae4f57fc76cc6c21c" Jul 24 09:00:18 bluth k3s[11030]: I0724 09:00:18.382975 11030 scope.go:110] "RemoveContainer" containerID="e0a015a83f4f225c152c833960e5c4fcee97f9db0f8aa22d1b75e978b477cf6d" Jul 24 09:00:18 bluth k3s[11030]: E0724 09:00:18.383219 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: e0a015a83f4f225c152c833960e5c4fcee97f9db0f8aa22d1b75e978b477cf6d" containerID="e0a015a83f4f225c152c833960e5c4fcee97f9db0f8aa22d1b75e978b477cf6d" Jul 24 09:00:18 bluth k3s[11030]: I0724 09:00:18.383234 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:e0a015a83f4f225c152c833960e5c4fcee97f9db0f8aa22d1b75e978b477cf6d} err="failed to get container status \"e0a015a83f4f225c152c833960e5c4fcee97f9db0f8aa22d1b75e978b477cf6d\": rpc error: code = Unknown desc = Error: No such container: e0a015a83f4f225c152c833960e5c4fcee97f9db0f8aa22d1b75e978b477cf6d" Jul 24 09:00:18 bluth k3s[11030]: I0724 09:00:18.383240 11030 scope.go:110] "RemoveContainer" containerID="9132e315e0d5bc7b9e04b040529b2d2a7d22802b8015cee1d336243b41077fbd" Jul 24 09:00:18 bluth k3s[11030]: I0724 09:00:18.383456 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:9132e315e0d5bc7b9e04b040529b2d2a7d22802b8015cee1d336243b41077fbd} err="failed to get container status \"9132e315e0d5bc7b9e04b040529b2d2a7d22802b8015cee1d336243b41077fbd\": rpc error: code = Unknown desc = Error: No such container: 9132e315e0d5bc7b9e04b040529b2d2a7d22802b8015cee1d336243b41077fbd" Jul 24 09:00:18 bluth k3s[11030]: I0724 09:00:18.383467 11030 scope.go:110] "RemoveContainer" containerID="72ac180a3a68d798f24195ff9f7f801b1dbb04a1566caadc0e7e6cf4c30013a2" Jul 24 09:00:18 bluth k3s[11030]: I0724 09:00:18.383678 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:72ac180a3a68d798f24195ff9f7f801b1dbb04a1566caadc0e7e6cf4c30013a2} err="failed to get container status \"72ac180a3a68d798f24195ff9f7f801b1dbb04a1566caadc0e7e6cf4c30013a2\": rpc error: code = Unknown desc = Error: No such container: 72ac180a3a68d798f24195ff9f7f801b1dbb04a1566caadc0e7e6cf4c30013a2" Jul 24 09:00:18 bluth k3s[11030]: I0724 09:00:18.383691 11030 scope.go:110] "RemoveContainer" containerID="d4aed5c62635e97bfd4e98215c786fc4419297c23985754ae4f57fc76cc6c21c" Jul 24 09:00:18 bluth k3s[11030]: I0724 09:00:18.383959 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:d4aed5c62635e97bfd4e98215c786fc4419297c23985754ae4f57fc76cc6c21c} err="failed to get container status \"d4aed5c62635e97bfd4e98215c786fc4419297c23985754ae4f57fc76cc6c21c\": rpc error: code = Unknown desc = Error: No such container: d4aed5c62635e97bfd4e98215c786fc4419297c23985754ae4f57fc76cc6c21c" Jul 24 09:00:18 bluth k3s[11030]: I0724 09:00:18.383970 11030 scope.go:110] "RemoveContainer" containerID="e0a015a83f4f225c152c833960e5c4fcee97f9db0f8aa22d1b75e978b477cf6d" Jul 24 09:00:18 bluth k3s[11030]: I0724 09:00:18.384259 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:e0a015a83f4f225c152c833960e5c4fcee97f9db0f8aa22d1b75e978b477cf6d} err="failed to get container status \"e0a015a83f4f225c152c833960e5c4fcee97f9db0f8aa22d1b75e978b477cf6d\": rpc error: code = Unknown desc = Error: No such container: e0a015a83f4f225c152c833960e5c4fcee97f9db0f8aa22d1b75e978b477cf6d" Jul 24 09:00:18 bluth k3s[11030]: I0724 09:00:18.384270 11030 scope.go:110] "RemoveContainer" containerID="72ac180a3a68d798f24195ff9f7f801b1dbb04a1566caadc0e7e6cf4c30013a2" Jul 24 09:00:18 bluth k3s[11030]: I0724 09:00:18.384469 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:72ac180a3a68d798f24195ff9f7f801b1dbb04a1566caadc0e7e6cf4c30013a2} err="failed to get container status \"72ac180a3a68d798f24195ff9f7f801b1dbb04a1566caadc0e7e6cf4c30013a2\": rpc error: code = Unknown desc = Error: No such container: 72ac180a3a68d798f24195ff9f7f801b1dbb04a1566caadc0e7e6cf4c30013a2" Jul 24 09:00:18 bluth k3s[11030]: I0724 09:00:18.384543 11030 scope.go:110] "RemoveContainer" containerID="d4aed5c62635e97bfd4e98215c786fc4419297c23985754ae4f57fc76cc6c21c" Jul 24 09:00:18 bluth k3s[11030]: I0724 09:00:18.384936 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:d4aed5c62635e97bfd4e98215c786fc4419297c23985754ae4f57fc76cc6c21c} err="failed to get container status \"d4aed5c62635e97bfd4e98215c786fc4419297c23985754ae4f57fc76cc6c21c\": rpc error: code = Unknown desc = Error: No such container: d4aed5c62635e97bfd4e98215c786fc4419297c23985754ae4f57fc76cc6c21c" Jul 24 09:00:18 bluth k3s[11030]: I0724 09:00:18.384947 11030 scope.go:110] "RemoveContainer" containerID="e0a015a83f4f225c152c833960e5c4fcee97f9db0f8aa22d1b75e978b477cf6d" Jul 24 09:00:18 bluth k3s[11030]: I0724 09:00:18.385151 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:e0a015a83f4f225c152c833960e5c4fcee97f9db0f8aa22d1b75e978b477cf6d} err="failed to get container status \"e0a015a83f4f225c152c833960e5c4fcee97f9db0f8aa22d1b75e978b477cf6d\": rpc error: code = Unknown desc = Error: No such container: e0a015a83f4f225c152c833960e5c4fcee97f9db0f8aa22d1b75e978b477cf6d" Jul 24 09:00:18 bluth k3s[11030]: I0724 09:00:18.385161 11030 scope.go:110] "RemoveContainer" containerID="d4aed5c62635e97bfd4e98215c786fc4419297c23985754ae4f57fc76cc6c21c" Jul 24 09:00:18 bluth k3s[11030]: I0724 09:00:18.385349 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:d4aed5c62635e97bfd4e98215c786fc4419297c23985754ae4f57fc76cc6c21c} err="failed to get container status \"d4aed5c62635e97bfd4e98215c786fc4419297c23985754ae4f57fc76cc6c21c\": rpc error: code = Unknown desc = Error: No such container: d4aed5c62635e97bfd4e98215c786fc4419297c23985754ae4f57fc76cc6c21c" Jul 24 09:00:18 bluth k3s[11030]: I0724 09:00:18.385358 11030 scope.go:110] "RemoveContainer" containerID="e0a015a83f4f225c152c833960e5c4fcee97f9db0f8aa22d1b75e978b477cf6d" Jul 24 09:00:18 bluth k3s[11030]: I0724 09:00:18.385560 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:e0a015a83f4f225c152c833960e5c4fcee97f9db0f8aa22d1b75e978b477cf6d} err="failed to get container status \"e0a015a83f4f225c152c833960e5c4fcee97f9db0f8aa22d1b75e978b477cf6d\": rpc error: code = Unknown desc = Error: No such container: e0a015a83f4f225c152c833960e5c4fcee97f9db0f8aa22d1b75e978b477cf6d" Jul 24 09:00:18 bluth k3s[11030]: I0724 09:00:18.385628 11030 scope.go:110] "RemoveContainer" containerID="e0a015a83f4f225c152c833960e5c4fcee97f9db0f8aa22d1b75e978b477cf6d" Jul 24 09:00:18 bluth k3s[11030]: I0724 09:00:18.385862 11030 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:e0a015a83f4f225c152c833960e5c4fcee97f9db0f8aa22d1b75e978b477cf6d} err="failed to get container status \"e0a015a83f4f225c152c833960e5c4fcee97f9db0f8aa22d1b75e978b477cf6d\": rpc error: code = Unknown desc = Error: No such container: e0a015a83f4f225c152c833960e5c4fcee97f9db0f8aa22d1b75e978b477cf6d" Jul 24 09:00:19 bluth k3s[11030]: E0724 09:00:19.759843 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:00:21 bluth k3s[11030]: E0724 09:00:21.792167 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:00:23 bluth k3s[11030]: E0724 09:00:23.755895 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:00:25 bluth k3s[11030]: I0724 09:00:25.569005 48663 event.go:278] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ix-esphome", Name:"esphome-658957fddc-6z2jm", UID:"19254d20-7ad1-49a1-b408-f6e4213e5b4e", APIVersion:"v1", ResourceVersion:"3027829", FieldPath:""}): type: 'Normal' reason: 'AddedInterface' Add eth0 [172.16.0.208/16] from ix-net Jul 24 09:00:25 bluth k3s[11030]: E0724 09:00:25.764756 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:00:25 bluth k3s[11030]: I0724 09:00:25.957855 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="df80f0d931640bcaca997a47eefbd3ea29f10222b73cec2ec6561d8b91890780" Jul 24 09:00:26 bluth k3s[11030]: I0724 09:00:26.225220 48828 event.go:278] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ix-prowlarr", Name:"prowlarr-5bcc99cc47-v86hm", UID:"5d1f2955-de8a-4e25-b1f8-e607463ed369", APIVersion:"v1", ResourceVersion:"3027837", FieldPath:""}): type: 'Normal' reason: 'AddedInterface' Add eth0 [172.16.0.209/16] from ix-net Jul 24 09:00:26 bluth k3s[11030]: I0724 09:00:26.708994 48906 event.go:278] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ix-radarr", Name:"radarr-7486f78d7f-ntt44", UID:"9ab9d8fe-e669-40bb-a0ce-7f7a4e5a69b5", APIVersion:"v1", ResourceVersion:"3027823", FieldPath:""}): type: 'Normal' reason: 'AddedInterface' Add eth0 [172.16.0.210/16] from ix-net Jul 24 09:00:27 bluth k3s[11030]: E0724 09:00:27.751843 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:00:29 bluth k3s[11030]: E0724 09:00:29.760127 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:00:31 bluth k3s[11030]: E0724 09:00:31.762274 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:00:32 bluth k3s[11030]: I0724 09:00:32.857909 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="de3bfadf58b135d1a7e346197ecaf8a849fd799016c8fd257375485dabd08778" Jul 24 09:00:32 bluth k3s[11030]: I0724 09:00:32.869485 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="a72851790b0e29c44549e9fb06bb300c5bdd4e0a13d04e4dd95cfb600b736d56" Jul 24 09:00:32 bluth k3s[11030]: I0724 09:00:32.880292 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="14cc60f0f49f5d50d2b476aeae6e67e1960cbfa08f63790b74fbd2db628dce46" Jul 24 09:00:32 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T09:00:32.911+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 09:00:32 bluth k3s[11030]: I0724 09:00:32.988970 49325 event.go:278] Event(v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"openebs-zfs-controller-0", UID:"18ad2548-e271-464c-b16a-22db78a782a4", APIVersion:"v1", ResourceVersion:"3028212", FieldPath:""}): type: 'Normal' reason: 'AddedInterface' Add eth0 [172.16.0.211/16] from ix-net Jul 24 09:00:33 bluth k3s[11030]: I0724 09:00:33.187522 49446 event.go:278] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ix-sonarr", Name:"sonarr-867d7dd548-fw6z4", UID:"1a1846b0-3361-4f4b-a1cd-d5f958d6a7ac", APIVersion:"v1", ResourceVersion:"3027845", FieldPath:""}): type: 'Normal' reason: 'AddedInterface' Add eth0 [172.16.0.212/16] from ix-net Jul 24 09:00:33 bluth k3s[11030]: E0724 09:00:33.766979 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:00:34 bluth k3s[11030]: time="2022-07-24T09:00:34+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-tdarr\", Name:\"tdarr-comm\", UID:\"c3b33312-b11b-4e16-915e-205202cc0a2d\", APIVersion:\"v1\", ResourceVersion:\"2504455\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-tdarr/svclb-tdarr-comm" Jul 24 09:00:34 bluth k3s[11030]: time="2022-07-24T09:00:34+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-tdarr\", Name:\"tdarr\", UID:\"90709eb9-3d84-4b91-b6ab-73ece8ffcb86\", APIVersion:\"v1\", ResourceVersion:\"3028106\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-tdarr/svclb-tdarr" Jul 24 09:00:35 bluth k3s[11030]: E0724 09:00:35.786189 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:00:36 bluth k3s[11030]: E0724 09:00:36.343304 11030 remote_runtime.go:673] "ExecSync cmd from runtime service failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" containerID="146634363c46a581c47b394440ec1456ddf446e18b5c19e09c5108d2dc1cc242" cmd=[/bin/plex_probe_check.sh] Jul 24 09:00:37 bluth k3s[11030]: E0724 09:00:37.760322 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:00:38 bluth k3s[11030]: I0724 09:00:38.575608 11030 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="51983f1594a47f893b5f3641e29acdb7ae180936d7096a930c18ce548a8c30f6" Jul 24 09:00:38 bluth k3s[11030]: I0724 09:00:38.727776 49895 event.go:278] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ix-qbittorrent", Name:"qbittorrent-6d7cf9f7cd-xrqwj", UID:"4d43a190-0b41-4f84-a84d-ce295694fcb5", APIVersion:"v1", ResourceVersion:"3027842", FieldPath:""}): type: 'Normal' reason: 'AddedInterface' Add eth0 [172.16.0.213/16] from ix-net Jul 24 09:00:39 bluth k3s[11030]: E0724 09:00:39.797368 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:00:41 bluth k3s[11030]: E0724 09:00:41.756818 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:00:43 bluth k3s[11030]: E0724 09:00:43.802984 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:00:45 bluth k3s[11030]: E0724 09:00:45.763431 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:00:47 bluth k3s[11030]: E0724 09:00:47.761189 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:00:49 bluth k3s[11030]: E0724 09:00:49.756222 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:00:51 bluth k3s[11030]: E0724 09:00:51.343328 11030 remote_runtime.go:673] "ExecSync cmd from runtime service failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" containerID="146634363c46a581c47b394440ec1456ddf446e18b5c19e09c5108d2dc1cc242" cmd=[/bin/plex_probe_check.sh] Jul 24 09:00:51 bluth k3s[11030]: E0724 09:00:51.768262 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:00:53 bluth k3s[11030]: E0724 09:00:53.770891 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:00:55 bluth k3s[11030]: E0724 09:00:55.772637 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:00:57 bluth k3s[11030]: E0724 09:00:57.752719 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:00:59 bluth k3s[11030]: E0724 09:00:59.760570 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:01:01 bluth k3s[11030]: E0724 09:01:01.760974 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:01:03 bluth k3s[11030]: E0724 09:01:03.768742 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:01:05 bluth k3s[11030]: E0724 09:01:05.781941 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:01:06 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T09:01:06.955+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 09:01:07 bluth k3s[11030]: E0724 09:01:07.760458 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:01:09 bluth k3s[11030]: E0724 09:01:09.761311 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:01:11 bluth k3s[11030]: E0724 09:01:11.796369 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:01:13 bluth k3s[11030]: E0724 09:01:13.756766 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:01:15 bluth k3s[11030]: E0724 09:01:15.760494 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:01:17 bluth k3s[11030]: E0724 09:01:17.761216 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:01:19 bluth k3s[11030]: E0724 09:01:19.767400 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:01:21 bluth k3s[11030]: E0724 09:01:21.756929 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:01:23 bluth k3s[11030]: E0724 09:01:23.770802 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:01:24 bluth k3s[11030]: I0724 09:01:24.201407 11030 trace.go:205] Trace[1996440289]: "GuaranteedUpdate etcd3" type:*core.Event (24-Jul-2022 09:01:20.412) (total time: 3788ms): Jul 24 09:01:24 bluth k3s[11030]: Trace[1996440289]: ---"Transaction committed" 3787ms (09:01:24.201) Jul 24 09:01:24 bluth k3s[11030]: Trace[1996440289]: [3.788514491s] [3.788514491s] END Jul 24 09:01:24 bluth k3s[11030]: I0724 09:01:24.201677 11030 trace.go:205] Trace[974086909]: "Patch" url:/api/v1/namespaces/ix-plex/events/plex-bd956645c-wxsbd.1704b5153718b676,user-agent:k3s/v1.23.5+k3s (linux/amd64) kubernetes/fbfa51e,audit-id:1d735d07-1adf-4eb3-9c77-525fb4994e78,client:127.0.0.1,accept:application/vnd.kubernetes.protobuf,application/json,protocol:HTTP/2.0 (24-Jul-2022 09:01:20.412) (total time: 3788ms): Jul 24 09:01:24 bluth k3s[11030]: Trace[974086909]: ---"Object stored in database" 3787ms (09:01:24.201) Jul 24 09:01:24 bluth k3s[11030]: Trace[974086909]: [3.788835303s] [3.788835303s] END Jul 24 09:01:24 bluth k3s[11030]: I0724 09:01:24.216423 11030 trace.go:205] Trace[1928955189]: "GuaranteedUpdate etcd3" type:*v1.Endpoints (24-Jul-2022 09:01:22.181) (total time: 2035ms): Jul 24 09:01:24 bluth k3s[11030]: Trace[1928955189]: ---"Transaction committed" 2033ms (09:01:24.216) Jul 24 09:01:24 bluth k3s[11030]: Trace[1928955189]: [2.035162966s] [2.035162966s] END Jul 24 09:01:25 bluth k3s[11030]: E0724 09:01:25.773732 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:01:27 bluth k3s[11030]: E0724 09:01:27.756057 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:01:29 bluth k3s[11030]: E0724 09:01:29.751944 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:01:31 bluth k3s[11030]: E0724 09:01:31.761758 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:01:33 bluth k3s[11030]: E0724 09:01:33.767294 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:01:35 bluth k3s[11030]: E0724 09:01:35.756342 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:01:37 bluth k3s[11030]: E0724 09:01:37.792524 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:01:39 bluth k3s[11030]: E0724 09:01:39.798501 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:01:41 bluth k3s[11030]: E0724 09:01:41.755970 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:01:43 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T09:01:43.630+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 09:01:43 bluth k3s[11030]: E0724 09:01:43.779425 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:01:45 bluth k3s[11030]: E0724 09:01:45.756414 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:01:47 bluth k3s[11030]: E0724 09:01:47.781239 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:01:49 bluth k3s[11030]: E0724 09:01:49.760737 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:01:51 bluth k3s[11030]: E0724 09:01:51.796694 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:01:53 bluth k3s[11030]: E0724 09:01:53.797307 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:01:55 bluth k3s[11030]: E0724 09:01:55.756439 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:01:57 bluth k3s[11030]: E0724 09:01:57.760527 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:01:59 bluth k3s[11030]: E0724 09:01:59.761076 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:02:01 bluth k3s[11030]: E0724 09:02:01.756450 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:02:03 bluth k3s[11030]: E0724 09:02:03.775266 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:02:05 bluth k3s[11030]: E0724 09:02:05.781565 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:02:07 bluth k3s[11030]: E0724 09:02:07.755460 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:02:09 bluth k3s[11030]: E0724 09:02:09.759981 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:02:11 bluth k3s[11030]: E0724 09:02:11.760879 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:02:13 bluth k3s[11030]: E0724 09:02:13.756605 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:02:15 bluth k3s[11030]: E0724 09:02:15.760217 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:02:17 bluth k3s[11030]: E0724 09:02:17.760371 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:02:19 bluth k3s[11030]: E0724 09:02:19.756678 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:02:20 bluth k3s[11030]: E0724 09:02:20.153437 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 1d1cfc1eaaa603573fb572799b978359b7c7c0b8049a050258638b3af7f0bc08" containerID="1d1cfc1eaaa603573fb572799b978359b7c7c0b8049a050258638b3af7f0bc08" Jul 24 09:02:20 bluth k3s[11030]: E0724 09:02:20.153461 11030 kuberuntime_manager.go:1079] "getPodContainerStatuses for pod failed" err="rpc error: code = Unknown desc = Error: No such container: 1d1cfc1eaaa603573fb572799b978359b7c7c0b8049a050258638b3af7f0bc08" pod="ix-prowlarr/prowlarr-5bcc99cc47-v86hm" Jul 24 09:02:20 bluth k3s[11030]: E0724 09:02:20.156336 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 59fdd31f32ea3d5ce195f590cc34f05a801fc35393d2429374a75346e730b8c6" containerID="59fdd31f32ea3d5ce195f590cc34f05a801fc35393d2429374a75346e730b8c6" Jul 24 09:02:20 bluth k3s[11030]: E0724 09:02:20.156356 11030 kuberuntime_manager.go:1079] "getPodContainerStatuses for pod failed" err="rpc error: code = Unknown desc = Error: No such container: 59fdd31f32ea3d5ce195f590cc34f05a801fc35393d2429374a75346e730b8c6" pod="ix-radarr/radarr-7486f78d7f-ntt44" Jul 24 09:02:20 bluth k3s[11030]: E0724 09:02:20.159367 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: ca3ec8bca49a8b7f35b8ccad68f5c8ae7fdaab3e27436ccbc6fede3ec0d5e5bd" containerID="ca3ec8bca49a8b7f35b8ccad68f5c8ae7fdaab3e27436ccbc6fede3ec0d5e5bd" Jul 24 09:02:20 bluth k3s[11030]: E0724 09:02:20.159386 11030 kuberuntime_manager.go:1079] "getPodContainerStatuses for pod failed" err="rpc error: code = Unknown desc = Error: No such container: ca3ec8bca49a8b7f35b8ccad68f5c8ae7fdaab3e27436ccbc6fede3ec0d5e5bd" pod="ix-esphome/esphome-658957fddc-6z2jm" Jul 24 09:02:21 bluth k3s[11030]: E0724 09:02:21.757271 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:02:22 bluth k3s[11030]: E0724 09:02:22.265793 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 9d4c9532a5c97b3ac8d7c8a6f065eb24e0d2e371a7e85dfe758ac5450172c11b" containerID="9d4c9532a5c97b3ac8d7c8a6f065eb24e0d2e371a7e85dfe758ac5450172c11b" Jul 24 09:02:22 bluth k3s[11030]: E0724 09:02:22.265867 11030 kuberuntime_manager.go:1079] "getPodContainerStatuses for pod failed" err="rpc error: code = Unknown desc = Error: No such container: 9d4c9532a5c97b3ac8d7c8a6f065eb24e0d2e371a7e85dfe758ac5450172c11b" pod="ix-sonarr/sonarr-867d7dd548-fw6z4" Jul 24 09:02:22 bluth k3s[11030]: E0724 09:02:22.288045 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: fce5f0ffc1b4a3db6f50e99d7eaf95b5b8e5f05130359341aa8741b0fe93a816" containerID="fce5f0ffc1b4a3db6f50e99d7eaf95b5b8e5f05130359341aa8741b0fe93a816" Jul 24 09:02:22 bluth k3s[11030]: E0724 09:02:22.288114 11030 kuberuntime_manager.go:1079] "getPodContainerStatuses for pod failed" err="rpc error: code = Unknown desc = Error: No such container: fce5f0ffc1b4a3db6f50e99d7eaf95b5b8e5f05130359341aa8741b0fe93a816" pod="kube-system/openebs-zfs-controller-0" Jul 24 09:02:23 bluth k3s[11030]: E0724 09:02:23.800852 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:02:25 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T09:02:25.421+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 09:02:25 bluth k3s[11030]: E0724 09:02:25.751594 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:02:27 bluth k3s[11030]: E0724 09:02:27.760463 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:02:29 bluth k3s[11030]: E0724 09:02:29.761787 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:02:31 bluth k3s[11030]: E0724 09:02:31.797603 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:02:33 bluth k3s[11030]: E0724 09:02:33.758124 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:02:35 bluth k3s[11030]: time="2022-07-24T09:02:35+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-prowlarr\", Name:\"prowlarr\", UID:\"bd43faf9-73db-4bcd-a678-c673a7b54098\", APIVersion:\"v1\", ResourceVersion:\"3028038\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-prowlarr/svclb-prowlarr" Jul 24 09:02:35 bluth k3s[11030]: E0724 09:02:35.756915 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:02:36 bluth k3s[11030]: time="2022-07-24T09:02:36+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-radarr\", Name:\"radarr\", UID:\"2ceba47d-fa27-462f-9816-af4352781795\", APIVersion:\"v1\", ResourceVersion:\"3028067\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-radarr/svclb-radarr" Jul 24 09:02:37 bluth k3s[11030]: E0724 09:02:37.756144 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:02:38 bluth k3s[11030]: time="2022-07-24T09:02:38+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-esphome\", Name:\"esphome\", UID:\"9d7f07bb-0ed3-4a9b-b9b3-ce707e119669\", APIVersion:\"v1\", ResourceVersion:\"3028148\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-esphome/svclb-esphome" Jul 24 09:02:38 bluth k3s[11030]: time="2022-07-24T09:02:38+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-sonarr\", Name:\"sonarr\", UID:\"b24571a2-a018-4716-ab7d-d6ab31e66e44\", APIVersion:\"v1\", ResourceVersion:\"3028146\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-sonarr/svclb-sonarr" Jul 24 09:02:39 bluth k3s[11030]: time="2022-07-24T09:02:39+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-qbittorrent\", Name:\"qbittorrent-torrentudp\", UID:\"1d92e74f-b912-46f1-b780-c740f80bb9ca\", APIVersion:\"v1\", ResourceVersion:\"3028051\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-qbittorrent/svclb-qbittorrent-torrentudp" Jul 24 09:02:39 bluth k3s[11030]: time="2022-07-24T09:02:39+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-qbittorrent\", Name:\"qbittorrent\", UID:\"728a4977-c72e-46a0-96f2-a9279b009171\", APIVersion:\"v1\", ResourceVersion:\"3027645\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-qbittorrent/svclb-qbittorrent" Jul 24 09:02:39 bluth k3s[11030]: time="2022-07-24T09:02:39+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-qbittorrent\", Name:\"qbittorrent-torrent\", UID:\"29b172f8-d799-4e0c-aac7-78ec5af5c01b\", APIVersion:\"v1\", ResourceVersion:\"3028124\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-qbittorrent/svclb-qbittorrent-torrent" Jul 24 09:02:39 bluth k3s[11030]: E0724 09:02:39.756528 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:02:41 bluth k3s[11030]: E0724 09:02:41.764638 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:02:43 bluth k3s[11030]: E0724 09:02:43.760334 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:02:45 bluth k3s[11030]: E0724 09:02:45.766278 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:02:47 bluth k3s[11030]: E0724 09:02:47.755997 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:02:49 bluth k3s[11030]: E0724 09:02:49.756606 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:02:51 bluth k3s[11030]: E0724 09:02:51.751762 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:02:53 bluth k3s[11030]: E0724 09:02:53.760557 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:02:54 bluth k3s[11030]: E0724 09:02:54.385931 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 7b7c0c7aa4d1613331c7b6c0ad975ab2084c1461e0a2c924e9abdad4658b4799" containerID="7b7c0c7aa4d1613331c7b6c0ad975ab2084c1461e0a2c924e9abdad4658b4799" Jul 24 09:02:54 bluth k3s[11030]: E0724 09:02:54.386008 11030 kuberuntime_manager.go:1079] "getPodContainerStatuses for pod failed" err="rpc error: code = Unknown desc = Error: No such container: 7b7c0c7aa4d1613331c7b6c0ad975ab2084c1461e0a2c924e9abdad4658b4799" pod="ix-tdarr/tdarr-746d65b466-vdnf9" Jul 24 09:02:55 bluth k3s[11030]: E0724 09:02:55.761511 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:02:57 bluth k3s[11030]: E0724 09:02:57.753671 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:02:59 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T09:02:59.387+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 09:02:59 bluth k3s[11030]: E0724 09:02:59.760842 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:03:01 bluth k3s[11030]: E0724 09:03:01.797367 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:03:03 bluth k3s[11030]: E0724 09:03:03.756200 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:03:05 bluth k3s[11030]: E0724 09:03:05.767608 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:03:07 bluth k3s[11030]: E0724 09:03:07.760932 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:03:09 bluth k3s[11030]: E0724 09:03:09.780236 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:03:11 bluth k3s[11030]: E0724 09:03:11.780260 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:03:12 bluth k3s[11030]: E0724 09:03:12.793034 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 2404936947febdbb96fb9185795e88a59bb3d682eddf6bf47dd7b1a5eb35cd70" containerID="2404936947febdbb96fb9185795e88a59bb3d682eddf6bf47dd7b1a5eb35cd70" Jul 24 09:03:12 bluth k3s[11030]: E0724 09:03:12.793058 11030 kuberuntime_manager.go:1079] "getPodContainerStatuses for pod failed" err="rpc error: code = Unknown desc = Error: No such container: 2404936947febdbb96fb9185795e88a59bb3d682eddf6bf47dd7b1a5eb35cd70" pod="kube-system/openebs-zfs-controller-0" Jul 24 09:03:13 bluth k3s[11030]: E0724 09:03:13.761038 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:03:15 bluth k3s[11030]: E0724 09:03:15.781283 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:03:16 bluth k3s[11030]: E0724 09:03:16.896009 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: a8aaa9dc24975bea37a357cec37a8693744c0a4afae37d728ec2d2e8af77e29f" containerID="a8aaa9dc24975bea37a357cec37a8693744c0a4afae37d728ec2d2e8af77e29f" Jul 24 09:03:16 bluth k3s[11030]: E0724 09:03:16.896101 11030 kuberuntime_manager.go:1079] "getPodContainerStatuses for pod failed" err="rpc error: code = Unknown desc = Error: No such container: a8aaa9dc24975bea37a357cec37a8693744c0a4afae37d728ec2d2e8af77e29f" pod="ix-radarr/radarr-7486f78d7f-ntt44" Jul 24 09:03:16 bluth k3s[11030]: E0724 09:03:16.912462 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 528576a483bbaeb5b2178bbf335b9ddfe255c74af6a4f1291e36032f843752a7" containerID="528576a483bbaeb5b2178bbf335b9ddfe255c74af6a4f1291e36032f843752a7" Jul 24 09:03:16 bluth k3s[11030]: E0724 09:03:16.912506 11030 kuberuntime_manager.go:1079] "getPodContainerStatuses for pod failed" err="rpc error: code = Unknown desc = Error: No such container: 528576a483bbaeb5b2178bbf335b9ddfe255c74af6a4f1291e36032f843752a7" pod="ix-esphome/esphome-658957fddc-6z2jm" Jul 24 09:03:16 bluth k3s[11030]: E0724 09:03:16.918392 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: cdc1573132c55ba2f89b90310ffb3dd22d0eaf85119d5d81bb5d2e3b2d19e5c7" containerID="cdc1573132c55ba2f89b90310ffb3dd22d0eaf85119d5d81bb5d2e3b2d19e5c7" Jul 24 09:03:16 bluth k3s[11030]: E0724 09:03:16.918427 11030 kuberuntime_manager.go:1079] "getPodContainerStatuses for pod failed" err="rpc error: code = Unknown desc = Error: No such container: cdc1573132c55ba2f89b90310ffb3dd22d0eaf85119d5d81bb5d2e3b2d19e5c7" pod="ix-sonarr/sonarr-867d7dd548-fw6z4" Jul 24 09:03:16 bluth k3s[11030]: E0724 09:03:16.923545 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: bf2243223246b976cf48f09b8b28732feaeda56f8c77abc67e3d53b19af661f9" containerID="bf2243223246b976cf48f09b8b28732feaeda56f8c77abc67e3d53b19af661f9" Jul 24 09:03:16 bluth k3s[11030]: E0724 09:03:16.923580 11030 kuberuntime_manager.go:1079] "getPodContainerStatuses for pod failed" err="rpc error: code = Unknown desc = Error: No such container: bf2243223246b976cf48f09b8b28732feaeda56f8c77abc67e3d53b19af661f9" pod="ix-prowlarr/prowlarr-5bcc99cc47-v86hm" Jul 24 09:03:17 bluth k3s[11030]: E0724 09:03:17.756412 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:03:19 bluth k3s[11030]: E0724 09:03:19.798445 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:03:21 bluth k3s[11030]: E0724 09:03:21.798970 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:03:23 bluth k3s[11030]: E0724 09:03:23.758369 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:03:25 bluth k3s[11030]: E0724 09:03:25.755804 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:03:26 bluth k3s[11030]: I0724 09:03:26.280185 11030 trace.go:205] Trace[1475300083]: "GuaranteedUpdate etcd3" type:*core.Event (24-Jul-2022 09:03:23.769) (total time: 2510ms): Jul 24 09:03:26 bluth k3s[11030]: Trace[1475300083]: ---"Transaction committed" 2509ms (09:03:26.280) Jul 24 09:03:26 bluth k3s[11030]: Trace[1475300083]: [2.510467907s] [2.510467907s] END Jul 24 09:03:26 bluth k3s[11030]: I0724 09:03:26.280277 11030 trace.go:205] Trace[457238949]: "Patch" url:/apis/events.k8s.io/v1/namespaces/ix-plex/events/plex-bd956645c-wxsbd.1704b4ae00fba88a,user-agent:k3s/v1.23.5+k3s (linux/amd64) kubernetes/fbfa51e/scheduler,audit-id:75ba8718-36f8-4bc3-a044-f59488f85a33,client:127.0.0.1,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (24-Jul-2022 09:03:23.769) (total time: 2510ms): Jul 24 09:03:26 bluth k3s[11030]: Trace[457238949]: ---"Object stored in database" 2509ms (09:03:26.280) Jul 24 09:03:26 bluth k3s[11030]: Trace[457238949]: [2.510604222s] [2.510604222s] END Jul 24 09:03:27 bluth k3s[11030]: E0724 09:03:27.761806 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:03:29 bluth k3s[11030]: E0724 09:03:29.770482 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:03:31 bluth k3s[11030]: E0724 09:03:31.783234 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:03:33 bluth k3s[11030]: E0724 09:03:33.770343 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:03:33 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T09:03:33.965+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 09:03:35 bluth k3s[11030]: E0724 09:03:35.782026 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:03:37 bluth k3s[11030]: E0724 09:03:37.761132 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:03:39 bluth k3s[11030]: E0724 09:03:39.768159 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:03:40 bluth k3s[11030]: I0724 09:03:40.840240 11030 cni.go:334] "CNI failed to retrieve network namespace path" err="cannot find network namespace for the terminated container \"6d2d1f41beed8c6cb0a510daf97a2b869656d3eb1f69580c3e40b7d603a6913d\"" Jul 24 09:03:40 bluth k3s[11030]: 2022-07-24T09:03:40+01:00 [error] Multus: failed to get the cached delegates file: open /var/lib/cni/multus/6d2d1f41beed8c6cb0a510daf97a2b869656d3eb1f69580c3e40b7d603a6913d: no such file or directory, cannot properly delete Jul 24 09:03:41 bluth k3s[11030]: E0724 09:03:41.792684 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:03:43 bluth k3s[11030]: E0724 09:03:43.757260 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:03:45 bluth k3s[11030]: E0724 09:03:45.762422 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:03:47 bluth k3s[11030]: E0724 09:03:47.756511 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:03:49 bluth k3s[11030]: E0724 09:03:49.768529 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:03:51 bluth k3s[11030]: E0724 09:03:51.758706 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:03:53 bluth k3s[11030]: E0724 09:03:53.761122 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:03:55 bluth k3s[11030]: E0724 09:03:55.640679 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: de6f254274c73facf5a14332b32854d4272c3b1bce8782074a3536e710fce85d" containerID="de6f254274c73facf5a14332b32854d4272c3b1bce8782074a3536e710fce85d" Jul 24 09:03:55 bluth k3s[11030]: E0724 09:03:55.640704 11030 kuberuntime_manager.go:1079] "getPodContainerStatuses for pod failed" err="rpc error: code = Unknown desc = Error: No such container: de6f254274c73facf5a14332b32854d4272c3b1bce8782074a3536e710fce85d" pod="kube-system/openebs-zfs-controller-0" Jul 24 09:03:55 bluth k3s[11030]: E0724 09:03:55.811182 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:03:57 bluth k3s[11030]: E0724 09:03:57.781551 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:03:59 bluth k3s[11030]: E0724 09:03:59.762272 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:04:01 bluth k3s[11030]: time="2022-07-24T09:04:01+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-qbittorrent\", Name:\"qbittorrent-torrent\", UID:\"29b172f8-d799-4e0c-aac7-78ec5af5c01b\", APIVersion:\"v1\", ResourceVersion:\"3028124\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-qbittorrent/svclb-qbittorrent-torrent" Jul 24 09:04:01 bluth k3s[11030]: time="2022-07-24T09:04:01+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-qbittorrent\", Name:\"qbittorrent\", UID:\"728a4977-c72e-46a0-96f2-a9279b009171\", APIVersion:\"v1\", ResourceVersion:\"3027645\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-qbittorrent/svclb-qbittorrent" Jul 24 09:04:01 bluth k3s[11030]: time="2022-07-24T09:04:01+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-qbittorrent\", Name:\"qbittorrent-torrentudp\", UID:\"1d92e74f-b912-46f1-b780-c740f80bb9ca\", APIVersion:\"v1\", ResourceVersion:\"3028051\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-qbittorrent/svclb-qbittorrent-torrentudp" Jul 24 09:04:01 bluth k3s[11030]: E0724 09:04:01.755484 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:04:03 bluth k3s[11030]: E0724 09:04:03.769509 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:04:05 bluth k3s[11030]: E0724 09:04:05.753925 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:04:07 bluth k3s[11030]: I0724 09:04:07.153948 11030 cni.go:334] "CNI failed to retrieve network namespace path" err="cannot find network namespace for the terminated container \"a6327cc849b9115cec3b175ecb675c69b212d724d426e29b9149fcfcbe004693\"" Jul 24 09:04:07 bluth k3s[11030]: 2022-07-24T09:04:07+01:00 [error] Multus: failed to get the cached delegates file: open /var/lib/cni/multus/a6327cc849b9115cec3b175ecb675c69b212d724d426e29b9149fcfcbe004693: no such file or directory, cannot properly delete Jul 24 09:04:07 bluth k3s[11030]: E0724 09:04:07.761462 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:04:09 bluth k3s[11030]: E0724 09:04:09.762632 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:04:11 bluth k3s[11030]: E0724 09:04:11.762391 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:04:12 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T09:04:12.097+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 09:04:13 bluth k3s[11030]: E0724 09:04:13.805118 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:04:15 bluth k3s[11030]: E0724 09:04:15.782264 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:04:16 bluth k3s[11030]: E0724 09:04:16.981970 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 37ac34e5a73c8484c2ef98320c5a3b2f8874c4c401a21131a1a517831df788d3" containerID="37ac34e5a73c8484c2ef98320c5a3b2f8874c4c401a21131a1a517831df788d3" Jul 24 09:04:16 bluth k3s[11030]: E0724 09:04:16.981994 11030 kuberuntime_manager.go:1079] "getPodContainerStatuses for pod failed" err="rpc error: code = Unknown desc = Error: No such container: 37ac34e5a73c8484c2ef98320c5a3b2f8874c4c401a21131a1a517831df788d3" pod="kube-system/openebs-zfs-controller-0" Jul 24 09:04:17 bluth k3s[11030]: E0724 09:04:17.778883 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:04:19 bluth k3s[11030]: E0724 09:04:19.769415 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:04:21 bluth k3s[11030]: E0724 09:04:21.803412 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:04:23 bluth k3s[11030]: E0724 09:04:23.756452 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:04:25 bluth k3s[11030]: E0724 09:04:25.762857 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:04:27 bluth k3s[11030]: E0724 09:04:27.757444 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:04:29 bluth k3s[11030]: E0724 09:04:29.757060 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:04:31 bluth k3s[11030]: E0724 09:04:31.778502 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:04:33 bluth k3s[11030]: I0724 09:04:33.014569 11030 trace.go:205] Trace[681009936]: "GuaranteedUpdate etcd3" type:*coordination.Lease (24-Jul-2022 09:04:30.506) (total time: 2508ms): Jul 24 09:04:33 bluth k3s[11030]: Trace[681009936]: ---"Transaction committed" 2507ms (09:04:33.014) Jul 24 09:04:33 bluth k3s[11030]: Trace[681009936]: [2.508156491s] [2.508156491s] END Jul 24 09:04:33 bluth k3s[11030]: I0724 09:04:33.014675 11030 trace.go:205] Trace[1424557601]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ix-truenas,user-agent:Go-http-client/2.0,audit-id:c769890f-9cec-4236-9276-1bc0341a0ab9,client:127.0.0.1,accept:application/vnd.kubernetes.protobuf,application/json,protocol:HTTP/2.0 (24-Jul-2022 09:04:30.506) (total time: 2508ms): Jul 24 09:04:33 bluth k3s[11030]: Trace[1424557601]: ---"Object stored in database" 2508ms (09:04:33.014) Jul 24 09:04:33 bluth k3s[11030]: Trace[1424557601]: [2.508344104s] [2.508344104s] END Jul 24 09:04:33 bluth k3s[11030]: E0724 09:04:33.757009 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:04:35 bluth k3s[11030]: E0724 09:04:35.810198 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:04:36 bluth k3s[11030]: time="2022-07-24T09:04:36+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-esphome\", Name:\"esphome\", UID:\"9d7f07bb-0ed3-4a9b-b9b3-ce707e119669\", APIVersion:\"v1\", ResourceVersion:\"3028148\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-esphome/svclb-esphome" Jul 24 09:04:37 bluth k3s[11030]: E0724 09:04:37.784148 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:04:39 bluth k3s[11030]: I0724 09:04:39.167241 11030 trace.go:205] Trace[1520448016]: "GuaranteedUpdate etcd3" type:*core.Event (24-Jul-2022 09:04:35.455) (total time: 3711ms): Jul 24 09:04:39 bluth k3s[11030]: Trace[1520448016]: ---"Transaction committed" 3707ms (09:04:39.167) Jul 24 09:04:39 bluth k3s[11030]: Trace[1520448016]: [3.711195167s] [3.711195167s] END Jul 24 09:04:39 bluth k3s[11030]: I0724 09:04:39.167540 11030 trace.go:205] Trace[1440183780]: "Patch" url:/api/v1/namespaces/ix-tdarr/events/tdarr-746d65b466-vdnf9.1704b534a1a4c5cc,user-agent:k3s/v1.23.5+k3s (linux/amd64) kubernetes/fbfa51e,audit-id:c39901a2-9c28-4a8e-804d-9d998940920d,client:127.0.0.1,accept:application/vnd.kubernetes.protobuf,application/json,protocol:HTTP/2.0 (24-Jul-2022 09:04:35.455) (total time: 3711ms): Jul 24 09:04:39 bluth k3s[11030]: Trace[1440183780]: ---"Object stored in database" 3708ms (09:04:39.167) Jul 24 09:04:39 bluth k3s[11030]: Trace[1440183780]: [3.711629538s] [3.711629538s] END Jul 24 09:04:39 bluth k3s[11030]: E0724 09:04:39.799097 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:04:41 bluth k3s[11030]: E0724 09:04:41.785175 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:04:43 bluth k3s[11030]: E0724 09:04:43.775416 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:04:45 bluth k3s[11030]: E0724 09:04:45.756422 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:04:47 bluth k3s[11030]: E0724 09:04:47.769877 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:04:47 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T09:04:47.810+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 09:04:49 bluth k3s[11030]: E0724 09:04:49.795204 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:04:51 bluth k3s[11030]: E0724 09:04:51.782794 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:04:53 bluth k3s[11030]: E0724 09:04:53.764690 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:04:55 bluth k3s[11030]: E0724 09:04:55.760210 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:04:57 bluth k3s[11030]: E0724 09:04:57.761522 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:04:59 bluth k3s[11030]: E0724 09:04:59.769767 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:04:59 bluth k3s[11030]: E0724 09:04:59.772870 11030 remote_runtime.go:572] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: df23cad9b99483c72a05a80f00a08c4c2bda5bd53526183981490927d785cf23" containerID="df23cad9b99483c72a05a80f00a08c4c2bda5bd53526183981490927d785cf23" Jul 24 09:04:59 bluth k3s[11030]: E0724 09:04:59.772893 11030 kuberuntime_manager.go:1079] "getPodContainerStatuses for pod failed" err="rpc error: code = Unknown desc = Error: No such container: df23cad9b99483c72a05a80f00a08c4c2bda5bd53526183981490927d785cf23" pod="kube-system/openebs-zfs-controller-0" Jul 24 09:05:01 bluth k3s[11030]: E0724 09:05:01.757986 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:05:03 bluth k3s[11030]: E0724 09:05:03.763013 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:05:05 bluth k3s[11030]: E0724 09:05:05.802452 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:05:07 bluth k3s[11030]: E0724 09:05:07.802023 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:05:09 bluth k3s[11030]: E0724 09:05:09.782816 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:05:11 bluth k3s[11030]: E0724 09:05:11.763248 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:05:13 bluth k3s[11030]: E0724 09:05:13.764431 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:05:15 bluth k3s[11030]: E0724 09:05:15.762132 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:05:16 bluth k3s[11030]: time="2022-07-24T09:05:16+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-sonarr\", Name:\"sonarr\", UID:\"b24571a2-a018-4716-ab7d-d6ab31e66e44\", APIVersion:\"v1\", ResourceVersion:\"3028146\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-sonarr/svclb-sonarr" Jul 24 09:05:17 bluth k3s[11030]: E0724 09:05:17.756583 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:05:19 bluth k3s[11030]: E0724 09:05:19.757554 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:05:21 bluth k3s[11030]: E0724 09:05:21.801242 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:05:23 bluth k3s[11030]: E0724 09:05:23.757817 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:05:25 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T09:05:25.019+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 09:05:25 bluth k3s[11030]: E0724 09:05:25.757814 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:05:26 bluth k3s[11030]: time="2022-07-24T09:05:26+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-radarr\", Name:\"radarr\", UID:\"2ceba47d-fa27-462f-9816-af4352781795\", APIVersion:\"v1\", ResourceVersion:\"3028067\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-radarr/svclb-radarr" Jul 24 09:05:27 bluth k3s[11030]: E0724 09:05:27.769298 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:05:29 bluth k3s[11030]: E0724 09:05:29.769127 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:05:31 bluth k3s[11030]: E0724 09:05:31.790749 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:05:33 bluth k3s[11030]: E0724 09:05:33.783216 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:05:35 bluth k3s[11030]: E0724 09:05:35.804462 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:05:36 bluth k3s[11030]: time="2022-07-24T09:05:36+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-prowlarr\", Name:\"prowlarr\", UID:\"bd43faf9-73db-4bcd-a678-c673a7b54098\", APIVersion:\"v1\", ResourceVersion:\"3028038\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-prowlarr/svclb-prowlarr" Jul 24 09:05:37 bluth k3s[11030]: E0724 09:05:37.763995 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:05:39 bluth k3s[11030]: E0724 09:05:39.764244 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:05:41 bluth k3s[11030]: E0724 09:05:41.770481 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:05:43 bluth k3s[11030]: E0724 09:05:43.762068 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:05:45 bluth k3s[11030]: E0724 09:05:45.772812 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:05:47 bluth k3s[11030]: E0724 09:05:47.753068 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:05:49 bluth k3s[11030]: E0724 09:05:49.762039 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:05:51 bluth k3s[11030]: E0724 09:05:51.761952 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:05:53 bluth k3s[11030]: E0724 09:05:53.801469 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:05:55 bluth k3s[11030]: E0724 09:05:55.784473 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:05:57 bluth k3s[11030]: E0724 09:05:57.782317 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:05:59 bluth k3s[11030]: E0724 09:05:59.762274 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:06:00 bluth k3s[11030]: I0724 09:06:00.191883 11030 trace.go:205] Trace[2137245462]: "GuaranteedUpdate etcd3" type:*coordination.Lease (24-Jul-2022 09:05:56.568) (total time: 3623ms): Jul 24 09:06:00 bluth k3s[11030]: Trace[2137245462]: ---"Transaction committed" 3623ms (09:06:00.191) Jul 24 09:06:00 bluth k3s[11030]: Trace[2137245462]: [3.623573867s] [3.623573867s] END Jul 24 09:06:00 bluth k3s[11030]: I0724 09:06:00.192219 11030 trace.go:205] Trace[356218372]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/snapshot-controller-leader,user-agent:snapshot-controller/v0.0.0 (linux/amd64) kubernetes/$Format,audit-id:ba0814ba-f550-4477-a845-d3d367330f22,client:172.16.0.211,accept:application/json, */*,protocol:HTTP/2.0 (24-Jul-2022 09:05:56.568) (total time: 3624ms): Jul 24 09:06:00 bluth k3s[11030]: Trace[356218372]: ---"Object stored in database" 3623ms (09:06:00.191) Jul 24 09:06:00 bluth k3s[11030]: Trace[356218372]: [3.624111284s] [3.624111284s] END Jul 24 09:06:00 bluth k3s[11030]: I0724 09:06:00.213068 11030 trace.go:205] Trace[838456600]: "GuaranteedUpdate etcd3" type:*coordination.Lease (24-Jul-2022 09:05:57.077) (total time: 3135ms): Jul 24 09:06:00 bluth k3s[11030]: Trace[838456600]: ---"Transaction committed" 3134ms (09:06:00.212) Jul 24 09:06:00 bluth k3s[11030]: Trace[838456600]: [3.135275413s] [3.135275413s] END Jul 24 09:06:00 bluth k3s[11030]: I0724 09:06:00.213282 11030 trace.go:205] Trace[1078458573]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/external-resizer-zfs-csi-openebs-io,user-agent:csi-resizer/v0.0.0 (linux/amd64) kubernetes/$Format,audit-id:23bd5842-7c9a-4dd5-ace8-15ad0d33e12c,client:172.16.0.211,accept:application/json, */*,protocol:HTTP/2.0 (24-Jul-2022 09:05:57.077) (total time: 3135ms): Jul 24 09:06:00 bluth k3s[11030]: Trace[1078458573]: ---"Object stored in database" 3135ms (09:06:00.213) Jul 24 09:06:00 bluth k3s[11030]: Trace[1078458573]: [3.135649004s] [3.135649004s] END Jul 24 09:06:00 bluth k3s[11030]: I0724 09:06:00.253314 11030 trace.go:205] Trace[2073176193]: "GuaranteedUpdate etcd3" type:*coordination.Lease (24-Jul-2022 09:05:57.719) (total time: 2533ms): Jul 24 09:06:00 bluth k3s[11030]: Trace[2073176193]: ---"Transaction committed" 2533ms (09:06:00.253) Jul 24 09:06:00 bluth k3s[11030]: Trace[2073176193]: [2.533536475s] [2.533536475s] END Jul 24 09:06:00 bluth k3s[11030]: I0724 09:06:00.253553 11030 trace.go:205] Trace[1263337439]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/external-snapshotter-leader-zfs-csi-openebs-io,user-agent:csi-snapshotter/v0.0.0 (linux/amd64) kubernetes/$Format,audit-id:e2000a2e-6326-4ead-b716-72f5d44ae372,client:172.16.0.211,accept:application/json, */*,protocol:HTTP/2.0 (24-Jul-2022 09:05:57.719) (total time: 2533ms): Jul 24 09:06:00 bluth k3s[11030]: Trace[1263337439]: ---"Object stored in database" 2533ms (09:06:00.253) Jul 24 09:06:00 bluth k3s[11030]: Trace[1263337439]: [2.533924206s] [2.533924206s] END Jul 24 09:06:00 bluth k3s[11030]: I0724 09:06:00.279640 11030 trace.go:205] Trace[2050140351]: "GuaranteedUpdate etcd3" type:*coordination.Lease (24-Jul-2022 09:05:58.747) (total time: 1532ms): Jul 24 09:06:00 bluth k3s[11030]: Trace[2050140351]: ---"Transaction committed" 1532ms (09:06:00.279) Jul 24 09:06:00 bluth k3s[11030]: Trace[2050140351]: [1.532458081s] [1.532458081s] END Jul 24 09:06:00 bluth k3s[11030]: I0724 09:06:00.279725 11030 trace.go:205] Trace[1908546002]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/zfs-csi-openebs-io,user-agent:csi-provisioner/v0.0.0 (linux/amd64) kubernetes/$Format,audit-id:1084f173-b4a8-4e42-a23c-94a71fcf425a,client:172.16.0.211,accept:application/json, */*,protocol:HTTP/2.0 (24-Jul-2022 09:05:58.747) (total time: 1532ms): Jul 24 09:06:00 bluth k3s[11030]: Trace[1908546002]: ---"Object stored in database" 1532ms (09:06:00.279) Jul 24 09:06:00 bluth k3s[11030]: Trace[1908546002]: [1.532663757s] [1.532663757s] END Jul 24 09:06:01 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T09:06:01.435+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 09:06:01 bluth k3s[11030]: E0724 09:06:01.786087 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:06:03 bluth k3s[11030]: E0724 09:06:03.778341 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:06:05 bluth k3s[11030]: E0724 09:06:05.776180 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:06:07 bluth k3s[11030]: E0724 09:06:07.772828 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:06:09 bluth k3s[11030]: E0724 09:06:09.752884 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:06:11 bluth k3s[11030]: E0724 09:06:11.762257 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:06:13 bluth k3s[11030]: E0724 09:06:13.803599 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:06:15 bluth k3s[11030]: E0724 09:06:15.757024 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:06:17 bluth k3s[11030]: E0724 09:06:17.761751 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:06:19 bluth k3s[11030]: E0724 09:06:19.769564 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:06:21 bluth k3s[11030]: E0724 09:06:21.770315 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:06:23 bluth k3s[11030]: E0724 09:06:23.756863 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:06:24 bluth k3s[11030]: I0724 09:06:24.462187 11030 trace.go:205] Trace[1650805406]: "GuaranteedUpdate etcd3" type:*coordination.Lease (24-Jul-2022 09:06:20.337) (total time: 4125ms): Jul 24 09:06:24 bluth k3s[11030]: Trace[1650805406]: ---"Transaction committed" 4124ms (09:06:24.462) Jul 24 09:06:24 bluth k3s[11030]: Trace[1650805406]: [4.125008929s] [4.125008929s] END Jul 24 09:06:24 bluth k3s[11030]: I0724 09:06:24.462477 11030 trace.go:205] Trace[170900938]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/snapshot-controller-leader,user-agent:snapshot-controller/v0.0.0 (linux/amd64) kubernetes/$Format,audit-id:cfbf5faf-8bfa-45cc-8338-a91a5976b582,client:172.16.0.211,accept:application/json, */*,protocol:HTTP/2.0 (24-Jul-2022 09:06:20.336) (total time: 4125ms): Jul 24 09:06:24 bluth k3s[11030]: Trace[170900938]: ---"Object stored in database" 4125ms (09:06:24.462) Jul 24 09:06:24 bluth k3s[11030]: Trace[170900938]: [4.125870819s] [4.125870819s] END Jul 24 09:06:24 bluth k3s[11030]: I0724 09:06:24.476775 11030 trace.go:205] Trace[1156322894]: "GuaranteedUpdate etcd3" type:*coordination.Lease (24-Jul-2022 09:06:20.335) (total time: 4141ms): Jul 24 09:06:24 bluth k3s[11030]: Trace[1156322894]: ---"Transaction committed" 4140ms (09:06:24.476) Jul 24 09:06:24 bluth k3s[11030]: Trace[1156322894]: [4.141103902s] [4.141103902s] END Jul 24 09:06:24 bluth k3s[11030]: I0724 09:06:24.476993 11030 trace.go:205] Trace[170051626]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/zfs-csi-openebs-io,user-agent:csi-provisioner/v0.0.0 (linux/amd64) kubernetes/$Format,audit-id:98c9607e-0096-4d55-a9ce-d2a47de0d64f,client:172.16.0.211,accept:application/json, */*,protocol:HTTP/2.0 (24-Jul-2022 09:06:20.335) (total time: 4141ms): Jul 24 09:06:24 bluth k3s[11030]: Trace[170051626]: ---"Object stored in database" 4141ms (09:06:24.476) Jul 24 09:06:24 bluth k3s[11030]: Trace[170051626]: [4.141674637s] [4.141674637s] END Jul 24 09:06:24 bluth k3s[11030]: I0724 09:06:24.481188 11030 trace.go:205] Trace[654758561]: "GuaranteedUpdate etcd3" type:*coordination.Lease (24-Jul-2022 09:06:20.340) (total time: 4141ms): Jul 24 09:06:24 bluth k3s[11030]: Trace[654758561]: ---"Transaction committed" 4140ms (09:06:24.481) Jul 24 09:06:24 bluth k3s[11030]: Trace[654758561]: [4.141071226s] [4.141071226s] END Jul 24 09:06:24 bluth k3s[11030]: I0724 09:06:24.481356 11030 trace.go:205] Trace[872228870]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/external-snapshotter-leader-zfs-csi-openebs-io,user-agent:csi-snapshotter/v0.0.0 (linux/amd64) kubernetes/$Format,audit-id:f15ff4c8-b97d-4821-8b14-cc9b0b7214a8,client:172.16.0.211,accept:application/json, */*,protocol:HTTP/2.0 (24-Jul-2022 09:06:20.339) (total time: 4141ms): Jul 24 09:06:24 bluth k3s[11030]: Trace[872228870]: ---"Object stored in database" 4141ms (09:06:24.481) Jul 24 09:06:24 bluth k3s[11030]: Trace[872228870]: [4.141631056s] [4.141631056s] END Jul 24 09:06:24 bluth k3s[11030]: I0724 09:06:24.539239 11030 trace.go:205] Trace[709711262]: "GuaranteedUpdate etcd3" type:*v1.Endpoints (24-Jul-2022 09:06:22.200) (total time: 2338ms): Jul 24 09:06:24 bluth k3s[11030]: Trace[709711262]: ---"Transaction committed" 2335ms (09:06:24.539) Jul 24 09:06:24 bluth k3s[11030]: Trace[709711262]: [2.338307112s] [2.338307112s] END Jul 24 09:06:24 bluth k3s[11030]: I0724 09:06:24.577402 11030 trace.go:205] Trace[183080125]: "GuaranteedUpdate etcd3" type:*coordination.Lease (24-Jul-2022 09:06:20.335) (total time: 4241ms): Jul 24 09:06:24 bluth k3s[11030]: Trace[183080125]: ---"Transaction committed" 4240ms (09:06:24.577) Jul 24 09:06:24 bluth k3s[11030]: Trace[183080125]: [4.241914633s] [4.241914633s] END Jul 24 09:06:24 bluth k3s[11030]: I0724 09:06:24.577622 11030 trace.go:205] Trace[1308931036]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/external-resizer-zfs-csi-openebs-io,user-agent:csi-resizer/v0.0.0 (linux/amd64) kubernetes/$Format,audit-id:0331da6a-c87b-4d9e-b8c7-269a52bb485c,client:172.16.0.211,accept:application/json, */*,protocol:HTTP/2.0 (24-Jul-2022 09:06:20.335) (total time: 4242ms): Jul 24 09:06:24 bluth k3s[11030]: Trace[1308931036]: ---"Object stored in database" 4242ms (09:06:24.577) Jul 24 09:06:24 bluth k3s[11030]: Trace[1308931036]: [4.24252975s] [4.24252975s] END Jul 24 09:06:25 bluth k3s[11030]: E0724 09:06:25.757683 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:06:27 bluth k3s[11030]: E0724 09:06:27.764111 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:06:29 bluth k3s[11030]: E0724 09:06:29.801461 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:06:31 bluth k3s[11030]: E0724 09:06:31.802507 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:06:33 bluth k3s[11030]: E0724 09:06:33.770075 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:06:35 bluth k3s[11030]: time="2022-07-24T09:06:35+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-tdarr\", Name:\"tdarr-comm\", UID:\"c3b33312-b11b-4e16-915e-205202cc0a2d\", APIVersion:\"v1\", ResourceVersion:\"2504455\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-tdarr/svclb-tdarr-comm" Jul 24 09:06:35 bluth k3s[11030]: time="2022-07-24T09:06:35+01:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"ix-tdarr\", Name:\"tdarr\", UID:\"90709eb9-3d84-4b91-b6ab-73ece8ffcb86\", APIVersion:\"v1\", ResourceVersion:\"3028106\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedDaemonSet' Applied LoadBalancer DaemonSet ix-tdarr/svclb-tdarr" Jul 24 09:06:35 bluth k3s[11030]: E0724 09:06:35.783191 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:06:37 bluth k3s[11030]: E0724 09:06:37.769190 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:06:39 bluth k3s[11030]: E0724 09:06:39.770857 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:06:41 bluth k3s[11030]: E0724 09:06:41.752399 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:06:43 bluth k3s[11030]: E0724 09:06:43.762692 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:06:44 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T09:06:44.578+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 09:06:45 bluth k3s[11030]: E0724 09:06:45.756127 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:06:47 bluth k3s[11030]: E0724 09:06:47.806973 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:06:49 bluth k3s[11030]: E0724 09:06:49.759479 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:06:51 bluth k3s[11030]: E0724 09:06:51.762655 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:06:53 bluth k3s[11030]: E0724 09:06:53.763963 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:06:55 bluth k3s[11030]: E0724 09:06:55.753046 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:06:57 bluth k3s[11030]: E0724 09:06:57.761581 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:06:59 bluth k3s[11030]: E0724 09:06:59.803661 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:07:01 bluth k3s[11030]: E0724 09:07:01.752788 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:07:03 bluth k3s[11030]: E0724 09:07:03.770682 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:07:05 bluth k3s[11030]: E0724 09:07:05.757421 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:07:07 bluth k3s[11030]: E0724 09:07:07.781266 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:07:09 bluth k3s[11030]: E0724 09:07:09.765190 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:07:11 bluth k3s[11030]: E0724 09:07:11.769824 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:07:13 bluth k3s[11030]: E0724 09:07:13.752735 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:07:15 bluth k3s[11030]: E0724 09:07:15.762775 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:07:17 bluth k3s[11030]: E0724 09:07:17.785680 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:07:19 bluth k3s[11030]: E0724 09:07:19.757597 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:07:21 bluth k3s[11030]: E0724 09:07:21.764786 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:07:23 bluth k3s[11030]: E0724 09:07:23.801759 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:07:25 bluth k3s[11030]: E0724 09:07:25.752464 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:07:26 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T09:07:26.088+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 09:07:27 bluth k3s[11030]: E0724 09:07:27.758213 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:07:29 bluth k3s[11030]: E0724 09:07:29.802177 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:07:31 bluth k3s[11030]: E0724 09:07:31.753078 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:07:33 bluth k3s[11030]: E0724 09:07:33.765643 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:07:35 bluth k3s[11030]: E0724 09:07:35.791129 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:07:37 bluth k3s[11030]: E0724 09:07:37.757219 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:07:39 bluth k3s[11030]: E0724 09:07:39.770456 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:07:41 bluth k3s[11030]: E0724 09:07:41.785309 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:07:43 bluth k3s[11030]: E0724 09:07:43.762344 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:07:45 bluth k3s[11030]: E0724 09:07:45.769913 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:07:47 bluth k3s[11030]: E0724 09:07:47.756992 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:07:49 bluth k3s[11030]: E0724 09:07:49.762461 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:07:50 bluth k3s[11030]: I0724 09:07:50.561948 11030 trace.go:205] Trace[1146565080]: "GuaranteedUpdate etcd3" type:*coordination.Lease (24-Jul-2022 09:07:49.932) (total time: 628ms): Jul 24 09:07:50 bluth k3s[11030]: Trace[1146565080]: ---"Transaction committed" 628ms (09:07:50.561) Jul 24 09:07:50 bluth k3s[11030]: Trace[1146565080]: [628.807388ms] [628.807388ms] END Jul 24 09:07:50 bluth k3s[11030]: I0724 09:07:50.562593 11030 trace.go:205] Trace[735498105]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/zfs-csi-openebs-io,user-agent:csi-provisioner/v0.0.0 (linux/amd64) kubernetes/$Format,audit-id:26abfee0-61c0-4caa-9cfc-872a46a14834,client:172.16.0.211,accept:application/json, */*,protocol:HTTP/2.0 (24-Jul-2022 09:07:49.932) (total time: 629ms): Jul 24 09:07:50 bluth k3s[11030]: Trace[735498105]: ---"Object stored in database" 629ms (09:07:50.562) Jul 24 09:07:50 bluth k3s[11030]: Trace[735498105]: [629.559721ms] [629.559721ms] END Jul 24 09:07:51 bluth k3s[11030]: E0724 09:07:51.757091 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:07:53 bluth k3s[11030]: E0724 09:07:53.762340 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:07:54 bluth k3s[11030]: I0724 09:07:54.818157 11030 trace.go:205] Trace[1136114217]: "GuaranteedUpdate etcd3" type:*v1.Endpoints (24-Jul-2022 09:07:52.204) (total time: 2613ms): Jul 24 09:07:54 bluth k3s[11030]: Trace[1136114217]: ---"Transaction committed" 2610ms (09:07:54.818) Jul 24 09:07:54 bluth k3s[11030]: Trace[1136114217]: [2.613403356s] [2.613403356s] END Jul 24 09:07:55 bluth k3s[11030]: E0724 09:07:55.758079 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:07:56 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T09:07:56.967+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 09:07:57 bluth k3s[11030]: E0724 09:07:57.762450 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:07:57 bluth k3s[11030]: I0724 09:07:57.909979 11030 trace.go:205] Trace[1533616239]: "GuaranteedUpdate etcd3" type:*coordination.Lease (24-Jul-2022 09:07:54.975) (total time: 2934ms): Jul 24 09:07:57 bluth k3s[11030]: Trace[1533616239]: ---"Transaction committed" 2933ms (09:07:57.909) Jul 24 09:07:57 bluth k3s[11030]: Trace[1533616239]: [2.934343923s] [2.934343923s] END Jul 24 09:07:57 bluth k3s[11030]: I0724 09:07:57.910236 11030 trace.go:205] Trace[217261105]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/external-resizer-zfs-csi-openebs-io,user-agent:csi-resizer/v0.0.0 (linux/amd64) kubernetes/$Format,audit-id:b15ac0eb-b19a-4e62-b63c-037f0b5f8173,client:172.16.0.211,accept:application/json, */*,protocol:HTTP/2.0 (24-Jul-2022 09:07:54.975) (total time: 2934ms): Jul 24 09:07:57 bluth k3s[11030]: Trace[217261105]: ---"Object stored in database" 2934ms (09:07:57.910) Jul 24 09:07:57 bluth k3s[11030]: Trace[217261105]: [2.934806708s] [2.934806708s] END Jul 24 09:07:57 bluth k3s[11030]: I0724 09:07:57.912012 11030 trace.go:205] Trace[44551259]: "GuaranteedUpdate etcd3" type:*coordination.Lease (24-Jul-2022 09:07:55.577) (total time: 2334ms): Jul 24 09:07:57 bluth k3s[11030]: Trace[44551259]: ---"Transaction committed" 2334ms (09:07:57.911) Jul 24 09:07:57 bluth k3s[11030]: Trace[44551259]: [2.334462878s] [2.334462878s] END Jul 24 09:07:57 bluth k3s[11030]: I0724 09:07:57.912699 11030 trace.go:205] Trace[1014275359]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/zfs-csi-openebs-io,user-agent:csi-provisioner/v0.0.0 (linux/amd64) kubernetes/$Format,audit-id:0fec09d7-6c4b-4e41-ae18-7119d45c256a,client:172.16.0.211,accept:application/json, */*,protocol:HTTP/2.0 (24-Jul-2022 09:07:55.577) (total time: 2335ms): Jul 24 09:07:57 bluth k3s[11030]: Trace[1014275359]: ---"Object stored in database" 2335ms (09:07:57.912) Jul 24 09:07:57 bluth k3s[11030]: Trace[1014275359]: [2.335267987s] [2.335267987s] END Jul 24 09:07:57 bluth k3s[11030]: I0724 09:07:57.915604 11030 trace.go:205] Trace[1952649911]: "GuaranteedUpdate etcd3" type:*coordination.Lease (24-Jul-2022 09:07:54.978) (total time: 2937ms): Jul 24 09:07:57 bluth k3s[11030]: Trace[1952649911]: ---"Transaction committed" 2936ms (09:07:57.915) Jul 24 09:07:57 bluth k3s[11030]: Trace[1952649911]: [2.937046464s] [2.937046464s] END Jul 24 09:07:57 bluth k3s[11030]: I0724 09:07:57.915845 11030 trace.go:205] Trace[1137252778]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/snapshot-controller-leader,user-agent:snapshot-controller/v0.0.0 (linux/amd64) kubernetes/$Format,audit-id:04019354-33c9-458a-a2cb-70d940206856,client:172.16.0.211,accept:application/json, */*,protocol:HTTP/2.0 (24-Jul-2022 09:07:54.978) (total time: 2937ms): Jul 24 09:07:57 bluth k3s[11030]: Trace[1137252778]: ---"Object stored in database" 2937ms (09:07:57.915) Jul 24 09:07:57 bluth k3s[11030]: Trace[1137252778]: [2.937496799s] [2.937496799s] END Jul 24 09:07:58 bluth k3s[11030]: I0724 09:07:58.012587 11030 trace.go:205] Trace[1403669541]: "GuaranteedUpdate etcd3" type:*coordination.Lease (24-Jul-2022 09:07:54.975) (total time: 3037ms): Jul 24 09:07:58 bluth k3s[11030]: Trace[1403669541]: ---"Transaction committed" 3036ms (09:07:58.012) Jul 24 09:07:58 bluth k3s[11030]: Trace[1403669541]: [3.037030655s] [3.037030655s] END Jul 24 09:07:58 bluth k3s[11030]: I0724 09:07:58.012839 11030 trace.go:205] Trace[1206931013]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/external-snapshotter-leader-zfs-csi-openebs-io,user-agent:csi-snapshotter/v0.0.0 (linux/amd64) kubernetes/$Format,audit-id:3faf5712-1dcf-4b5e-88d2-94fd01ed0831,client:172.16.0.211,accept:application/json, */*,protocol:HTTP/2.0 (24-Jul-2022 09:07:54.975) (total time: 3037ms): Jul 24 09:07:58 bluth k3s[11030]: Trace[1206931013]: ---"Object stored in database" 3037ms (09:07:58.012) Jul 24 09:07:58 bluth k3s[11030]: Trace[1206931013]: [3.037518649s] [3.037518649s] END Jul 24 09:07:59 bluth k3s[11030]: E0724 09:07:59.757174 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:08:01 bluth k3s[11030]: E0724 09:08:01.762259 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:08:03 bluth k3s[11030]: E0724 09:08:03.763519 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:08:05 bluth k3s[11030]: E0724 09:08:05.805011 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:08:07 bluth k3s[11030]: E0724 09:08:07.785223 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:08:09 bluth k3s[11030]: E0724 09:08:09.762261 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:08:11 bluth k3s[11030]: E0724 09:08:11.757007 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:08:13 bluth k3s[11030]: E0724 09:08:13.762480 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:08:15 bluth k3s[11030]: E0724 09:08:15.756460 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:08:17 bluth k3s[11030]: E0724 09:08:17.758485 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:08:19 bluth k3s[11030]: E0724 09:08:19.804426 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:08:21 bluth k3s[11030]: E0724 09:08:21.756849 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:08:23 bluth k3s[11030]: E0724 09:08:23.760711 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:08:25 bluth k3s[11030]: E0724 09:08:25.753473 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:08:27 bluth k3s[11030]: E0724 09:08:27.759030 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:08:29 bluth k3s[11030]: E0724 09:08:29.761352 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:08:31 bluth k3s[11030]: E0724 09:08:31.762234 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:08:33 bluth k3s[11030]: E0724 09:08:33.752342 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:08:33 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T09:08:33.820+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 09:08:35 bluth k3s[11030]: E0724 09:08:35.765588 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:08:37 bluth k3s[11030]: E0724 09:08:37.762277 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:08:39 bluth k3s[11030]: E0724 09:08:39.769696 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:08:41 bluth k3s[11030]: E0724 09:08:41.781147 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:08:43 bluth k3s[11030]: E0724 09:08:43.752242 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:08:45 bluth k3s[11030]: E0724 09:08:45.762176 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:08:47 bluth k3s[11030]: E0724 09:08:47.756655 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:08:49 bluth k3s[11030]: E0724 09:08:49.761841 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:08:51 bluth k3s[11030]: E0724 09:08:51.761962 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:08:53 bluth k3s[11030]: E0724 09:08:53.762010 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:08:55 bluth k3s[11030]: E0724 09:08:55.756940 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:08:57 bluth k3s[11030]: E0724 09:08:57.762328 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:08:59 bluth k3s[11030]: E0724 09:08:59.785090 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:09:01 bluth k3s[11030]: E0724 09:09:01.758392 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:09:03 bluth k3s[11030]: E0724 09:09:03.761993 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:09:05 bluth k3s[11030]: E0724 09:09:05.762431 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:09:06 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T09:09:06.691+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 09:09:07 bluth k3s[11030]: E0724 09:09:07.756505 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:09:09 bluth k3s[11030]: E0724 09:09:09.768966 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:09:11 bluth k3s[11030]: E0724 09:09:11.757275 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:09:13 bluth k3s[11030]: E0724 09:09:13.803870 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:09:15 bluth k3s[11030]: E0724 09:09:15.770057 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:09:17 bluth k3s[11030]: E0724 09:09:17.782686 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:09:19 bluth k3s[11030]: E0724 09:09:19.762010 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:09:21 bluth k3s[11030]: E0724 09:09:21.762358 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:09:23 bluth k3s[11030]: E0724 09:09:23.757063 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:09:25 bluth k3s[11030]: E0724 09:09:25.758371 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:09:27 bluth k3s[11030]: E0724 09:09:27.768951 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:09:29 bluth k3s[11030]: E0724 09:09:29.757127 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:09:31 bluth k3s[11030]: E0724 09:09:31.761802 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:09:33 bluth k3s[11030]: E0724 09:09:33.770402 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:09:35 bluth k3s[11030]: E0724 09:09:35.777048 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:09:37 bluth k3s[11030]: E0724 09:09:37.762232 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:09:39 bluth k3s[11030]: E0724 09:09:39.754821 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:09:41 bluth k3s[11030]: I0724 09:09:41.182139 11030 trace.go:205] Trace[800638407]: "GuaranteedUpdate etcd3" type:*coordination.Lease (24-Jul-2022 09:09:38.291) (total time: 2890ms): Jul 24 09:09:41 bluth k3s[11030]: Trace[800638407]: ---"Transaction committed" 2890ms (09:09:41.182) Jul 24 09:09:41 bluth k3s[11030]: Trace[800638407]: [2.890988038s] [2.890988038s] END Jul 24 09:09:41 bluth k3s[11030]: I0724 09:09:41.182433 11030 trace.go:205] Trace[795104760]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/external-resizer-zfs-csi-openebs-io,user-agent:csi-resizer/v0.0.0 (linux/amd64) kubernetes/$Format,audit-id:15eceb58-d349-43c9-a0b8-413c92629c8c,client:172.16.0.211,accept:application/json, */*,protocol:HTTP/2.0 (24-Jul-2022 09:09:38.290) (total time: 2891ms): Jul 24 09:09:41 bluth k3s[11030]: Trace[795104760]: ---"Object stored in database" 2891ms (09:09:41.182) Jul 24 09:09:41 bluth k3s[11030]: Trace[795104760]: [2.891379395s] [2.891379395s] END Jul 24 09:09:41 bluth k3s[11030]: E0724 09:09:41.762135 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:09:43 bluth k3s[11030]: I0724 09:09:43.463995 11030 trace.go:205] Trace[293404852]: "GuaranteedUpdate etcd3" type:*v1.Endpoints (24-Jul-2022 09:09:42.213) (total time: 1250ms): Jul 24 09:09:43 bluth k3s[11030]: Trace[293404852]: ---"Transaction committed" 1247ms (09:09:43.463) Jul 24 09:09:43 bluth k3s[11030]: Trace[293404852]: [1.250122953s] [1.250122953s] END Jul 24 09:09:43 bluth k3s[11030]: E0724 09:09:43.762408 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:09:45 bluth k3s[11030]: E0724 09:09:45.757273 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:09:46 bluth k3s[11030]: I0724 09:09:46.752639 11030 trace.go:205] Trace[484863764]: "GuaranteedUpdate etcd3" type:*coordination.Lease (24-Jul-2022 09:09:46.211) (total time: 541ms): Jul 24 09:09:46 bluth k3s[11030]: Trace[484863764]: ---"Transaction committed" 541ms (09:09:46.752) Jul 24 09:09:46 bluth k3s[11030]: Trace[484863764]: [541.382881ms] [541.382881ms] END Jul 24 09:09:46 bluth k3s[11030]: I0724 09:09:46.752728 11030 trace.go:205] Trace[1512737097]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/external-resizer-zfs-csi-openebs-io,user-agent:csi-resizer/v0.0.0 (linux/amd64) kubernetes/$Format,audit-id:aa69b19b-8ba2-413f-9242-c34c3a15ea5b,client:172.16.0.211,accept:application/json, */*,protocol:HTTP/2.0 (24-Jul-2022 09:09:46.211) (total time: 541ms): Jul 24 09:09:46 bluth k3s[11030]: Trace[1512737097]: ---"Object stored in database" 541ms (09:09:46.752) Jul 24 09:09:46 bluth k3s[11030]: Trace[1512737097]: [541.587301ms] [541.587301ms] END Jul 24 09:09:47 bluth k3s[11030]: E0724 09:09:47.760850 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:09:49 bluth k3s[11030]: I0724 09:09:49.690256 11030 trace.go:205] Trace[463488865]: "GuaranteedUpdate etcd3" type:*coordination.Lease (24-Jul-2022 09:09:48.317) (total time: 1373ms): Jul 24 09:09:49 bluth k3s[11030]: Trace[463488865]: ---"Transaction committed" 1372ms (09:09:49.690) Jul 24 09:09:49 bluth k3s[11030]: Trace[463488865]: [1.373187498s] [1.373187498s] END Jul 24 09:09:49 bluth k3s[11030]: I0724 09:09:49.690532 11030 trace.go:205] Trace[2144053162]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/zfs-csi-openebs-io,user-agent:csi-provisioner/v0.0.0 (linux/amd64) kubernetes/$Format,audit-id:8aefe7f8-9e6e-4938-b2a8-5c8181bb208f,client:172.16.0.211,accept:application/json, */*,protocol:HTTP/2.0 (24-Jul-2022 09:09:48.316) (total time: 1373ms): Jul 24 09:09:49 bluth k3s[11030]: Trace[2144053162]: ---"Object stored in database" 1373ms (09:09:49.690) Jul 24 09:09:49 bluth k3s[11030]: Trace[2144053162]: [1.373594421s] [1.373594421s] END Jul 24 09:09:49 bluth k3s[11030]: I0724 09:09:49.749439 11030 trace.go:205] Trace[939214187]: "GuaranteedUpdate etcd3" type:*coordination.Lease (24-Jul-2022 09:09:48.316) (total time: 1432ms): Jul 24 09:09:49 bluth k3s[11030]: Trace[939214187]: ---"Transaction committed" 1432ms (09:09:49.749) Jul 24 09:09:49 bluth k3s[11030]: Trace[939214187]: [1.432523964s] [1.432523964s] END Jul 24 09:09:49 bluth k3s[11030]: I0724 09:09:49.749506 11030 trace.go:205] Trace[1775999837]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/snapshot-controller-leader,user-agent:snapshot-controller/v0.0.0 (linux/amd64) kubernetes/$Format,audit-id:35e577bf-21bd-4e3f-991b-7a0827af3285,client:172.16.0.211,accept:application/json, */*,protocol:HTTP/2.0 (24-Jul-2022 09:09:48.316) (total time: 1432ms): Jul 24 09:09:49 bluth k3s[11030]: Trace[1775999837]: ---"Object stored in database" 1432ms (09:09:49.749) Jul 24 09:09:49 bluth k3s[11030]: Trace[1775999837]: [1.432750638s] [1.432750638s] END Jul 24 09:09:49 bluth k3s[11030]: I0724 09:09:49.750286 11030 trace.go:205] Trace[393009022]: "GuaranteedUpdate etcd3" type:*coordination.Lease (24-Jul-2022 09:09:48.317) (total time: 1432ms): Jul 24 09:09:49 bluth k3s[11030]: Trace[393009022]: ---"Transaction committed" 1432ms (09:09:49.750) Jul 24 09:09:49 bluth k3s[11030]: Trace[393009022]: [1.432539693s] [1.432539693s] END Jul 24 09:09:49 bluth k3s[11030]: I0724 09:09:49.750361 11030 trace.go:205] Trace[315638126]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/external-snapshotter-leader-zfs-csi-openebs-io,user-agent:csi-snapshotter/v0.0.0 (linux/amd64) kubernetes/$Format,audit-id:47c50f8e-6e9e-4d3f-8883-1d13922f9c34,client:172.16.0.211,accept:application/json, */*,protocol:HTTP/2.0 (24-Jul-2022 09:09:48.317) (total time: 1432ms): Jul 24 09:09:49 bluth k3s[11030]: Trace[315638126]: ---"Object stored in database" 1432ms (09:09:49.750) Jul 24 09:09:49 bluth k3s[11030]: Trace[315638126]: [1.432764396s] [1.432764396s] END Jul 24 09:09:49 bluth k3s[11030]: E0724 09:09:49.762140 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:09:50 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T09:09:50.715+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 09:09:51 bluth k3s[11030]: E0724 09:09:51.752846 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:09:53 bluth k3s[11030]: E0724 09:09:53.762890 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:09:55 bluth k3s[11030]: E0724 09:09:55.756555 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:09:57 bluth k3s[11030]: E0724 09:09:57.766712 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:09:59 bluth k3s[11030]: E0724 09:09:59.761946 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:10:01 bluth k3s[11030]: E0724 09:10:01.770304 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:10:03 bluth k3s[11030]: E0724 09:10:03.764789 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:10:05 bluth k3s[11030]: E0724 09:10:05.763559 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:10:07 bluth k3s[11030]: E0724 09:10:07.752868 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:10:09 bluth k3s[11030]: E0724 09:10:09.759780 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:10:11 bluth k3s[11030]: E0724 09:10:11.762552 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:10:13 bluth k3s[11030]: E0724 09:10:13.762410 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:10:15 bluth k3s[11030]: E0724 09:10:15.758991 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:10:17 bluth k3s[11030]: E0724 09:10:17.762338 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:10:19 bluth k3s[11030]: E0724 09:10:19.753955 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:10:21 bluth k3s[11030]: E0724 09:10:21.762747 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:10:23 bluth k3s[11030]: E0724 09:10:23.805165 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:10:25 bluth k3s[11030]: E0724 09:10:25.752976 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:10:27 bluth k3s[11030]: E0724 09:10:27.783504 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:10:29 bluth k3s[11030]: E0724 09:10:29.757195 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:10:31 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T09:10:31.131+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 09:10:31 bluth k3s[11030]: E0724 09:10:31.770161 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:10:33 bluth k3s[11030]: E0724 09:10:33.769958 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:10:35 bluth k3s[11030]: E0724 09:10:35.807419 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:10:37 bluth k3s[11030]: E0724 09:10:37.757132 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:10:39 bluth k3s[11030]: E0724 09:10:39.781594 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:10:41 bluth k3s[11030]: E0724 09:10:41.764710 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:10:43 bluth k3s[11030]: E0724 09:10:43.804168 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:10:45 bluth k3s[11030]: E0724 09:10:45.782704 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:10:47 bluth k3s[11030]: E0724 09:10:47.791078 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:10:49 bluth k3s[11030]: E0724 09:10:49.762484 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:10:51 bluth k3s[11030]: E0724 09:10:51.763286 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:10:53 bluth k3s[11030]: E0724 09:10:53.762425 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:10:55 bluth k3s[11030]: E0724 09:10:55.762502 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:10:57 bluth k3s[11030]: E0724 09:10:57.756595 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:10:59 bluth k3s[11030]: E0724 09:10:59.797605 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:11:01 bluth k3s[11030]: E0724 09:11:01.756817 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:11:02 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T09:11:02.240+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 09:11:03 bluth k3s[11030]: E0724 09:11:03.762518 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:11:05 bluth k3s[11030]: E0724 09:11:05.762604 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:11:07 bluth k3s[11030]: E0724 09:11:07.756674 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:11:09 bluth k3s[11030]: E0724 09:11:09.770265 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:11:11 bluth k3s[11030]: E0724 09:11:11.757219 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:11:13 bluth k3s[11030]: E0724 09:11:13.762288 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:11:15 bluth k3s[11030]: E0724 09:11:15.762581 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:11:17 bluth k3s[11030]: E0724 09:11:17.756983 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:11:19 bluth k3s[11030]: E0724 09:11:19.770945 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:11:21 bluth k3s[11030]: E0724 09:11:21.757576 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:11:23 bluth k3s[11030]: I0724 09:11:23.024811 11030 trace.go:205] Trace[1144488470]: "GuaranteedUpdate etcd3" type:*coordination.Lease (24-Jul-2022 09:11:22.391) (total time: 633ms): Jul 24 09:11:23 bluth k3s[11030]: Trace[1144488470]: ---"Transaction committed" 633ms (09:11:23.024) Jul 24 09:11:23 bluth k3s[11030]: Trace[1144488470]: [633.440086ms] [633.440086ms] END Jul 24 09:11:23 bluth k3s[11030]: I0724 09:11:23.025129 11030 trace.go:205] Trace[423621452]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ix-truenas,user-agent:Go-http-client/2.0,audit-id:51c3f7c1-aa16-448f-b952-d677aab9c02c,client:127.0.0.1,accept:application/vnd.kubernetes.protobuf,application/json,protocol:HTTP/2.0 (24-Jul-2022 09:11:22.391) (total time: 633ms): Jul 24 09:11:23 bluth k3s[11030]: Trace[423621452]: ---"Object stored in database" 633ms (09:11:23.024) Jul 24 09:11:23 bluth k3s[11030]: Trace[423621452]: [633.823641ms] [633.823641ms] END Jul 24 09:11:23 bluth k3s[11030]: I0724 09:11:23.053008 11030 trace.go:205] Trace[1659299831]: "GuaranteedUpdate etcd3" type:*v1.Endpoints (24-Jul-2022 09:11:22.217) (total time: 835ms): Jul 24 09:11:23 bluth k3s[11030]: Trace[1659299831]: ---"Transaction committed" 833ms (09:11:23.052) Jul 24 09:11:23 bluth k3s[11030]: Trace[1659299831]: [835.946688ms] [835.946688ms] END Jul 24 09:11:23 bluth k3s[11030]: I0724 09:11:23.086777 11030 trace.go:205] Trace[1288060974]: "GuaranteedUpdate etcd3" type:*coordination.Lease (24-Jul-2022 09:11:22.054) (total time: 1031ms): Jul 24 09:11:23 bluth k3s[11030]: Trace[1288060974]: ---"Transaction committed" 1031ms (09:11:23.086) Jul 24 09:11:23 bluth k3s[11030]: Trace[1288060974]: [1.03178377s] [1.03178377s] END Jul 24 09:11:23 bluth k3s[11030]: I0724 09:11:23.086992 11030 trace.go:205] Trace[872632595]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/external-resizer-zfs-csi-openebs-io,user-agent:csi-resizer/v0.0.0 (linux/amd64) kubernetes/$Format,audit-id:6677cbaa-775b-4606-970b-a6e05b75d1dd,client:172.16.0.211,accept:application/json, */*,protocol:HTTP/2.0 (24-Jul-2022 09:11:22.054) (total time: 1032ms): Jul 24 09:11:23 bluth k3s[11030]: Trace[872632595]: ---"Object stored in database" 1031ms (09:11:23.086) Jul 24 09:11:23 bluth k3s[11030]: Trace[872632595]: [1.032116473s] [1.032116473s] END Jul 24 09:11:23 bluth k3s[11030]: E0724 09:11:23.762411 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:11:25 bluth k3s[11030]: E0724 09:11:25.752064 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:11:27 bluth k3s[11030]: E0724 09:11:27.761221 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:11:29 bluth k3s[11030]: E0724 09:11:29.802530 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:11:31 bluth k3s[11030]: E0724 09:11:31.798438 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:11:32 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T09:11:32.401+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 09:11:33 bluth k3s[11030]: E0724 09:11:33.758422 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:11:35 bluth k3s[11030]: E0724 09:11:35.806668 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:11:37 bluth k3s[11030]: E0724 09:11:37.756114 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:11:39 bluth k3s[11030]: E0724 09:11:39.762651 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:11:41 bluth k3s[11030]: E0724 09:11:41.780171 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:11:43 bluth k3s[11030]: E0724 09:11:43.757471 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:11:45 bluth k3s[11030]: E0724 09:11:45.762488 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:11:47 bluth k3s[11030]: E0724 09:11:47.756825 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:11:49 bluth k3s[11030]: E0724 09:11:49.762386 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:11:51 bluth k3s[11030]: E0724 09:11:51.763310 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:11:53 bluth k3s[11030]: E0724 09:11:53.756873 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:11:55 bluth k3s[11030]: E0724 09:11:55.762493 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:11:57 bluth k3s[11030]: E0724 09:11:57.762757 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:11:59 bluth k3s[11030]: E0724 09:11:59.756982 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:12:01 bluth k3s[11030]: E0724 09:12:01.769438 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:12:03 bluth k3s[11030]: E0724 09:12:03.803496 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:12:05 bluth k3s[11030]: E0724 09:12:05.806184 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:12:07 bluth k3s[11030]: E0724 09:12:07.756977 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:12:09 bluth k3s[11030]: E0724 09:12:09.757531 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:12:11 bluth k3s[11030]: E0724 09:12:11.769560 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:12:11 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T09:12:11.803+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 09:12:13 bluth k3s[11030]: E0724 09:12:13.756482 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:12:15 bluth k3s[11030]: E0724 09:12:15.762531 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:12:17 bluth k3s[11030]: E0724 09:12:17.757047 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:12:19 bluth k3s[11030]: E0724 09:12:19.762454 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:12:21 bluth k3s[11030]: E0724 09:12:21.769581 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:12:23 bluth k3s[11030]: E0724 09:12:23.770508 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:12:25 bluth k3s[11030]: E0724 09:12:25.752878 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:12:27 bluth k3s[11030]: E0724 09:12:27.764292 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:12:29 bluth k3s[11030]: E0724 09:12:29.762329 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:12:31 bluth k3s[11030]: E0724 09:12:31.770640 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:12:33 bluth k3s[11030]: E0724 09:12:33.757947 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:12:35 bluth k3s[11030]: E0724 09:12:35.809315 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:12:37 bluth k3s[11030]: E0724 09:12:37.757123 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:12:39 bluth k3s[11030]: E0724 09:12:39.762560 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:12:41 bluth k3s[11030]: E0724 09:12:41.762554 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:12:43 bluth k3s[11030]: E0724 09:12:43.762917 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:12:45 bluth k3s[11030]: E0724 09:12:45.762550 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:12:47 bluth k3s[11030]: E0724 09:12:47.762957 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:12:49 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T09:12:49.579+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 09:12:49 bluth k3s[11030]: E0724 09:12:49.769684 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:12:51 bluth k3s[11030]: E0724 09:12:51.769507 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:12:53 bluth k3s[11030]: E0724 09:12:53.762294 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:12:55 bluth k3s[11030]: E0724 09:12:55.763012 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:12:57 bluth k3s[11030]: E0724 09:12:57.763091 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:12:59 bluth k3s[11030]: E0724 09:12:59.757178 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:13:01 bluth k3s[11030]: E0724 09:13:01.762721 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:13:03 bluth k3s[11030]: E0724 09:13:03.757117 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:13:05 bluth k3s[11030]: E0724 09:13:05.805488 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:13:07 bluth k3s[11030]: E0724 09:13:07.757151 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:13:09 bluth k3s[11030]: E0724 09:13:09.770875 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:13:11 bluth k3s[11030]: E0724 09:13:11.753348 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:13:12 bluth k3s[11030]: I0724 09:13:12.172208 11030 trace.go:205] Trace[1578559356]: "GuaranteedUpdate etcd3" type:*coordination.Lease (24-Jul-2022 09:13:10.326) (total time: 1845ms): Jul 24 09:13:12 bluth k3s[11030]: Trace[1578559356]: ---"Transaction committed" 1845ms (09:13:12.172) Jul 24 09:13:12 bluth k3s[11030]: Trace[1578559356]: [1.845794132s] [1.845794132s] END Jul 24 09:13:12 bluth k3s[11030]: I0724 09:13:12.172464 11030 trace.go:205] Trace[1987752301]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/zfs-csi-openebs-io,user-agent:csi-provisioner/v0.0.0 (linux/amd64) kubernetes/$Format,audit-id:81ff3ecf-3dde-4979-980d-dc296c41c084,client:172.16.0.211,accept:application/json, */*,protocol:HTTP/2.0 (24-Jul-2022 09:13:10.326) (total time: 1846ms): Jul 24 09:13:12 bluth k3s[11030]: Trace[1987752301]: ---"Object stored in database" 1845ms (09:13:12.172) Jul 24 09:13:12 bluth k3s[11030]: Trace[1987752301]: [1.846179359s] [1.846179359s] END Jul 24 09:13:13 bluth k3s[11030]: E0724 09:13:13.769285 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:13:15 bluth k3s[11030]: E0724 09:13:15.757346 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:13:17 bluth k3s[11030]: E0724 09:13:17.762903 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:13:19 bluth k3s[11030]: E0724 09:13:19.755432 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:13:20 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T09:13:20.501+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 09:13:21 bluth k3s[11030]: E0724 09:13:21.758730 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:13:23 bluth k3s[11030]: E0724 09:13:23.762829 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:13:25 bluth k3s[11030]: E0724 09:13:25.752295 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:13:27 bluth k3s[11030]: E0724 09:13:27.770976 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:13:29 bluth k3s[11030]: E0724 09:13:29.752692 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:13:31 bluth k3s[11030]: E0724 09:13:31.762877 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:13:33 bluth k3s[11030]: E0724 09:13:33.752685 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:13:35 bluth k3s[11030]: E0724 09:13:35.756449 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:13:37 bluth k3s[11030]: E0724 09:13:37.762673 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:13:39 bluth k3s[11030]: E0724 09:13:39.756259 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:13:41 bluth k3s[11030]: E0724 09:13:41.762493 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:13:43 bluth k3s[11030]: E0724 09:13:43.785249 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:13:45 bluth k3s[11030]: E0724 09:13:45.761914 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:13:47 bluth k3s[11030]: E0724 09:13:47.764912 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:13:49 bluth k3s[11030]: E0724 09:13:49.757416 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:13:51 bluth k3s[11030]: E0724 09:13:51.771081 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:13:53 bluth k3s[11030]: E0724 09:13:53.754273 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:13:55 bluth k3s[11030]: E0724 09:13:55.770287 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:13:55 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T09:13:55.881+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 09:13:57 bluth k3s[11030]: E0724 09:13:57.757512 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:13:59 bluth k3s[11030]: E0724 09:13:59.761679 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:14:01 bluth k3s[11030]: E0724 09:14:01.761669 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:14:03 bluth k3s[11030]: E0724 09:14:03.762604 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:14:05 bluth k3s[11030]: E0724 09:14:05.811321 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:14:07 bluth k3s[11030]: E0724 09:14:07.786477 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:14:09 bluth k3s[11030]: E0724 09:14:09.756648 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:14:11 bluth k3s[11030]: E0724 09:14:11.762017 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:14:13 bluth k3s[11030]: E0724 09:14:13.757921 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:14:15 bluth k3s[11030]: E0724 09:14:15.782084 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:14:17 bluth k3s[11030]: E0724 09:14:17.756699 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:14:19 bluth k3s[11030]: E0724 09:14:19.769938 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:14:21 bluth k3s[11030]: E0724 09:14:21.756721 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:14:23 bluth k3s[11030]: E0724 09:14:23.766114 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:14:25 bluth k3s[11030]: E0724 09:14:25.753622 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:14:27 bluth k3s[11030]: E0724 09:14:27.761956 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:14:29 bluth k3s[11030]: E0724 09:14:29.756636 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:14:31 bluth k3s[11030]: E0724 09:14:31.761769 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:14:33 bluth k3s[11030]: E0724 09:14:33.786845 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:14:35 bluth k3s[11030]: E0724 09:14:35.765040 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:14:37 bluth k3s[11030]: E0724 09:14:37.761262 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:14:39 bluth k3s[11030]: E0724 09:14:39.770115 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:14:40 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T09:14:40.300+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 09:14:41 bluth k3s[11030]: E0724 09:14:41.757253 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:14:43 bluth k3s[11030]: E0724 09:14:43.763200 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:14:45 bluth k3s[11030]: E0724 09:14:45.756580 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:14:47 bluth k3s[11030]: E0724 09:14:47.761978 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:14:49 bluth k3s[11030]: E0724 09:14:49.763276 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:14:51 bluth k3s[11030]: E0724 09:14:51.756807 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:14:53 bluth k3s[11030]: E0724 09:14:53.762268 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:14:54 bluth k3s[11030]: I0724 09:14:54.475149 11030 trace.go:205] Trace[2027815436]: "GuaranteedUpdate etcd3" type:*coordination.Lease (24-Jul-2022 09:14:53.921) (total time: 553ms): Jul 24 09:14:54 bluth k3s[11030]: Trace[2027815436]: ---"Transaction committed" 552ms (09:14:54.475) Jul 24 09:14:54 bluth k3s[11030]: Trace[2027815436]: [553.17016ms] [553.17016ms] END Jul 24 09:14:54 bluth k3s[11030]: I0724 09:14:54.475425 11030 trace.go:205] Trace[1629883744]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/external-resizer-zfs-csi-openebs-io,user-agent:csi-resizer/v0.0.0 (linux/amd64) kubernetes/$Format,audit-id:f871fb45-b790-44d4-b05b-1e4e853699b4,client:172.16.0.211,accept:application/json, */*,protocol:HTTP/2.0 (24-Jul-2022 09:14:53.921) (total time: 553ms): Jul 24 09:14:54 bluth k3s[11030]: Trace[1629883744]: ---"Object stored in database" 553ms (09:14:54.475) Jul 24 09:14:54 bluth k3s[11030]: Trace[1629883744]: [553.546034ms] [553.546034ms] END Jul 24 09:14:55 bluth k3s[11030]: E0724 09:14:55.755184 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:14:57 bluth k3s[11030]: E0724 09:14:57.762318 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:14:59 bluth k3s[11030]: E0724 09:14:59.786365 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:15:01 bluth k3s[11030]: E0724 09:15:01.757884 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:15:03 bluth k3s[11030]: E0724 09:15:03.762104 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:15:05 bluth k3s[11030]: E0724 09:15:05.805938 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:15:07 bluth k3s[11030]: E0724 09:15:07.765234 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:15:09 bluth k3s[11030]: E0724 09:15:09.762379 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:15:11 bluth k3s[11030]: E0724 09:15:11.769004 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:15:12 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T09:15:12.955+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 09:15:13 bluth k3s[11030]: E0724 09:15:13.758191 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:15:15 bluth k3s[11030]: E0724 09:15:15.762411 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:15:17 bluth k3s[11030]: E0724 09:15:17.756958 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:15:19 bluth k3s[11030]: E0724 09:15:19.762347 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:15:21 bluth k3s[11030]: E0724 09:15:21.770702 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:15:23 bluth k3s[11030]: E0724 09:15:23.756767 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:15:25 bluth k3s[11030]: E0724 09:15:25.758857 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:15:27 bluth k3s[11030]: E0724 09:15:27.770065 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:15:29 bluth k3s[11030]: E0724 09:15:29.756957 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:15:31 bluth k3s[11030]: E0724 09:15:31.762505 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:15:33 bluth k3s[11030]: E0724 09:15:33.757040 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:15:35 bluth k3s[11030]: E0724 09:15:35.815735 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:15:37 bluth k3s[11030]: E0724 09:15:37.775905 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:15:39 bluth k3s[11030]: E0724 09:15:39.769491 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:15:41 bluth k3s[11030]: E0724 09:15:41.752402 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:15:43 bluth k3s[11030]: E0724 09:15:43.762483 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:15:45 bluth k3s[11030]: E0724 09:15:45.764233 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:15:47 bluth k3s[11030]: E0724 09:15:47.786467 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:15:48 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T09:15:48.048+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 09:15:49 bluth k3s[11030]: E0724 09:15:49.757169 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:15:51 bluth k3s[11030]: E0724 09:15:51.762377 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:15:53 bluth k3s[11030]: E0724 09:15:53.788240 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:15:55 bluth k3s[11030]: E0724 09:15:55.757142 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:15:57 bluth k3s[11030]: E0724 09:15:57.782181 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:15:59 bluth k3s[11030]: E0724 09:15:59.780893 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:16:01 bluth k3s[11030]: E0724 09:16:01.757262 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:16:03 bluth k3s[11030]: E0724 09:16:03.762645 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:16:05 bluth k3s[11030]: E0724 09:16:05.807198 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:16:07 bluth k3s[11030]: E0724 09:16:07.757398 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:16:09 bluth k3s[11030]: E0724 09:16:09.780896 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:16:11 bluth k3s[11030]: E0724 09:16:11.756128 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:16:13 bluth k3s[11030]: E0724 09:16:13.756904 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:16:15 bluth k3s[11030]: E0724 09:16:15.762344 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:16:17 bluth k3s[11030]: E0724 09:16:17.756116 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:16:19 bluth k3s[11030]: E0724 09:16:19.758200 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:16:21 bluth k3s[11030]: E0724 09:16:21.764385 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:16:22 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T09:16:22.770+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 09:16:23 bluth k3s[11030]: I0724 09:16:23.019959 11030 trace.go:205] Trace[1239988556]: "GuaranteedUpdate etcd3" type:*coordination.Lease (24-Jul-2022 09:16:20.907) (total time: 2112ms): Jul 24 09:16:23 bluth k3s[11030]: Trace[1239988556]: ---"Transaction committed" 2112ms (09:16:23.019) Jul 24 09:16:23 bluth k3s[11030]: Trace[1239988556]: [2.112677413s] [2.112677413s] END Jul 24 09:16:23 bluth k3s[11030]: I0724 09:16:23.020945 11030 trace.go:205] Trace[1508285107]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/snapshot-controller-leader,user-agent:snapshot-controller/v0.0.0 (linux/amd64) kubernetes/$Format,audit-id:15786ee8-4549-44f4-8a9b-a597b908e70f,client:172.16.0.211,accept:application/json, */*,protocol:HTTP/2.0 (24-Jul-2022 09:16:20.906) (total time: 2113ms): Jul 24 09:16:23 bluth k3s[11030]: Trace[1508285107]: ---"Object stored in database" 2113ms (09:16:23.020) Jul 24 09:16:23 bluth k3s[11030]: Trace[1508285107]: [2.113825184s] [2.113825184s] END Jul 24 09:16:23 bluth k3s[11030]: I0724 09:16:23.043181 11030 trace.go:205] Trace[772822465]: "GuaranteedUpdate etcd3" type:*coordination.Lease (24-Jul-2022 09:16:20.907) (total time: 2136ms): Jul 24 09:16:23 bluth k3s[11030]: Trace[772822465]: ---"Transaction committed" 2135ms (09:16:23.043) Jul 24 09:16:23 bluth k3s[11030]: Trace[772822465]: [2.136109791s] [2.136109791s] END Jul 24 09:16:23 bluth k3s[11030]: I0724 09:16:23.043421 11030 trace.go:205] Trace[1270045432]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/external-snapshotter-leader-zfs-csi-openebs-io,user-agent:csi-snapshotter/v0.0.0 (linux/amd64) kubernetes/$Format,audit-id:5a46d960-6998-4b2c-9260-3ce2f36e743e,client:172.16.0.211,accept:application/json, */*,protocol:HTTP/2.0 (24-Jul-2022 09:16:20.906) (total time: 2136ms): Jul 24 09:16:23 bluth k3s[11030]: Trace[1270045432]: ---"Object stored in database" 2136ms (09:16:23.043) Jul 24 09:16:23 bluth k3s[11030]: Trace[1270045432]: [2.136791178s] [2.136791178s] END Jul 24 09:16:23 bluth k3s[11030]: I0724 09:16:23.069979 11030 trace.go:205] Trace[1502470948]: "GuaranteedUpdate etcd3" type:*v1.Endpoints (24-Jul-2022 09:16:22.233) (total time: 836ms): Jul 24 09:16:23 bluth k3s[11030]: Trace[1502470948]: ---"Transaction committed" 833ms (09:16:23.069) Jul 24 09:16:23 bluth k3s[11030]: Trace[1502470948]: [836.171885ms] [836.171885ms] END Jul 24 09:16:23 bluth k3s[11030]: E0724 09:16:23.757292 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:16:25 bluth k3s[11030]: E0724 09:16:25.765562 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:16:27 bluth k3s[11030]: E0724 09:16:27.762456 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:16:29 bluth k3s[11030]: E0724 09:16:29.805313 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:16:31 bluth k3s[11030]: E0724 09:16:31.757214 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:16:33 bluth k3s[11030]: E0724 09:16:33.762487 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:16:35 bluth k3s[11030]: E0724 09:16:35.800785 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:16:37 bluth k3s[11030]: E0724 09:16:37.769722 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:16:39 bluth k3s[11030]: E0724 09:16:39.757044 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:16:41 bluth k3s[11030]: E0724 09:16:41.770593 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:16:43 bluth k3s[11030]: E0724 09:16:43.757194 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:16:45 bluth k3s[11030]: E0724 09:16:45.769641 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:16:47 bluth k3s[11030]: E0724 09:16:47.756942 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:16:49 bluth k3s[11030]: E0724 09:16:49.762238 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:16:51 bluth k3s[11030]: E0724 09:16:51.769676 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:16:53 bluth k3s[11030]: E0724 09:16:53.757139 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:16:55 bluth k3s[11030]: E0724 09:16:55.761342 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:16:57 bluth k3s[11030]: E0724 09:16:57.769793 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:16:59 bluth k3s[11030]: E0724 09:16:59.758108 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:17:00 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T09:17:00.462+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 09:17:01 bluth k3s[11030]: E0724 09:17:01.762618 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:17:03 bluth k3s[11030]: E0724 09:17:03.786030 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:17:05 bluth k3s[11030]: E0724 09:17:05.787029 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:17:07 bluth k3s[11030]: E0724 09:17:07.763443 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:17:09 bluth k3s[11030]: E0724 09:17:09.786305 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:17:11 bluth k3s[11030]: E0724 09:17:11.770708 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:17:13 bluth k3s[11030]: E0724 09:17:13.756981 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:17:15 bluth k3s[11030]: E0724 09:17:15.762344 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:17:17 bluth k3s[11030]: E0724 09:17:17.761557 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:17:19 bluth k3s[11030]: E0724 09:17:19.757084 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:17:21 bluth k3s[11030]: E0724 09:17:21.762465 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:17:23 bluth k3s[11030]: E0724 09:17:23.770792 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:17:25 bluth k3s[11030]: E0724 09:17:25.753214 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:17:27 bluth k3s[11030]: E0724 09:17:27.763534 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:17:29 bluth k3s[11030]: E0724 09:17:29.754362 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:17:31 bluth k3s[11030]: E0724 09:17:31.762731 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:17:33 bluth k3s[11030]: E0724 09:17:33.757012 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:17:35 bluth k3s[11030]: E0724 09:17:35.807235 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:17:37 bluth k3s[11030]: E0724 09:17:37.756869 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:17:39 bluth k3s[11030]: E0724 09:17:39.764243 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:17:40 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T09:17:40.498+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 09:17:41 bluth k3s[11030]: E0724 09:17:41.762407 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:17:43 bluth k3s[11030]: E0724 09:17:43.753397 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:17:45 bluth k3s[11030]: E0724 09:17:45.771244 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:17:47 bluth k3s[11030]: E0724 09:17:47.757041 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:17:49 bluth k3s[11030]: E0724 09:17:49.761211 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:17:51 bluth k3s[11030]: E0724 09:17:51.761233 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:17:53 bluth k3s[11030]: E0724 09:17:53.763737 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:17:55 bluth k3s[11030]: E0724 09:17:55.804277 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:17:57 bluth k3s[11030]: E0724 09:17:57.769135 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:17:59 bluth k3s[11030]: E0724 09:17:59.757017 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:18:01 bluth k3s[11030]: E0724 09:18:01.770566 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:18:03 bluth k3s[11030]: E0724 09:18:03.752977 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:18:05 bluth k3s[11030]: E0724 09:18:05.804735 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:18:06 bluth k3s[11030]: I0724 09:18:06.223534 11030 trace.go:205] Trace[1201943826]: "GuaranteedUpdate etcd3" type:*coordination.Lease (24-Jul-2022 09:18:05.123) (total time: 1100ms): Jul 24 09:18:06 bluth k3s[11030]: Trace[1201943826]: ---"Transaction committed" 1099ms (09:18:06.223) Jul 24 09:18:06 bluth k3s[11030]: Trace[1201943826]: [1.100044032s] [1.100044032s] END Jul 24 09:18:06 bluth k3s[11030]: I0724 09:18:06.223782 11030 trace.go:205] Trace[1070910156]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/external-resizer-zfs-csi-openebs-io,user-agent:csi-resizer/v0.0.0 (linux/amd64) kubernetes/$Format,audit-id:09fb3904-61d2-4d86-91e8-88568e7895f3,client:172.16.0.211,accept:application/json, */*,protocol:HTTP/2.0 (24-Jul-2022 09:18:05.123) (total time: 1100ms): Jul 24 09:18:06 bluth k3s[11030]: Trace[1070910156]: ---"Object stored in database" 1100ms (09:18:06.223) Jul 24 09:18:06 bluth k3s[11030]: Trace[1070910156]: [1.100409555s] [1.100409555s] END Jul 24 09:18:07 bluth k3s[11030]: E0724 09:18:07.770490 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:18:09 bluth k3s[11030]: I0724 09:18:09.426240 11030 trace.go:205] Trace[1643726040]: "GuaranteedUpdate etcd3" type:*coordination.Lease (24-Jul-2022 09:18:08.370) (total time: 1055ms): Jul 24 09:18:09 bluth k3s[11030]: Trace[1643726040]: ---"Transaction committed" 1055ms (09:18:09.426) Jul 24 09:18:09 bluth k3s[11030]: Trace[1643726040]: [1.05564791s] [1.05564791s] END Jul 24 09:18:09 bluth k3s[11030]: I0724 09:18:09.426546 11030 trace.go:205] Trace[1349893722]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/zfs-csi-openebs-io,user-agent:csi-provisioner/v0.0.0 (linux/amd64) kubernetes/$Format,audit-id:b7825b09-b2b7-4600-b649-cfa07003149e,client:172.16.0.211,accept:application/json, */*,protocol:HTTP/2.0 (24-Jul-2022 09:18:08.370) (total time: 1056ms): Jul 24 09:18:09 bluth k3s[11030]: Trace[1349893722]: ---"Object stored in database" 1055ms (09:18:09.426) Jul 24 09:18:09 bluth k3s[11030]: Trace[1349893722]: [1.056081531s] [1.056081531s] END Jul 24 09:18:09 bluth k3s[11030]: I0724 09:18:09.504575 11030 trace.go:205] Trace[1862555713]: "GuaranteedUpdate etcd3" type:*coordination.Lease (24-Jul-2022 09:18:08.371) (total time: 1133ms): Jul 24 09:18:09 bluth k3s[11030]: Trace[1862555713]: ---"Transaction committed" 1132ms (09:18:09.504) Jul 24 09:18:09 bluth k3s[11030]: Trace[1862555713]: [1.133155275s] [1.133155275s] END Jul 24 09:18:09 bluth k3s[11030]: I0724 09:18:09.504805 11030 trace.go:205] Trace[369655394]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/external-snapshotter-leader-zfs-csi-openebs-io,user-agent:csi-snapshotter/v0.0.0 (linux/amd64) kubernetes/$Format,audit-id:24c4d268-c6e4-4068-b3b2-51e59ff775e8,client:172.16.0.211,accept:application/json, */*,protocol:HTTP/2.0 (24-Jul-2022 09:18:08.371) (total time: 1133ms): Jul 24 09:18:09 bluth k3s[11030]: Trace[369655394]: ---"Object stored in database" 1133ms (09:18:09.504) Jul 24 09:18:09 bluth k3s[11030]: Trace[369655394]: [1.133486715s] [1.133486715s] END Jul 24 09:18:09 bluth k3s[11030]: I0724 09:18:09.605012 11030 trace.go:205] Trace[1767024724]: "GuaranteedUpdate etcd3" type:*coordination.Lease (24-Jul-2022 09:18:08.371) (total time: 1233ms): Jul 24 09:18:09 bluth k3s[11030]: Trace[1767024724]: ---"Transaction committed" 1233ms (09:18:09.604) Jul 24 09:18:09 bluth k3s[11030]: Trace[1767024724]: [1.233858363s] [1.233858363s] END Jul 24 09:18:09 bluth k3s[11030]: I0724 09:18:09.605274 11030 trace.go:205] Trace[293585317]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/snapshot-controller-leader,user-agent:snapshot-controller/v0.0.0 (linux/amd64) kubernetes/$Format,audit-id:7ad78aa9-27d9-43f5-9769-2b2f709e9a40,client:172.16.0.211,accept:application/json, */*,protocol:HTTP/2.0 (24-Jul-2022 09:18:08.370) (total time: 1234ms): Jul 24 09:18:09 bluth k3s[11030]: Trace[293585317]: ---"Object stored in database" 1233ms (09:18:09.605) Jul 24 09:18:09 bluth k3s[11030]: Trace[293585317]: [1.234233673s] [1.234233673s] END Jul 24 09:18:09 bluth k3s[11030]: E0724 09:18:09.756906 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:18:11 bluth k3s[11030]: E0724 09:18:11.762408 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:18:13 bluth k3s[11030]: E0724 09:18:13.761597 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:18:15 bluth k3s[11030]: E0724 09:18:15.754416 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:18:17 bluth k3s[11030]: E0724 09:18:17.762219 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:18:18 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T09:18:18.568+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 09:18:19 bluth k3s[11030]: E0724 09:18:19.756615 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:18:21 bluth k3s[11030]: E0724 09:18:21.780728 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:18:23 bluth k3s[11030]: E0724 09:18:23.786215 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:18:25 bluth k3s[11030]: E0724 09:18:25.752595 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:18:27 bluth k3s[11030]: E0724 09:18:27.760529 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:18:29 bluth k3s[11030]: E0724 09:18:29.758437 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:18:31 bluth k3s[11030]: E0724 09:18:31.752610 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:18:33 bluth k3s[11030]: E0724 09:18:33.761778 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:18:35 bluth k3s[11030]: E0724 09:18:35.806202 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:18:37 bluth k3s[11030]: E0724 09:18:37.764162 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:18:39 bluth k3s[11030]: E0724 09:18:39.761657 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:18:41 bluth k3s[11030]: E0724 09:18:41.753230 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:18:43 bluth k3s[11030]: E0724 09:18:43.762868 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:18:45 bluth k3s[11030]: E0724 09:18:45.756486 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:18:47 bluth k3s[11030]: E0724 09:18:47.761743 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:18:49 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T09:18:49.685+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 09:18:49 bluth k3s[11030]: E0724 09:18:49.756559 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:18:51 bluth k3s[11030]: E0724 09:18:51.790753 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:18:53 bluth k3s[11030]: E0724 09:18:53.758051 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:18:55 bluth k3s[11030]: E0724 09:18:55.762950 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:18:57 bluth k3s[11030]: E0724 09:18:57.756594 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:18:59 bluth k3s[11030]: E0724 09:18:59.761638 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:19:01 bluth k3s[11030]: E0724 09:19:01.786082 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:19:03 bluth k3s[11030]: E0724 09:19:03.770104 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:19:05 bluth k3s[11030]: E0724 09:19:05.789005 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:19:07 bluth k3s[11030]: E0724 09:19:07.758508 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:19:09 bluth k3s[11030]: E0724 09:19:09.762979 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:19:11 bluth k3s[11030]: E0724 09:19:11.782279 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:19:13 bluth k3s[11030]: E0724 09:19:13.756561 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:19:15 bluth k3s[11030]: E0724 09:19:15.762091 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:19:17 bluth k3s[11030]: E0724 09:19:17.761716 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:19:19 bluth k3s[11030]: E0724 09:19:19.756991 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:19:21 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T09:19:21.549+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 09:19:21 bluth k3s[11030]: E0724 09:19:21.762074 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:19:23 bluth k3s[11030]: E0724 09:19:23.756583 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:19:25 bluth k3s[11030]: E0724 09:19:25.758924 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:19:27 bluth k3s[11030]: E0724 09:19:27.761772 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:19:29 bluth k3s[11030]: E0724 09:19:29.810883 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:19:31 bluth k3s[11030]: E0724 09:19:31.754467 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:19:33 bluth k3s[11030]: E0724 09:19:33.758614 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:19:35 bluth k3s[11030]: E0724 09:19:35.752612 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:19:37 bluth k3s[11030]: E0724 09:19:37.762615 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:19:39 bluth k3s[11030]: E0724 09:19:39.756550 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:19:41 bluth k3s[11030]: E0724 09:19:41.762816 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:19:43 bluth k3s[11030]: E0724 09:19:43.758727 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:19:45 bluth k3s[11030]: E0724 09:19:45.762832 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:19:45 bluth k3s[11030]: I0724 09:19:45.993792 11030 trace.go:205] Trace[1981747010]: "GuaranteedUpdate etcd3" type:*coordination.Lease (24-Jul-2022 09:19:44.841) (total time: 1152ms): Jul 24 09:19:45 bluth k3s[11030]: Trace[1981747010]: ---"Transaction committed" 1151ms (09:19:45.993) Jul 24 09:19:45 bluth k3s[11030]: Trace[1981747010]: [1.152737173s] [1.152737173s] END Jul 24 09:19:45 bluth k3s[11030]: I0724 09:19:45.993868 11030 trace.go:205] Trace[232822335]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/snapshot-controller-leader,user-agent:snapshot-controller/v0.0.0 (linux/amd64) kubernetes/$Format,audit-id:dd60a273-f362-415c-aa2d-964858b27ae5,client:172.16.0.211,accept:application/json, */*,protocol:HTTP/2.0 (24-Jul-2022 09:19:44.840) (total time: 1153ms): Jul 24 09:19:45 bluth k3s[11030]: Trace[232822335]: ---"Object stored in database" 1152ms (09:19:45.993) Jul 24 09:19:45 bluth k3s[11030]: Trace[232822335]: [1.153268059s] [1.153268059s] END Jul 24 09:19:47 bluth k3s[11030]: E0724 09:19:47.757738 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:19:49 bluth k3s[11030]: E0724 09:19:49.782602 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:19:51 bluth k3s[11030]: E0724 09:19:51.753133 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:19:53 bluth k3s[11030]: E0724 09:19:53.762678 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:19:55 bluth k3s[11030]: E0724 09:19:55.757692 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:19:57 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T09:19:57.132+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 09:19:57 bluth k3s[11030]: E0724 09:19:57.761675 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:19:59 bluth k3s[11030]: E0724 09:19:59.768559 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:20:01 bluth k3s[11030]: E0724 09:20:01.766915 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:20:03 bluth k3s[11030]: E0724 09:20:03.761603 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:20:05 bluth k3s[11030]: E0724 09:20:05.805988 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:20:07 bluth k3s[11030]: E0724 09:20:07.758822 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:20:09 bluth k3s[11030]: E0724 09:20:09.762701 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:20:11 bluth k3s[11030]: E0724 09:20:11.756333 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:20:13 bluth k3s[11030]: E0724 09:20:13.761811 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:20:15 bluth k3s[11030]: E0724 09:20:15.756628 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:20:17 bluth k3s[11030]: E0724 09:20:17.758481 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:20:19 bluth k3s[11030]: E0724 09:20:19.769002 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:20:21 bluth k3s[11030]: E0724 09:20:21.757285 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:20:23 bluth k3s[11030]: E0724 09:20:23.762912 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:20:25 bluth k3s[11030]: E0724 09:20:25.752887 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:20:27 bluth k3s[11030]: E0724 09:20:27.761726 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:20:29 bluth k3s[11030]: E0724 09:20:29.769108 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:20:31 bluth k3s[11030]: E0724 09:20:31.769953 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:20:33 bluth k3s[11030]: E0724 09:20:33.753691 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:20:35 bluth k3s[11030]: E0724 09:20:35.804195 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:20:37 bluth k3s[11030]: E0724 09:20:37.756620 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:20:39 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T09:20:39.730+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 09:20:39 bluth k3s[11030]: E0724 09:20:39.762604 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:20:41 bluth k3s[11030]: E0724 09:20:41.756313 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:20:43 bluth k3s[11030]: E0724 09:20:43.761987 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:20:45 bluth k3s[11030]: E0724 09:20:45.754806 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:20:47 bluth k3s[11030]: E0724 09:20:47.762804 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:20:49 bluth k3s[11030]: E0724 09:20:49.756641 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:20:51 bluth k3s[11030]: E0724 09:20:51.762988 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:20:53 bluth k3s[11030]: E0724 09:20:53.758870 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:20:55 bluth k3s[11030]: E0724 09:20:55.761639 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:20:57 bluth k3s[11030]: E0724 09:20:57.770204 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:20:59 bluth k3s[11030]: E0724 09:20:59.756748 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:21:01 bluth k3s[11030]: E0724 09:21:01.761166 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:21:03 bluth k3s[11030]: E0724 09:21:03.762819 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:21:05 bluth k3s[11030]: E0724 09:21:05.804997 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:21:07 bluth k3s[11030]: E0724 09:21:07.756497 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:21:09 bluth k3s[11030]: E0724 09:21:09.761755 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:21:11 bluth k3s[11030]: E0724 09:21:11.761857 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:21:13 bluth k3s[11030]: E0724 09:21:13.769935 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:21:15 bluth k3s[11030]: E0724 09:21:15.756590 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:21:15 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T09:21:15.843+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 09:21:17 bluth k3s[11030]: E0724 09:21:17.762856 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:21:19 bluth k3s[11030]: E0724 09:21:19.752216 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:21:21 bluth k3s[11030]: E0724 09:21:21.761966 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:21:23 bluth k3s[11030]: E0724 09:21:23.805641 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:21:25 bluth k3s[11030]: E0724 09:21:25.753760 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:21:27 bluth k3s[11030]: E0724 09:21:27.761927 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:21:29 bluth k3s[11030]: E0724 09:21:29.804413 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:21:31 bluth k3s[11030]: E0724 09:21:31.757546 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:21:33 bluth k3s[11030]: E0724 09:21:33.761725 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:21:35 bluth k3s[11030]: E0724 09:21:35.805969 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:21:37 bluth k3s[11030]: E0724 09:21:37.752339 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:21:39 bluth k3s[11030]: E0724 09:21:39.762944 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:21:41 bluth k3s[11030]: E0724 09:21:41.757537 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:21:43 bluth k3s[11030]: E0724 09:21:43.770211 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:21:45 bluth k3s[11030]: E0724 09:21:45.757582 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:21:47 bluth k3s[11030]: E0724 09:21:47.761806 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:21:49 bluth k3s[11030]: E0724 09:21:49.771586 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:21:51 bluth k3s[11030]: E0724 09:21:51.757567 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:21:53 bluth k3s[11030]: E0724 09:21:53.757594 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:21:55 bluth k3s[11030]: E0724 09:21:55.806720 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:21:57 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T09:21:57.622+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 09:21:57 bluth k3s[11030]: E0724 09:21:57.757620 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:21:59 bluth k3s[11030]: E0724 09:21:59.761592 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:22:01 bluth k3s[11030]: E0724 09:22:01.762831 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:22:03 bluth k3s[11030]: E0724 09:22:03.770379 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:22:05 bluth k3s[11030]: E0724 09:22:05.783694 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:22:07 bluth k3s[11030]: E0724 09:22:07.761676 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:22:09 bluth k3s[11030]: E0724 09:22:09.761739 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:22:11 bluth k3s[11030]: E0724 09:22:11.769972 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:22:13 bluth k3s[11030]: E0724 09:22:13.756736 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:22:15 bluth k3s[11030]: E0724 09:22:15.770090 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:22:17 bluth k3s[11030]: E0724 09:22:17.756586 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:22:19 bluth k3s[11030]: E0724 09:22:19.756968 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:22:21 bluth k3s[11030]: E0724 09:22:21.770290 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:22:23 bluth k3s[11030]: E0724 09:22:23.756634 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:22:25 bluth k3s[11030]: E0724 09:22:25.757682 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:22:27 bluth k3s[11030]: E0724 09:22:27.761652 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:22:29 bluth k3s[11030]: E0724 09:22:29.762782 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:22:31 bluth k3s[11030]: E0724 09:22:31.756694 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:22:33 bluth k3s[11030]: E0724 09:22:33.770058 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:22:34 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T09:22:34.712+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 09:22:35 bluth k3s[11030]: E0724 09:22:35.759592 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:22:37 bluth k3s[11030]: E0724 09:22:37.762715 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:22:39 bluth k3s[11030]: E0724 09:22:39.756581 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:22:41 bluth k3s[11030]: E0724 09:22:41.762976 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:22:43 bluth k3s[11030]: E0724 09:22:43.756631 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:22:45 bluth k3s[11030]: E0724 09:22:45.785943 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:22:47 bluth k3s[11030]: E0724 09:22:47.770254 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:22:49 bluth k3s[11030]: E0724 09:22:49.786365 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:22:51 bluth k3s[11030]: E0724 09:22:51.762982 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:22:53 bluth k3s[11030]: E0724 09:22:53.781586 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:22:55 bluth k3s[11030]: E0724 09:22:55.767614 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:22:57 bluth k3s[11030]: E0724 09:22:57.761902 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:22:59 bluth k3s[11030]: E0724 09:22:59.769218 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:23:01 bluth k3s[11030]: E0724 09:23:01.756769 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:23:03 bluth k3s[11030]: E0724 09:23:03.800552 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:23:05 bluth k3s[11030]: E0724 09:23:05.765823 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:23:07 bluth k3s[11030]: E0724 09:23:07.764339 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:23:09 bluth k3s[11030]: E0724 09:23:09.806385 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:23:11 bluth k3s[11030]: E0724 09:23:11.770400 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:23:13 bluth k3s[11030]: E0724 09:23:13.756674 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:23:15 bluth k3s[11030]: E0724 09:23:15.779431 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:23:17 bluth k3s[11030]: E0724 09:23:17.760893 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:23:18 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T09:23:18.220+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 09:23:19 bluth k3s[11030]: E0724 09:23:19.770475 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:23:21 bluth k3s[11030]: E0724 09:23:21.756894 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:23:23 bluth k3s[11030]: E0724 09:23:23.764666 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:23:25 bluth k3s[11030]: E0724 09:23:25.752811 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:23:27 bluth k3s[11030]: E0724 09:23:27.770092 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:23:29 bluth k3s[11030]: E0724 09:23:29.770649 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:23:31 bluth k3s[11030]: E0724 09:23:31.784150 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:23:33 bluth k3s[11030]: E0724 09:23:33.758208 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:23:35 bluth k3s[11030]: E0724 09:23:35.805990 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:23:37 bluth k3s[11030]: E0724 09:23:37.764329 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:23:39 bluth k3s[11030]: E0724 09:23:39.761658 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:23:41 bluth k3s[11030]: E0724 09:23:41.768928 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:23:43 bluth k3s[11030]: E0724 09:23:43.759092 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:23:45 bluth k3s[11030]: E0724 09:23:45.761689 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:23:47 bluth k3s[11030]: E0724 09:23:47.769957 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:23:49 bluth k3s[11030]: E0724 09:23:49.756846 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:23:51 bluth k3s[11030]: E0724 09:23:51.758399 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:23:53 bluth k3s[11030]: E0724 09:23:53.752259 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:23:55 bluth k3s[11030]: E0724 09:23:55.785104 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:23:57 bluth k3s[11030]: E0724 09:23:57.757026 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:23:59 bluth k3s[11030]: E0724 09:23:59.761770 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:24:00 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T09:24:00.698+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 09:24:01 bluth k3s[11030]: E0724 09:24:01.761957 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:24:03 bluth k3s[11030]: E0724 09:24:03.756941 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:24:05 bluth k3s[11030]: E0724 09:24:05.805145 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:24:07 bluth k3s[11030]: E0724 09:24:07.786215 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:24:09 bluth k3s[11030]: E0724 09:24:09.763945 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:24:11 bluth k3s[11030]: E0724 09:24:11.756750 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:24:13 bluth k3s[11030]: E0724 09:24:13.769908 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:24:15 bluth k3s[11030]: E0724 09:24:15.756741 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:24:17 bluth k3s[11030]: E0724 09:24:17.762333 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:24:19 bluth k3s[11030]: E0724 09:24:19.755911 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:24:21 bluth k3s[11030]: E0724 09:24:21.757311 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:24:23 bluth k3s[11030]: E0724 09:24:23.770110 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:24:25 bluth k3s[11030]: E0724 09:24:25.753084 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:24:27 bluth k3s[11030]: E0724 09:24:27.761965 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:24:29 bluth k3s[11030]: E0724 09:24:29.761978 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:24:31 bluth k3s[11030]: E0724 09:24:31.759198 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:24:33 bluth k3s[11030]: E0724 09:24:33.762750 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:24:35 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T09:24:35.212+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 09:24:35 bluth k3s[11030]: E0724 09:24:35.806149 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:24:37 bluth k3s[11030]: E0724 09:24:37.756795 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:24:39 bluth k3s[11030]: E0724 09:24:39.762048 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:24:41 bluth k3s[11030]: E0724 09:24:41.756869 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:24:43 bluth k3s[11030]: E0724 09:24:43.773454 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:24:45 bluth k3s[11030]: E0724 09:24:45.758100 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:24:47 bluth k3s[11030]: E0724 09:24:47.762078 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:24:49 bluth k3s[11030]: E0724 09:24:49.756941 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:24:51 bluth k3s[11030]: E0724 09:24:51.761910 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:24:53 bluth k3s[11030]: E0724 09:24:53.756377 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:24:55 bluth k3s[11030]: E0724 09:24:55.762043 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:24:56 bluth k3s[11030]: I0724 09:24:56.385770 11030 trace.go:205] Trace[151046551]: "GuaranteedUpdate etcd3" type:*coordination.Lease (24-Jul-2022 09:24:55.830) (total time: 555ms): Jul 24 09:24:56 bluth k3s[11030]: Trace[151046551]: ---"Transaction committed" 555ms (09:24:56.385) Jul 24 09:24:56 bluth k3s[11030]: Trace[151046551]: [555.653658ms] [555.653658ms] END Jul 24 09:24:56 bluth k3s[11030]: I0724 09:24:56.386019 11030 trace.go:205] Trace[1843357089]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/external-snapshotter-leader-zfs-csi-openebs-io,user-agent:csi-snapshotter/v0.0.0 (linux/amd64) kubernetes/$Format,audit-id:1b41bb4a-9d36-49c6-8d29-8ab7edab9412,client:172.16.0.211,accept:application/json, */*,protocol:HTTP/2.0 (24-Jul-2022 09:24:55.829) (total time: 556ms): Jul 24 09:24:56 bluth k3s[11030]: Trace[1843357089]: ---"Object stored in database" 555ms (09:24:56.385) Jul 24 09:24:56 bluth k3s[11030]: Trace[1843357089]: [556.025858ms] [556.025858ms] END Jul 24 09:24:57 bluth k3s[11030]: E0724 09:24:57.762286 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:24:59 bluth k3s[11030]: E0724 09:24:59.786283 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:25:01 bluth k3s[11030]: E0724 09:25:01.770247 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:25:03 bluth k3s[11030]: E0724 09:25:03.765520 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:25:05 bluth k3s[11030]: E0724 09:25:05.805662 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:25:07 bluth k3s[11030]: E0724 09:25:07.755830 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:25:09 bluth k3s[11030]: E0724 09:25:09.762038 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:25:10 bluth k3s[11030]: {"level":"warn","ts":"2022-07-24T09:25:10.871+0100","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0014defc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Jul 24 09:25:11 bluth k3s[11030]: E0724 09:25:11.756433 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:25:13 bluth k3s[11030]: E0724 09:25:13.762105 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:25:15 bluth k3s[11030]: E0724 09:25:15.805713 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:25:17 bluth k3s[11030]: E0724 09:25:17.757036 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:25:19 bluth k3s[11030]: E0724 09:25:19.770282 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:25:21 bluth k3s[11030]: E0724 09:25:21.757270 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:25:23 bluth k3s[11030]: E0724 09:25:23.770200 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:25:25 bluth k3s[11030]: E0724 09:25:25.753050 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:25:27 bluth k3s[11030]: E0724 09:25:27.762051 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8 Jul 24 09:25:29 bluth k3s[11030]: E0724 09:25:29.759678 11030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"05a7efb7-a3a8-4d6d-8852-d5fbccf4fc2b\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=8