
-------------
1) K3s logs
-------------

Dec 22 17:43:06 truenas k3s[16911]: E1222 17:43:06.721599   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName:26601002-2b7f-44d7-b247-12092d0a23c8 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.221575541 +0100 CET m=+45.008381342 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "26601002-2b7f-44d7-b247-12092d0a23c8" (UID: "26601002-2b7f-44d7-b247-12092d0a23c8") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:06 truenas k3s[16911]: I1222 17:43:06.721749   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\") pod \"10d85e4d-b0d8-4823-b20d-cda28e641673\" (UID: \"10d85e4d-b0d8-4823-b20d-cda28e641673\") "
Dec 22 17:43:06 truenas k3s[16911]: E1222 17:43:06.721848   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName:10d85e4d-b0d8-4823-b20d-cda28e641673 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.221830811 +0100 CET m=+45.008636612 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "10d85e4d-b0d8-4823-b20d-cda28e641673" (UID: "10d85e4d-b0d8-4823-b20d-cda28e641673") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:06 truenas k3s[16911]: I1222 17:43:06.721959   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:06 truenas k3s[16911]: I1222 17:43:06.722014   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:06 truenas k3s[16911]: E1222 17:43:06.722048   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.222033723 +0100 CET m=+45.008839524 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:06 truenas k3s[16911]: I1222 17:43:06.722066   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-80edd197-58be-4fb7-8420-25e736120a31\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:06 truenas k3s[16911]: E1222 17:43:06.722076   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.222065212 +0100 CET m=+45.008871003 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:06 truenas k3s[16911]: E1222 17:43:06.722156   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.22214362 +0100 CET m=+45.008949411 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:06 truenas k3s[16911]: E1222 17:43:06.722335   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.222328618 +0100 CET m=+45.009134409 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:06 truenas k3s[16911]: E1222 17:43:06.722355   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.222350479 +0100 CET m=+45.009156260 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "nextcloud-nginx-b674d44dd-vwfct" (UID: "faa24a36-35eb-4764-8ab1-d2630b99b684") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:06 truenas k3s[16911]: E1222 17:43:06.722366   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.2223617 +0100 CET m=+45.009167491 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:06 truenas k3s[16911]: E1222 17:43:06.722372   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.222369134 +0100 CET m=+45.009174925 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:06 truenas k3s[16911]: E1222 17:43:06.722413   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.222402416 +0100 CET m=+45.009208207 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:06 truenas k3s[16911]: E1222 17:43:06.722428   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.222420821 +0100 CET m=+45.009226612 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "nextcloud-nginx-b674d44dd-vwfct" (UID: "faa24a36-35eb-4764-8ab1-d2630b99b684") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:06 truenas k3s[16911]: I1222 17:43:06.822637   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\") pod \"26601002-2b7f-44d7-b247-12092d0a23c8\" (UID: \"26601002-2b7f-44d7-b247-12092d0a23c8\") "
Dec 22 17:43:06 truenas k3s[16911]: E1222 17:43:06.822743   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName:26601002-2b7f-44d7-b247-12092d0a23c8 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.322715062 +0100 CET m=+45.109520843 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "26601002-2b7f-44d7-b247-12092d0a23c8" (UID: "26601002-2b7f-44d7-b247-12092d0a23c8") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:06 truenas k3s[16911]: I1222 17:43:06.822899   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\") pod \"10d85e4d-b0d8-4823-b20d-cda28e641673\" (UID: \"10d85e4d-b0d8-4823-b20d-cda28e641673\") "
Dec 22 17:43:06 truenas k3s[16911]: E1222 17:43:06.822986   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName:10d85e4d-b0d8-4823-b20d-cda28e641673 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.322972998 +0100 CET m=+45.109778789 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "10d85e4d-b0d8-4823-b20d-cda28e641673" (UID: "10d85e4d-b0d8-4823-b20d-cda28e641673") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:06 truenas k3s[16911]: I1222 17:43:06.823003   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:06 truenas k3s[16911]: I1222 17:43:06.823042   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:06 truenas k3s[16911]: I1222 17:43:06.823078   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-80edd197-58be-4fb7-8420-25e736120a31\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:06 truenas k3s[16911]: E1222 17:43:06.823084   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.323074158 +0100 CET m=+45.109879949 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:06 truenas k3s[16911]: E1222 17:43:06.823126   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.323104795 +0100 CET m=+45.109910586 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:06 truenas k3s[16911]: E1222 17:43:06.823211   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.323201758 +0100 CET m=+45.110007549 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:06 truenas k3s[16911]: I1222 17:43:06.823212   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-eaf1e5ab-9a06-441a-b138-70749798101f\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:06 truenas k3s[16911]: E1222 17:43:06.823311   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.323294382 +0100 CET m=+45.110100173 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:06 truenas k3s[16911]: E1222 17:43:06.823567   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.323558719 +0100 CET m=+45.110364520 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:06 truenas k3s[16911]: E1222 17:43:06.823588   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.323582845 +0100 CET m=+45.110388636 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "nextcloud-nginx-b674d44dd-vwfct" (UID: "faa24a36-35eb-4764-8ab1-d2630b99b684") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:06 truenas k3s[16911]: E1222 17:43:06.823595   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.323591441 +0100 CET m=+45.110397242 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "nextcloud-nginx-b674d44dd-vwfct" (UID: "faa24a36-35eb-4764-8ab1-d2630b99b684") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:06 truenas k3s[16911]: E1222 17:43:06.823616   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.323608202 +0100 CET m=+45.110413993 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:06 truenas k3s[16911]: E1222 17:43:06.823663   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.323653477 +0100 CET m=+45.110459258 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:06 truenas k3s[16911]: E1222 17:43:06.823721   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.323715264 +0100 CET m=+45.110521055 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:06 truenas k3s[16911]: I1222 17:43:06.924798   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-eaf1e5ab-9a06-441a-b138-70749798101f\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:06 truenas k3s[16911]: E1222 17:43:06.924963   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.424909598 +0100 CET m=+45.211715399 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:06 truenas k3s[16911]: I1222 17:43:06.925129   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\") pod \"26601002-2b7f-44d7-b247-12092d0a23c8\" (UID: \"26601002-2b7f-44d7-b247-12092d0a23c8\") "
Dec 22 17:43:06 truenas k3s[16911]: E1222 17:43:06.925241   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName:26601002-2b7f-44d7-b247-12092d0a23c8 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.425218369 +0100 CET m=+45.212024170 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "26601002-2b7f-44d7-b247-12092d0a23c8" (UID: "26601002-2b7f-44d7-b247-12092d0a23c8") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:06 truenas k3s[16911]: I1222 17:43:06.925272   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\") pod \"10d85e4d-b0d8-4823-b20d-cda28e641673\" (UID: \"10d85e4d-b0d8-4823-b20d-cda28e641673\") "
Dec 22 17:43:06 truenas k3s[16911]: E1222 17:43:06.925358   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName:10d85e4d-b0d8-4823-b20d-cda28e641673 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.425342893 +0100 CET m=+45.212148694 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "10d85e4d-b0d8-4823-b20d-cda28e641673" (UID: "10d85e4d-b0d8-4823-b20d-cda28e641673") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:06 truenas k3s[16911]: I1222 17:43:06.925545   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:06 truenas k3s[16911]: I1222 17:43:06.925588   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:06 truenas k3s[16911]: E1222 17:43:06.925616   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.425605376 +0100 CET m=+45.212411167 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:06 truenas k3s[16911]: I1222 17:43:06.925631   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-80edd197-58be-4fb7-8420-25e736120a31\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:06 truenas k3s[16911]: E1222 17:43:06.925638   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.4256287 +0100 CET m=+45.212434491 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:06 truenas k3s[16911]: E1222 17:43:06.925720   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.425712668 +0100 CET m=+45.212518459 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:06 truenas k3s[16911]: E1222 17:43:06.926000   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.425992464 +0100 CET m=+45.212798255 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:06 truenas k3s[16911]: E1222 17:43:06.926014   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.426007913 +0100 CET m=+45.212813704 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:06 truenas k3s[16911]: E1222 17:43:06.926023   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.426018283 +0100 CET m=+45.212824074 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "nextcloud-nginx-b674d44dd-vwfct" (UID: "faa24a36-35eb-4764-8ab1-d2630b99b684") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:06 truenas k3s[16911]: E1222 17:43:06.926033   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.426027701 +0100 CET m=+45.212833492 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "nextcloud-nginx-b674d44dd-vwfct" (UID: "faa24a36-35eb-4764-8ab1-d2630b99b684") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:06 truenas k3s[16911]: E1222 17:43:06.926069   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.426063197 +0100 CET m=+45.212868988 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:06 truenas k3s[16911]: E1222 17:43:06.926080   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.426075681 +0100 CET m=+45.212881482 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.026974   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.027070   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-80edd197-58be-4fb7-8420-25e736120a31\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.027145   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.52709721 +0100 CET m=+45.313903011 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.027196   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.527167572 +0100 CET m=+45.313973373 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.027289   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-eaf1e5ab-9a06-441a-b138-70749798101f\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.027375   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.527359093 +0100 CET m=+45.314164894 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.027421   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\") pod \"26601002-2b7f-44d7-b247-12092d0a23c8\" (UID: \"26601002-2b7f-44d7-b247-12092d0a23c8\") "
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.027499   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName:26601002-2b7f-44d7-b247-12092d0a23c8 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.527485841 +0100 CET m=+45.314291652 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "26601002-2b7f-44d7-b247-12092d0a23c8" (UID: "26601002-2b7f-44d7-b247-12092d0a23c8") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.027711   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\") pod \"10d85e4d-b0d8-4823-b20d-cda28e641673\" (UID: \"10d85e4d-b0d8-4823-b20d-cda28e641673\") "
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.027785   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName:10d85e4d-b0d8-4823-b20d-cda28e641673 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.527776498 +0100 CET m=+45.314582289 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "10d85e4d-b0d8-4823-b20d-cda28e641673" (UID: "10d85e4d-b0d8-4823-b20d-cda28e641673") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.027906   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.027983   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.527974129 +0100 CET m=+45.314779930 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.028274   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.528268202 +0100 CET m=+45.315073993 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.028285   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.528281077 +0100 CET m=+45.315086878 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.028294   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.528289102 +0100 CET m=+45.315094903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.028301   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.528297397 +0100 CET m=+45.315103188 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "nextcloud-preview-cron-28387720-dq7tq" (UID: "f625e3e8-e7a8-4439-8d8c-6f10b1c7978a") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.028309   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.52830419 +0100 CET m=+45.315109991 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "nextcloud-nginx-b674d44dd-vwfct" (UID: "faa24a36-35eb-4764-8ab1-d2630b99b684") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.028347   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.528339456 +0100 CET m=+45.315145247 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.078785   16911 event.go:294] "Event occurred" object="ix-cloudflareddns/cloudflareddns-56c9c85bb8" fieldPath="" kind="ReplicaSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulCreate" message="Created pod: cloudflareddns-56c9c85bb8-h822b"
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.083911   16911 topology_manager.go:210] "Topology Admit Handler"
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.128980   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-eaf1e5ab-9a06-441a-b138-70749798101f\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.129113   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.629084035 +0100 CET m=+45.415889826 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.129180   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\") pod \"26601002-2b7f-44d7-b247-12092d0a23c8\" (UID: \"26601002-2b7f-44d7-b247-12092d0a23c8\") "
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.129283   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName:26601002-2b7f-44d7-b247-12092d0a23c8 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.629269043 +0100 CET m=+45.416074834 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "26601002-2b7f-44d7-b247-12092d0a23c8" (UID: "26601002-2b7f-44d7-b247-12092d0a23c8") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.129360   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\") pod \"10d85e4d-b0d8-4823-b20d-cda28e641673\" (UID: \"10d85e4d-b0d8-4823-b20d-cda28e641673\") "
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.129443   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName:10d85e4d-b0d8-4823-b20d-cda28e641673 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.629430827 +0100 CET m=+45.416236618 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "10d85e4d-b0d8-4823-b20d-cda28e641673" (UID: "10d85e4d-b0d8-4823-b20d-cda28e641673") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.129505   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.129549   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.629535103 +0100 CET m=+45.416340894 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.129573   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.129639   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-80edd197-58be-4fb7-8420-25e736120a31\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.129652   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.629635853 +0100 CET m=+45.416441644 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.129726   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.629711455 +0100 CET m=+45.416517246 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.129998   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.629992964 +0100 CET m=+45.416798755 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.130010   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.630005107 +0100 CET m=+45.416810898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "nextcloud-preview-cron-28387720-dq7tq" (UID: "f625e3e8-e7a8-4439-8d8c-6f10b1c7978a") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.130018   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.630014064 +0100 CET m=+45.416819855 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.130025   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.630021698 +0100 CET m=+45.416827489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.130077   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.63007065 +0100 CET m=+45.416876441 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.130086   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.630082042 +0100 CET m=+45.416887833 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "nextcloud-nextcloud-cron-28387640-kvwwv" (UID: "485f2017-fcab-46d8-a0f8-71bf110c1836") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.230587   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\") pod \"10d85e4d-b0d8-4823-b20d-cda28e641673\" (UID: \"10d85e4d-b0d8-4823-b20d-cda28e641673\") "
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.230735   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName:10d85e4d-b0d8-4823-b20d-cda28e641673 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.73069342 +0100 CET m=+45.517499211 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "10d85e4d-b0d8-4823-b20d-cda28e641673" (UID: "10d85e4d-b0d8-4823-b20d-cda28e641673") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.230763   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.230846   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.230855   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.730839084 +0100 CET m=+45.517644875 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.230922   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-80edd197-58be-4fb7-8420-25e736120a31\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.230961   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.730937118 +0100 CET m=+45.517742899 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.231005   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.73099164 +0100 CET m=+45.517797432 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.231058   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-eaf1e5ab-9a06-441a-b138-70749798101f\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.231144   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.731130221 +0100 CET m=+45.517936012 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.231297   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\") pod \"26601002-2b7f-44d7-b247-12092d0a23c8\" (UID: \"26601002-2b7f-44d7-b247-12092d0a23c8\") "
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.231393   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName:26601002-2b7f-44d7-b247-12092d0a23c8 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.731376204 +0100 CET m=+45.518181995 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "26601002-2b7f-44d7-b247-12092d0a23c8" (UID: "26601002-2b7f-44d7-b247-12092d0a23c8") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.231682   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.731672571 +0100 CET m=+45.518478362 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "nextcloud-preview-cron-28387720-dq7tq" (UID: "f625e3e8-e7a8-4439-8d8c-6f10b1c7978a") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.231694   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.731689944 +0100 CET m=+45.518495735 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.231705   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.731699632 +0100 CET m=+45.518505423 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.231726   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.731718427 +0100 CET m=+45.518524218 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.231740   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.731732885 +0100 CET m=+45.518538676 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "nextcloud-nginx-b674d44dd-vwfct" (UID: "faa24a36-35eb-4764-8ab1-d2630b99b684") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.231769   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.73175729 +0100 CET m=+45.518563092 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.332379   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\") pod \"10d85e4d-b0d8-4823-b20d-cda28e641673\" (UID: \"10d85e4d-b0d8-4823-b20d-cda28e641673\") "
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.332486   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName:10d85e4d-b0d8-4823-b20d-cda28e641673 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.832462445 +0100 CET m=+45.619268226 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "10d85e4d-b0d8-4823-b20d-cda28e641673" (UID: "10d85e4d-b0d8-4823-b20d-cda28e641673") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.332502   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.332543   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.332574   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.832565278 +0100 CET m=+45.619371069 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.332588   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-80edd197-58be-4fb7-8420-25e736120a31\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.332607   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.832594253 +0100 CET m=+45.619400044 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.332638   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.832627225 +0100 CET m=+45.619433016 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.332689   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-eaf1e5ab-9a06-441a-b138-70749798101f\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.332731   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\") pod \"26601002-2b7f-44d7-b247-12092d0a23c8\" (UID: \"26601002-2b7f-44d7-b247-12092d0a23c8\") "
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.332767   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName:26601002-2b7f-44d7-b247-12092d0a23c8 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.83275796 +0100 CET m=+45.619563751 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "26601002-2b7f-44d7-b247-12092d0a23c8" (UID: "26601002-2b7f-44d7-b247-12092d0a23c8") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.332787   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.832771275 +0100 CET m=+45.619577077 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.333180   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.833173272 +0100 CET m=+45.619979073 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "nextcloud-nginx-b674d44dd-vwfct" (UID: "faa24a36-35eb-4764-8ab1-d2630b99b684") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.333193   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.83318845 +0100 CET m=+45.619994241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.333206   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.833197126 +0100 CET m=+45.620002927 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.333220   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.83321492 +0100 CET m=+45.620020711 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "nextcloud-nginx-b674d44dd-vwfct" (UID: "faa24a36-35eb-4764-8ab1-d2630b99b684") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.333230   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.83322556 +0100 CET m=+45.620031351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.333238   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.833234236 +0100 CET m=+45.620040037 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.390427   16911 server.go:144] "Got registration request from device plugin with resource" resourceName="amd.com/gpu"
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.433820   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-eaf1e5ab-9a06-441a-b138-70749798101f\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.433879   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\") pod \"26601002-2b7f-44d7-b247-12092d0a23c8\" (UID: \"26601002-2b7f-44d7-b247-12092d0a23c8\") "
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.433958   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.933920906 +0100 CET m=+45.720726697 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.434048   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName:26601002-2b7f-44d7-b247-12092d0a23c8 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.934033978 +0100 CET m=+45.720839779 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "26601002-2b7f-44d7-b247-12092d0a23c8" (UID: "26601002-2b7f-44d7-b247-12092d0a23c8") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.434222   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\") pod \"10d85e4d-b0d8-4823-b20d-cda28e641673\" (UID: \"10d85e4d-b0d8-4823-b20d-cda28e641673\") "
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.434306   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName:10d85e4d-b0d8-4823-b20d-cda28e641673 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.934290511 +0100 CET m=+45.721096312 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "10d85e4d-b0d8-4823-b20d-cda28e641673" (UID: "10d85e4d-b0d8-4823-b20d-cda28e641673") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.434389   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.434446   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.434464   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.934456653 +0100 CET m=+45.721262434 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.434523   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-80edd197-58be-4fb7-8420-25e736120a31\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.434552   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.934535692 +0100 CET m=+45.721341483 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.434600   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.934593491 +0100 CET m=+45.721399282 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.434823   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.934816901 +0100 CET m=+45.721622692 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.434838   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.934831158 +0100 CET m=+45.721636949 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.434856   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.934847588 +0100 CET m=+45.721653379 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "nextcloud-nginx-b674d44dd-vwfct" (UID: "faa24a36-35eb-4764-8ab1-d2630b99b684") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.434868   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.93486434 +0100 CET m=+45.721670131 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.434881   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.934873006 +0100 CET m=+45.721678797 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.434895   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:07.934890559 +0100 CET m=+45.721696350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "nextcloud-nginx-b674d44dd-vwfct" (UID: "faa24a36-35eb-4764-8ab1-d2630b99b684") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.535467   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\") pod \"26601002-2b7f-44d7-b247-12092d0a23c8\" (UID: \"26601002-2b7f-44d7-b247-12092d0a23c8\") "
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.535592   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName:26601002-2b7f-44d7-b247-12092d0a23c8 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.035560068 +0100 CET m=+45.822365869 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "26601002-2b7f-44d7-b247-12092d0a23c8" (UID: "26601002-2b7f-44d7-b247-12092d0a23c8") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.535675   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\") pod \"10d85e4d-b0d8-4823-b20d-cda28e641673\" (UID: \"10d85e4d-b0d8-4823-b20d-cda28e641673\") "
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.535754   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName:10d85e4d-b0d8-4823-b20d-cda28e641673 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.035742892 +0100 CET m=+45.822548683 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "10d85e4d-b0d8-4823-b20d-cda28e641673" (UID: "10d85e4d-b0d8-4823-b20d-cda28e641673") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.535773   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.535813   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.535848   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.035838842 +0100 CET m=+45.822644633 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.535861   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-80edd197-58be-4fb7-8420-25e736120a31\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.535864   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.035855153 +0100 CET m=+45.822660944 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.535934   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-eaf1e5ab-9a06-441a-b138-70749798101f\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.535948   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.035940674 +0100 CET m=+45.822746465 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.536044   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.036028639 +0100 CET m=+45.822834430 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.536187   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.036182008 +0100 CET m=+45.822987799 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.536196   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.036192778 +0100 CET m=+45.822998559 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "nextcloud-nginx-b674d44dd-vwfct" (UID: "faa24a36-35eb-4764-8ab1-d2630b99b684") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.536275   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.036264463 +0100 CET m=+45.823070264 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "nextcloud-nginx-b674d44dd-vwfct" (UID: "faa24a36-35eb-4764-8ab1-d2630b99b684") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.536289   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.03628444 +0100 CET m=+45.823090241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.536298   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.036293567 +0100 CET m=+45.823099358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.636878   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\") pod \"26601002-2b7f-44d7-b247-12092d0a23c8\" (UID: \"26601002-2b7f-44d7-b247-12092d0a23c8\") "
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.636960   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\") pod \"10d85e4d-b0d8-4823-b20d-cda28e641673\" (UID: \"10d85e4d-b0d8-4823-b20d-cda28e641673\") "
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.637006   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName:26601002-2b7f-44d7-b247-12092d0a23c8 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.136974717 +0100 CET m=+45.923780508 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "26601002-2b7f-44d7-b247-12092d0a23c8" (UID: "26601002-2b7f-44d7-b247-12092d0a23c8") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.637053   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName:10d85e4d-b0d8-4823-b20d-cda28e641673 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.13701871 +0100 CET m=+45.923824501 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "10d85e4d-b0d8-4823-b20d-cda28e641673" (UID: "10d85e4d-b0d8-4823-b20d-cda28e641673") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.637344   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.637421   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.137408923 +0100 CET m=+45.924214725 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.637423   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.637454   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.137443128 +0100 CET m=+45.924248919 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.637503   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-80edd197-58be-4fb7-8420-25e736120a31\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.637594   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.137577841 +0100 CET m=+45.924383632 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.637853   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.137847228 +0100 CET m=+45.924653019 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.637863   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.137859461 +0100 CET m=+45.924665252 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.637870   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.137866514 +0100 CET m=+45.924672305 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.637880   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.137872245 +0100 CET m=+45.924678036 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "nextcloud-nginx-b674d44dd-vwfct" (UID: "faa24a36-35eb-4764-8ab1-d2630b99b684") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.637894   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.137889477 +0100 CET m=+45.924695268 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.637903   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.137898955 +0100 CET m=+45.924704746 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "nextcloud-c9c757d4f-cwjcs" (UID: "2f7da9b0-b444-4bc5-934c-cf0c339d084e") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.738965   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-eaf1e5ab-9a06-441a-b138-70749798101f\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.739034   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\") pod \"26601002-2b7f-44d7-b247-12092d0a23c8\" (UID: \"26601002-2b7f-44d7-b247-12092d0a23c8\") "
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.739084   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.23905659 +0100 CET m=+46.025862381 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.739113   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName:26601002-2b7f-44d7-b247-12092d0a23c8 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.239099371 +0100 CET m=+46.025905172 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "26601002-2b7f-44d7-b247-12092d0a23c8" (UID: "26601002-2b7f-44d7-b247-12092d0a23c8") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.739286   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\") pod \"10d85e4d-b0d8-4823-b20d-cda28e641673\" (UID: \"10d85e4d-b0d8-4823-b20d-cda28e641673\") "
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.739393   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName:10d85e4d-b0d8-4823-b20d-cda28e641673 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.239373987 +0100 CET m=+46.026179778 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "10d85e4d-b0d8-4823-b20d-cda28e641673" (UID: "10d85e4d-b0d8-4823-b20d-cda28e641673") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.739429   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.739500   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.239487591 +0100 CET m=+46.026293382 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.739518   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.739562   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-80edd197-58be-4fb7-8420-25e736120a31\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.739621   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.239600413 +0100 CET m=+46.026406204 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.739657   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.239646109 +0100 CET m=+46.026451900 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.739842   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.239835295 +0100 CET m=+46.026641086 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "nextcloud-nextcloud-cron-28387640-kvwwv" (UID: "485f2017-fcab-46d8-a0f8-71bf110c1836") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.739890   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.239881973 +0100 CET m=+46.026687764 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "nextcloud-nextcloud-cron-28387640-kvwwv" (UID: "485f2017-fcab-46d8-a0f8-71bf110c1836") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.739905   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.239899636 +0100 CET m=+46.026705427 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.739913   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.239908643 +0100 CET m=+46.026714434 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.739921   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.239916307 +0100 CET m=+46.026722108 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.739930   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.239926045 +0100 CET m=+46.026731837 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.841205   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.841288   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.841370   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.341315557 +0100 CET m=+46.128121348 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.841402   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-80edd197-58be-4fb7-8420-25e736120a31\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.841416   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.341389235 +0100 CET m=+46.128195036 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.841545   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.341520683 +0100 CET m=+46.128326484 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.841581   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-eaf1e5ab-9a06-441a-b138-70749798101f\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.841671   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.341649775 +0100 CET m=+46.128455566 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.841699   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\") pod \"26601002-2b7f-44d7-b247-12092d0a23c8\" (UID: \"26601002-2b7f-44d7-b247-12092d0a23c8\") "
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.841803   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName:26601002-2b7f-44d7-b247-12092d0a23c8 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.341781012 +0100 CET m=+46.128586813 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "26601002-2b7f-44d7-b247-12092d0a23c8" (UID: "26601002-2b7f-44d7-b247-12092d0a23c8") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.842084   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\") pod \"10d85e4d-b0d8-4823-b20d-cda28e641673\" (UID: \"10d85e4d-b0d8-4823-b20d-cda28e641673\") "
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.842204   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName:10d85e4d-b0d8-4823-b20d-cda28e641673 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.342179101 +0100 CET m=+46.128984892 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "10d85e4d-b0d8-4823-b20d-cda28e641673" (UID: "10d85e4d-b0d8-4823-b20d-cda28e641673") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.842670   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.342660737 +0100 CET m=+46.129466528 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.842689   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.342682217 +0100 CET m=+46.129488008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.842699   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.342694039 +0100 CET m=+46.129499830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.842708   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.342704269 +0100 CET m=+46.129510060 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "nextcloud-nginx-b674d44dd-vwfct" (UID: "faa24a36-35eb-4764-8ab1-d2630b99b684") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.842721   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.342713586 +0100 CET m=+46.129519377 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.842737   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.342726681 +0100 CET m=+46.129532482 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "nextcloud-c9c757d4f-cwjcs" (UID: "2f7da9b0-b444-4bc5-934c-cf0c339d084e") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.943091   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.943176   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.943206   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.443178809 +0100 CET m=+46.229984600 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.943242   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-80edd197-58be-4fb7-8420-25e736120a31\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.943261   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.44324865 +0100 CET m=+46.230054441 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.943348   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.443338289 +0100 CET m=+46.230144080 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.943367   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-eaf1e5ab-9a06-441a-b138-70749798101f\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.943436   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\") pod \"26601002-2b7f-44d7-b247-12092d0a23c8\" (UID: \"26601002-2b7f-44d7-b247-12092d0a23c8\") "
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.943444   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.443435021 +0100 CET m=+46.230240812 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.943506   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName:26601002-2b7f-44d7-b247-12092d0a23c8 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.443497208 +0100 CET m=+46.230302999 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "26601002-2b7f-44d7-b247-12092d0a23c8" (UID: "26601002-2b7f-44d7-b247-12092d0a23c8") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: I1222 17:43:07.943705   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\") pod \"10d85e4d-b0d8-4823-b20d-cda28e641673\" (UID: \"10d85e4d-b0d8-4823-b20d-cda28e641673\") "
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.943778   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName:10d85e4d-b0d8-4823-b20d-cda28e641673 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.443771544 +0100 CET m=+46.230577325 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "10d85e4d-b0d8-4823-b20d-cda28e641673" (UID: "10d85e4d-b0d8-4823-b20d-cda28e641673") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.944077   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.444068252 +0100 CET m=+46.230874043 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "nextcloud-c9c757d4f-cwjcs" (UID: "2f7da9b0-b444-4bc5-934c-cf0c339d084e") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.944089   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.444083521 +0100 CET m=+46.230889312 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.944096   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.444091796 +0100 CET m=+46.230897587 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "nextcloud-nginx-b674d44dd-vwfct" (UID: "faa24a36-35eb-4764-8ab1-d2630b99b684") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.944102   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.44409895 +0100 CET m=+46.230904741 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.944108   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.444105071 +0100 CET m=+46.230910852 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:07 truenas k3s[16911]: E1222 17:43:07.944156   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.44414708 +0100 CET m=+46.230952881 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.044705   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-eaf1e5ab-9a06-441a-b138-70749798101f\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.044808   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\") pod \"26601002-2b7f-44d7-b247-12092d0a23c8\" (UID: \"26601002-2b7f-44d7-b247-12092d0a23c8\") "
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.044851   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.544812439 +0100 CET m=+46.331618231 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.044893   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName:26601002-2b7f-44d7-b247-12092d0a23c8 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.544871811 +0100 CET m=+46.331677602 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "26601002-2b7f-44d7-b247-12092d0a23c8" (UID: "26601002-2b7f-44d7-b247-12092d0a23c8") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.045144   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\") pod \"10d85e4d-b0d8-4823-b20d-cda28e641673\" (UID: \"10d85e4d-b0d8-4823-b20d-cda28e641673\") "
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.045221   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName:10d85e4d-b0d8-4823-b20d-cda28e641673 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.5451957 +0100 CET m=+46.332001491 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "10d85e4d-b0d8-4823-b20d-cda28e641673" (UID: "10d85e4d-b0d8-4823-b20d-cda28e641673") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.045402   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.045505   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.045508   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.545481708 +0100 CET m=+46.332287499 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.045601   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.545576877 +0100 CET m=+46.332382678 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.045631   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-80edd197-58be-4fb7-8420-25e736120a31\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.045737   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.545714486 +0100 CET m=+46.332520277 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.046091   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.546077809 +0100 CET m=+46.332883600 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.046110   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.546101033 +0100 CET m=+46.332906824 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "nextcloud-nginx-b674d44dd-vwfct" (UID: "faa24a36-35eb-4764-8ab1-d2630b99b684") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.046125   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.546115931 +0100 CET m=+46.332921722 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.046139   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.546130328 +0100 CET m=+46.332936119 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "nextcloud-nginx-b674d44dd-vwfct" (UID: "faa24a36-35eb-4764-8ab1-d2630b99b684") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.046155   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.546145066 +0100 CET m=+46.332950857 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.046170   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.546161266 +0100 CET m=+46.332967057 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.147130   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.147208   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.147253   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.64722245 +0100 CET m=+46.434028241 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.147287   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.647271963 +0100 CET m=+46.434077754 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.147288   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-80edd197-58be-4fb7-8420-25e736120a31\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.147321   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.647309834 +0100 CET m=+46.434115625 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.147394   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-eaf1e5ab-9a06-441a-b138-70749798101f\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.147481   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.647468773 +0100 CET m=+46.434274564 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.147497   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\") pod \"26601002-2b7f-44d7-b247-12092d0a23c8\" (UID: \"26601002-2b7f-44d7-b247-12092d0a23c8\") "
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.147578   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName:26601002-2b7f-44d7-b247-12092d0a23c8 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.647568932 +0100 CET m=+46.434374733 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "26601002-2b7f-44d7-b247-12092d0a23c8" (UID: "26601002-2b7f-44d7-b247-12092d0a23c8") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.147671   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\") pod \"10d85e4d-b0d8-4823-b20d-cda28e641673\" (UID: \"10d85e4d-b0d8-4823-b20d-cda28e641673\") "
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.147748   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName:10d85e4d-b0d8-4823-b20d-cda28e641673 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.647738551 +0100 CET m=+46.434544342 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "10d85e4d-b0d8-4823-b20d-cda28e641673" (UID: "10d85e4d-b0d8-4823-b20d-cda28e641673") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.147839   16911 reconciler_common.go:253] "operationExecutor.VerifyControllerAttachedVolume started for volume \"shared\" (UniqueName: \"kubernetes.io/empty-dir/fec4f553-b289-49fe-ab03-9284189e5216-shared\") pod \"cloudflareddns-56c9c85bb8-h822b\" (UID: \"fec4f553-b289-49fe-ab03-9284189e5216\") " pod="ix-cloudflareddns/cloudflareddns-56c9c85bb8-h822b"
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.147970   16911 reconciler_common.go:253] "operationExecutor.VerifyControllerAttachedVolume started for volume \"varlogs\" (UniqueName: \"kubernetes.io/empty-dir/fec4f553-b289-49fe-ab03-9284189e5216-varlogs\") pod \"cloudflareddns-56c9c85bb8-h822b\" (UID: \"fec4f553-b289-49fe-ab03-9284189e5216\") " pod="ix-cloudflareddns/cloudflareddns-56c9c85bb8-h822b"
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.147999   16911 reconciler_common.go:253] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fec4f553-b289-49fe-ab03-9284189e5216-tmp\") pod \"cloudflareddns-56c9c85bb8-h822b\" (UID: \"fec4f553-b289-49fe-ab03-9284189e5216\") " pod="ix-cloudflareddns/cloudflareddns-56c9c85bb8-h822b"
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.148011   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.648001656 +0100 CET m=+46.434807447 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.148021   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.648015321 +0100 CET m=+46.434821112 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "nextcloud-nginx-b674d44dd-vwfct" (UID: "faa24a36-35eb-4764-8ab1-d2630b99b684") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.148028   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.648024859 +0100 CET m=+46.434830640 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.148079   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.64807308 +0100 CET m=+46.434878871 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "nextcloud-c9c757d4f-cwjcs" (UID: "2f7da9b0-b444-4bc5-934c-cf0c339d084e") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.148098   16911 reconciler_common.go:253] "operationExecutor.VerifyControllerAttachedVolume started for volume \"varrun\" (UniqueName: \"kubernetes.io/empty-dir/fec4f553-b289-49fe-ab03-9284189e5216-varrun\") pod \"cloudflareddns-56c9c85bb8-h822b\" (UID: \"fec4f553-b289-49fe-ab03-9284189e5216\") " pod="ix-cloudflareddns/cloudflareddns-56c9c85bb8-h822b"
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.148123   16911 reconciler_common.go:253] "operationExecutor.VerifyControllerAttachedVolume started for volume \"devshm\" (UniqueName: \"kubernetes.io/empty-dir/fec4f553-b289-49fe-ab03-9284189e5216-devshm\") pod \"cloudflareddns-56c9c85bb8-h822b\" (UID: \"fec4f553-b289-49fe-ab03-9284189e5216\") " pod="ix-cloudflareddns/cloudflareddns-56c9c85bb8-h822b"
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.148127   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.648120349 +0100 CET m=+46.434926140 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.148147   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.648138272 +0100 CET m=+46.434944063 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2") pod "cloudflareddns-56c9c85bb8-h822b" (UID: "fec4f553-b289-49fe-ab03-9284189e5216") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.148223   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.648213404 +0100 CET m=+46.435019195 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.248410   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.248488   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-80edd197-58be-4fb7-8420-25e736120a31\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.248558   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.748516452 +0100 CET m=+46.535322243 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.248589   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2\") pod \"4b670339-8e2a-4556-bbff-b6e9b0a720c6\" (UID: \"4b670339-8e2a-4556-bbff-b6e9b0a720c6\") "
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.248609   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.748585111 +0100 CET m=+46.535390902 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.248664   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2 podName:4b670339-8e2a-4556-bbff-b6e9b0a720c6 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.748647999 +0100 CET m=+46.535453800 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2") pod "4b670339-8e2a-4556-bbff-b6e9b0a720c6" (UID: "4b670339-8e2a-4556-bbff-b6e9b0a720c6") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.248690   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-eaf1e5ab-9a06-441a-b138-70749798101f\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.248783   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.748766071 +0100 CET m=+46.535571862 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.248809   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\") pod \"26601002-2b7f-44d7-b247-12092d0a23c8\" (UID: \"26601002-2b7f-44d7-b247-12092d0a23c8\") "
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.248901   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName:26601002-2b7f-44d7-b247-12092d0a23c8 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.748883742 +0100 CET m=+46.535689564 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "26601002-2b7f-44d7-b247-12092d0a23c8" (UID: "26601002-2b7f-44d7-b247-12092d0a23c8") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.249180   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\") pod \"10d85e4d-b0d8-4823-b20d-cda28e641673\" (UID: \"10d85e4d-b0d8-4823-b20d-cda28e641673\") "
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.249260   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName:10d85e4d-b0d8-4823-b20d-cda28e641673 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.749248128 +0100 CET m=+46.536053919 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "10d85e4d-b0d8-4823-b20d-cda28e641673" (UID: "10d85e4d-b0d8-4823-b20d-cda28e641673") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.249366   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.249437   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.749426824 +0100 CET m=+46.536232605 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.249740   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.749732739 +0100 CET m=+46.536538530 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.249762   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.749750062 +0100 CET m=+46.536555853 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.249781   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.749769849 +0100 CET m=+46.536575640 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "nextcloud-nginx-b674d44dd-vwfct" (UID: "faa24a36-35eb-4764-8ab1-d2630b99b684") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.249827   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.749814032 +0100 CET m=+46.536619813 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.249865   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.74985549 +0100 CET m=+46.536661301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "nextcloud-nginx-b674d44dd-vwfct" (UID: "faa24a36-35eb-4764-8ab1-d2630b99b684") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.249882   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.749874616 +0100 CET m=+46.536680407 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2") pod "cloudflareddns-56c9c85bb8-h822b" (UID: "fec4f553-b289-49fe-ab03-9284189e5216") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.249898   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.7498872 +0100 CET m=+46.536692991 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.350452   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.350539   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-80edd197-58be-4fb7-8420-25e736120a31\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.350589   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.850550175 +0100 CET m=+46.637355996 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.350638   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2\") pod \"4b670339-8e2a-4556-bbff-b6e9b0a720c6\" (UID: \"4b670339-8e2a-4556-bbff-b6e9b0a720c6\") "
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.350650   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.85062717 +0100 CET m=+46.637432961 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.350700   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-eaf1e5ab-9a06-441a-b138-70749798101f\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.350704   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2 podName:4b670339-8e2a-4556-bbff-b6e9b0a720c6 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.850687273 +0100 CET m=+46.637493154 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2") pod "4b670339-8e2a-4556-bbff-b6e9b0a720c6" (UID: "4b670339-8e2a-4556-bbff-b6e9b0a720c6") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.350780   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\") pod \"26601002-2b7f-44d7-b247-12092d0a23c8\" (UID: \"26601002-2b7f-44d7-b247-12092d0a23c8\") "
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.350799   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.850777212 +0100 CET m=+46.637583013 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.350849   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName:26601002-2b7f-44d7-b247-12092d0a23c8 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.850831184 +0100 CET m=+46.637636975 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "26601002-2b7f-44d7-b247-12092d0a23c8" (UID: "26601002-2b7f-44d7-b247-12092d0a23c8") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.351025   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\") pod \"10d85e4d-b0d8-4823-b20d-cda28e641673\" (UID: \"10d85e4d-b0d8-4823-b20d-cda28e641673\") "
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.351128   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName:10d85e4d-b0d8-4823-b20d-cda28e641673 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.851109868 +0100 CET m=+46.637915659 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "10d85e4d-b0d8-4823-b20d-cda28e641673" (UID: "10d85e4d-b0d8-4823-b20d-cda28e641673") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.351263   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.351416   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.851396147 +0100 CET m=+46.638201938 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.351587   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.851575284 +0100 CET m=+46.638381075 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2") pod "cloudflareddns-56c9c85bb8-h822b" (UID: "fec4f553-b289-49fe-ab03-9284189e5216") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.351690   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.851676394 +0100 CET m=+46.638482185 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.351716   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.851702874 +0100 CET m=+46.638508675 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "nextcloud-preview-cron-28387720-dq7tq" (UID: "f625e3e8-e7a8-4439-8d8c-6f10b1c7978a") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.351728   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.851723222 +0100 CET m=+46.638529013 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "nextcloud-nginx-b674d44dd-vwfct" (UID: "faa24a36-35eb-4764-8ab1-d2630b99b684") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.351736   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.851731528 +0100 CET m=+46.638537319 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.351765   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.85175912 +0100 CET m=+46.638564911 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.351772   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.851768227 +0100 CET m=+46.638574018 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.452569   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.452665   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-80edd197-58be-4fb7-8420-25e736120a31\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.452733   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.952682495 +0100 CET m=+46.739488286 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.452781   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.95275444 +0100 CET m=+46.739560231 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.452782   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2\") pod \"4b670339-8e2a-4556-bbff-b6e9b0a720c6\" (UID: \"4b670339-8e2a-4556-bbff-b6e9b0a720c6\") "
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.452827   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2 podName:4b670339-8e2a-4556-bbff-b6e9b0a720c6 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.952809483 +0100 CET m=+46.739615274 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2") pod "4b670339-8e2a-4556-bbff-b6e9b0a720c6" (UID: "4b670339-8e2a-4556-bbff-b6e9b0a720c6") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.452873   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-eaf1e5ab-9a06-441a-b138-70749798101f\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.452975   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.952960247 +0100 CET m=+46.739766038 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.452998   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\") pod \"26601002-2b7f-44d7-b247-12092d0a23c8\" (UID: \"26601002-2b7f-44d7-b247-12092d0a23c8\") "
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.453120   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName:26601002-2b7f-44d7-b247-12092d0a23c8 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.953097666 +0100 CET m=+46.739903477 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "26601002-2b7f-44d7-b247-12092d0a23c8" (UID: "26601002-2b7f-44d7-b247-12092d0a23c8") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.453537   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\") pod \"10d85e4d-b0d8-4823-b20d-cda28e641673\" (UID: \"10d85e4d-b0d8-4823-b20d-cda28e641673\") "
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.453634   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName:10d85e4d-b0d8-4823-b20d-cda28e641673 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.953618014 +0100 CET m=+46.740423805 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "10d85e4d-b0d8-4823-b20d-cda28e641673" (UID: "10d85e4d-b0d8-4823-b20d-cda28e641673") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.453736   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.453819   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.953805797 +0100 CET m=+46.740611588 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.454207   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.954192344 +0100 CET m=+46.740998135 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.454230   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.954218714 +0100 CET m=+46.741024505 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "nextcloud-nginx-b674d44dd-vwfct" (UID: "faa24a36-35eb-4764-8ab1-d2630b99b684") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.454246   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.954237379 +0100 CET m=+46.741043170 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.454260   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.954252177 +0100 CET m=+46.741057968 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "nextcloud-nginx-b674d44dd-vwfct" (UID: "faa24a36-35eb-4764-8ab1-d2630b99b684") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.454274   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.954265802 +0100 CET m=+46.741071594 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2") pod "cloudflareddns-56c9c85bb8-h822b" (UID: "fec4f553-b289-49fe-ab03-9284189e5216") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.454317   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.954306499 +0100 CET m=+46.741112300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.454369   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:08.954355391 +0100 CET m=+46.741161182 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.496303   16911 event.go:294] "Event occurred" object="ix-traefik/traefik-5996c85fd6" fieldPath="" kind="ReplicaSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulCreate" message="Created pod: traefik-5996c85fd6-n64lg"
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.503067   16911 topology_manager.go:210] "Topology Admit Handler"
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.555103   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.555203   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-80edd197-58be-4fb7-8420-25e736120a31\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.555322   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.055273115 +0100 CET m=+46.842078906 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.555360   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2\") pod \"4b670339-8e2a-4556-bbff-b6e9b0a720c6\" (UID: \"4b670339-8e2a-4556-bbff-b6e9b0a720c6\") "
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.555445   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-eaf1e5ab-9a06-441a-b138-70749798101f\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.555487   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2 podName:4b670339-8e2a-4556-bbff-b6e9b0a720c6 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.055452613 +0100 CET m=+46.842258404 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2") pod "4b670339-8e2a-4556-bbff-b6e9b0a720c6" (UID: "4b670339-8e2a-4556-bbff-b6e9b0a720c6") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.555517   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\") pod \"26601002-2b7f-44d7-b247-12092d0a23c8\" (UID: \"26601002-2b7f-44d7-b247-12092d0a23c8\") "
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.555613   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.055591043 +0100 CET m=+46.842396834 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.555667   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName:26601002-2b7f-44d7-b247-12092d0a23c8 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.055650615 +0100 CET m=+46.842456426 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "26601002-2b7f-44d7-b247-12092d0a23c8" (UID: "26601002-2b7f-44d7-b247-12092d0a23c8") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.555800   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\") pod \"10d85e4d-b0d8-4823-b20d-cda28e641673\" (UID: \"10d85e4d-b0d8-4823-b20d-cda28e641673\") "
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.555845   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.055822107 +0100 CET m=+46.842627919 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.555888   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName:10d85e4d-b0d8-4823-b20d-cda28e641673 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.055870569 +0100 CET m=+46.842676360 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "10d85e4d-b0d8-4823-b20d-cda28e641673" (UID: "10d85e4d-b0d8-4823-b20d-cda28e641673") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.556107   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.556435   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.056417788 +0100 CET m=+46.843223579 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.556601   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.056592767 +0100 CET m=+46.843398558 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.556619   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.056611572 +0100 CET m=+46.843417363 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.556689   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.056679209 +0100 CET m=+46.843485000 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "nextcloud-nginx-b674d44dd-vwfct" (UID: "faa24a36-35eb-4764-8ab1-d2630b99b684") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.556725   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.056714987 +0100 CET m=+46.843520778 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.556743   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.056735666 +0100 CET m=+46.843541457 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2") pod "cloudflareddns-56c9c85bb8-h822b" (UID: "fec4f553-b289-49fe-ab03-9284189e5216") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.556873   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.056860841 +0100 CET m=+46.843666632 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "nextcloud-nginx-b674d44dd-vwfct" (UID: "faa24a36-35eb-4764-8ab1-d2630b99b684") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.558231   16911 controller.go:615] quota admission added evaluator for: zfsnodes.zfs.openebs.io
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.657723   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.658090   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-80edd197-58be-4fb7-8420-25e736120a31\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.658208   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.158177385 +0100 CET m=+46.944983176 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.658336   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2\") pod \"4b670339-8e2a-4556-bbff-b6e9b0a720c6\" (UID: \"4b670339-8e2a-4556-bbff-b6e9b0a720c6\") "
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.658425   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-eaf1e5ab-9a06-441a-b138-70749798101f\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.658474   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.158440229 +0100 CET m=+46.945246020 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.658502   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\") pod \"26601002-2b7f-44d7-b247-12092d0a23c8\" (UID: \"26601002-2b7f-44d7-b247-12092d0a23c8\") "
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.658511   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2 podName:4b670339-8e2a-4556-bbff-b6e9b0a720c6 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.158490213 +0100 CET m=+46.945296004 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2") pod "4b670339-8e2a-4556-bbff-b6e9b0a720c6" (UID: "4b670339-8e2a-4556-bbff-b6e9b0a720c6") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.658557   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.158542472 +0100 CET m=+46.945348263 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.658595   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName:26601002-2b7f-44d7-b247-12092d0a23c8 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.158577237 +0100 CET m=+46.945383028 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "26601002-2b7f-44d7-b247-12092d0a23c8" (UID: "26601002-2b7f-44d7-b247-12092d0a23c8") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.658881   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\") pod \"10d85e4d-b0d8-4823-b20d-cda28e641673\" (UID: \"10d85e4d-b0d8-4823-b20d-cda28e641673\") "
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.659133   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName:10d85e4d-b0d8-4823-b20d-cda28e641673 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.159113426 +0100 CET m=+46.945919217 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "10d85e4d-b0d8-4823-b20d-cda28e641673" (UID: "10d85e4d-b0d8-4823-b20d-cda28e641673") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.659215   16911 reconciler_common.go:253] "operationExecutor.VerifyControllerAttachedVolume started for volume \"shared\" (UniqueName: \"kubernetes.io/empty-dir/53422154-3655-45af-bcdc-bbcd3468cabc-shared\") pod \"traefik-5996c85fd6-n64lg\" (UID: \"53422154-3655-45af-bcdc-bbcd3468cabc\") " pod="ix-traefik/traefik-5996c85fd6-n64lg"
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.659337   16911 reconciler_common.go:253] "operationExecutor.VerifyControllerAttachedVolume started for volume \"varlogs\" (UniqueName: \"kubernetes.io/empty-dir/53422154-3655-45af-bcdc-bbcd3468cabc-varlogs\") pod \"traefik-5996c85fd6-n64lg\" (UID: \"53422154-3655-45af-bcdc-bbcd3468cabc\") " pod="ix-traefik/traefik-5996c85fd6-n64lg"
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.659370   16911 reconciler_common.go:253] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/53422154-3655-45af-bcdc-bbcd3468cabc-plugins\") pod \"traefik-5996c85fd6-n64lg\" (UID: \"53422154-3655-45af-bcdc-bbcd3468cabc\") " pod="ix-traefik/traefik-5996c85fd6-n64lg"
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.659401   16911 reconciler_common.go:253] "operationExecutor.VerifyControllerAttachedVolume started for volume \"devshm\" (UniqueName: \"kubernetes.io/empty-dir/53422154-3655-45af-bcdc-bbcd3468cabc-devshm\") pod \"traefik-5996c85fd6-n64lg\" (UID: \"53422154-3655-45af-bcdc-bbcd3468cabc\") " pod="ix-traefik/traefik-5996c85fd6-n64lg"
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.659487   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.159469415 +0100 CET m=+46.946275206 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.659748   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.159734433 +0100 CET m=+46.946540225 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.659823   16911 reconciler_common.go:253] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/53422154-3655-45af-bcdc-bbcd3468cabc-tmp\") pod \"traefik-5996c85fd6-n64lg\" (UID: \"53422154-3655-45af-bcdc-bbcd3468cabc\") " pod="ix-traefik/traefik-5996c85fd6-n64lg"
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.659827   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.1598125 +0100 CET m=+46.946618291 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "nextcloud-nextcloud-cron-28387640-kvwwv" (UID: "485f2017-fcab-46d8-a0f8-71bf110c1836") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.659855   16911 reconciler_common.go:253] "operationExecutor.VerifyControllerAttachedVolume started for volume \"varrun\" (UniqueName: \"kubernetes.io/empty-dir/53422154-3655-45af-bcdc-bbcd3468cabc-varrun\") pod \"traefik-5996c85fd6-n64lg\" (UID: \"53422154-3655-45af-bcdc-bbcd3468cabc\") " pod="ix-traefik/traefik-5996c85fd6-n64lg"
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.659928   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.159916145 +0100 CET m=+46.946721936 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.659979   16911 reconciler_common.go:253] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9ffj\" (UniqueName: \"kubernetes.io/projected/53422154-3655-45af-bcdc-bbcd3468cabc-kube-api-access-f9ffj\") pod \"traefik-5996c85fd6-n64lg\" (UID: \"53422154-3655-45af-bcdc-bbcd3468cabc\") " pod="ix-traefik/traefik-5996c85fd6-n64lg"
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.660109   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.160086766 +0100 CET m=+46.946892557 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "nextcloud-preview-cron-28387720-dq7tq" (UID: "f625e3e8-e7a8-4439-8d8c-6f10b1c7978a") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.660250   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.160240595 +0100 CET m=+46.947046386 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.660271   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.160261896 +0100 CET m=+46.947067687 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2") pod "cloudflareddns-56c9c85bb8-h822b" (UID: "fec4f553-b289-49fe-ab03-9284189e5216") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.761524   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.761610   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.261558753 +0100 CET m=+47.048364544 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.764906   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.765268   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.265228815 +0100 CET m=+47.052034616 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.765535   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-80edd197-58be-4fb7-8420-25e736120a31\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.765634   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2\") pod \"4b670339-8e2a-4556-bbff-b6e9b0a720c6\" (UID: \"4b670339-8e2a-4556-bbff-b6e9b0a720c6\") "
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.765709   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-eaf1e5ab-9a06-441a-b138-70749798101f\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.765786   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\") pod \"26601002-2b7f-44d7-b247-12092d0a23c8\" (UID: \"26601002-2b7f-44d7-b247-12092d0a23c8\") "
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.766145   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\") pod \"10d85e4d-b0d8-4823-b20d-cda28e641673\" (UID: \"10d85e4d-b0d8-4823-b20d-cda28e641673\") "
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.766472   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.266450152 +0100 CET m=+47.053255943 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.766853   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName:10d85e4d-b0d8-4823-b20d-cda28e641673 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.266832842 +0100 CET m=+47.053638643 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "10d85e4d-b0d8-4823-b20d-cda28e641673" (UID: "10d85e4d-b0d8-4823-b20d-cda28e641673") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.766959   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.266940023 +0100 CET m=+47.053745814 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.767052   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName:26601002-2b7f-44d7-b247-12092d0a23c8 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.267032838 +0100 CET m=+47.053838629 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "26601002-2b7f-44d7-b247-12092d0a23c8" (UID: "26601002-2b7f-44d7-b247-12092d0a23c8") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.767468   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.267456535 +0100 CET m=+47.054262326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.767582   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.267569116 +0100 CET m=+47.054374907 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.767678   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2 podName:4b670339-8e2a-4556-bbff-b6e9b0a720c6 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.267657172 +0100 CET m=+47.054462963 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2") pod "4b670339-8e2a-4556-bbff-b6e9b0a720c6" (UID: "4b670339-8e2a-4556-bbff-b6e9b0a720c6") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.767843   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.267828825 +0100 CET m=+47.054634616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.868273   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-eaf1e5ab-9a06-441a-b138-70749798101f\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.868549   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.368506848 +0100 CET m=+47.155312639 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.868953   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.869070   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.869202   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.36917229 +0100 CET m=+47.155978081 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.869342   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.369324095 +0100 CET m=+47.156129876 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.869721   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.369711103 +0100 CET m=+47.156516884 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.869746   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.369734266 +0100 CET m=+47.156540057 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.869761   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.369754304 +0100 CET m=+47.156560095 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "nextcloud-nginx-b674d44dd-vwfct" (UID: "faa24a36-35eb-4764-8ab1-d2630b99b684") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.869779   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.369766677 +0100 CET m=+47.156572468 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "nextcloud-nginx-b674d44dd-vwfct" (UID: "faa24a36-35eb-4764-8ab1-d2630b99b684") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.869838   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.369829185 +0100 CET m=+47.156634976 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2") pod "cloudflareddns-56c9c85bb8-h822b" (UID: "fec4f553-b289-49fe-ab03-9284189e5216") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.869854   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.369848281 +0100 CET m=+47.156654072 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.869865   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.369859181 +0100 CET m=+47.156664992 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.970361   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.970441   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.970504   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.47046516 +0100 CET m=+47.257270951 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.970538   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-80edd197-58be-4fb7-8420-25e736120a31\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.970550   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.470525924 +0100 CET m=+47.257331715 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.970635   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.470606706 +0100 CET m=+47.257412497 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.970668   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2\") pod \"4b670339-8e2a-4556-bbff-b6e9b0a720c6\" (UID: \"4b670339-8e2a-4556-bbff-b6e9b0a720c6\") "
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.970753   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-eaf1e5ab-9a06-441a-b138-70749798101f\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.970765   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2 podName:4b670339-8e2a-4556-bbff-b6e9b0a720c6 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.470747721 +0100 CET m=+47.257553512 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2") pod "4b670339-8e2a-4556-bbff-b6e9b0a720c6" (UID: "4b670339-8e2a-4556-bbff-b6e9b0a720c6") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.970834   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\") pod \"26601002-2b7f-44d7-b247-12092d0a23c8\" (UID: \"26601002-2b7f-44d7-b247-12092d0a23c8\") "
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.970840   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.470822031 +0100 CET m=+47.257627822 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.970939   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName:26601002-2b7f-44d7-b247-12092d0a23c8 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.470925055 +0100 CET m=+47.257730846 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "26601002-2b7f-44d7-b247-12092d0a23c8" (UID: "26601002-2b7f-44d7-b247-12092d0a23c8") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: I1222 17:43:08.971111   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\") pod \"10d85e4d-b0d8-4823-b20d-cda28e641673\" (UID: \"10d85e4d-b0d8-4823-b20d-cda28e641673\") "
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.971216   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName:10d85e4d-b0d8-4823-b20d-cda28e641673 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.471200142 +0100 CET m=+47.258005933 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "10d85e4d-b0d8-4823-b20d-cda28e641673" (UID: "10d85e4d-b0d8-4823-b20d-cda28e641673") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.971770   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.471758803 +0100 CET m=+47.258564594 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "nextcloud-nginx-b674d44dd-vwfct" (UID: "faa24a36-35eb-4764-8ab1-d2630b99b684") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.971790   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.471782568 +0100 CET m=+47.258588359 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.971810   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.471797236 +0100 CET m=+47.258603027 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.971921   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.471910619 +0100 CET m=+47.258716410 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.971939   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.471929294 +0100 CET m=+47.258735085 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2") pod "cloudflareddns-56c9c85bb8-h822b" (UID: "fec4f553-b289-49fe-ab03-9284189e5216") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.971953   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.471943411 +0100 CET m=+47.258749202 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:08 truenas k3s[16911]: E1222 17:43:08.972069   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.472059158 +0100 CET m=+47.258864949 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "nextcloud-c9c757d4f-cwjcs" (UID: "2f7da9b0-b444-4bc5-934c-cf0c339d084e") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.071975   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.072109   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.572067552 +0100 CET m=+47.358873343 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.072144   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-80edd197-58be-4fb7-8420-25e736120a31\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.072250   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2\") pod \"4b670339-8e2a-4556-bbff-b6e9b0a720c6\" (UID: \"4b670339-8e2a-4556-bbff-b6e9b0a720c6\") "
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.072296   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.572260455 +0100 CET m=+47.359066266 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.072346   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-eaf1e5ab-9a06-441a-b138-70749798101f\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.072351   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2 podName:4b670339-8e2a-4556-bbff-b6e9b0a720c6 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.572333913 +0100 CET m=+47.359139694 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2") pod "4b670339-8e2a-4556-bbff-b6e9b0a720c6" (UID: "4b670339-8e2a-4556-bbff-b6e9b0a720c6") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.072450   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.572434262 +0100 CET m=+47.359240053 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.072478   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\") pod \"26601002-2b7f-44d7-b247-12092d0a23c8\" (UID: \"26601002-2b7f-44d7-b247-12092d0a23c8\") "
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.072600   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName:26601002-2b7f-44d7-b247-12092d0a23c8 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.572575468 +0100 CET m=+47.359381259 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "26601002-2b7f-44d7-b247-12092d0a23c8" (UID: "26601002-2b7f-44d7-b247-12092d0a23c8") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.072835   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\") pod \"10d85e4d-b0d8-4823-b20d-cda28e641673\" (UID: \"10d85e4d-b0d8-4823-b20d-cda28e641673\") "
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.072938   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName:10d85e4d-b0d8-4823-b20d-cda28e641673 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.572917311 +0100 CET m=+47.359723102 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "10d85e4d-b0d8-4823-b20d-cda28e641673" (UID: "10d85e4d-b0d8-4823-b20d-cda28e641673") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.073040   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.073198   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.573180967 +0100 CET m=+47.359986758 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.073709   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.573698279 +0100 CET m=+47.360504071 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.073728   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.573720401 +0100 CET m=+47.360526192 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "nextcloud-nginx-b674d44dd-vwfct" (UID: "faa24a36-35eb-4764-8ab1-d2630b99b684") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.073830   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.573819067 +0100 CET m=+47.360624858 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.073880   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.573871095 +0100 CET m=+47.360676886 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.073980   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.573972295 +0100 CET m=+47.360778076 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.074018   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.574008984 +0100 CET m=+47.360814775 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "nextcloud-nextcloud-cron-28387640-kvwwv" (UID: "485f2017-fcab-46d8-a0f8-71bf110c1836") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.074149   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.574140521 +0100 CET m=+47.360946312 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2") pod "cloudflareddns-56c9c85bb8-h822b" (UID: "fec4f553-b289-49fe-ab03-9284189e5216") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.174811   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.174880   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.174934   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-80edd197-58be-4fb7-8420-25e736120a31\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.174961   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.674923121 +0100 CET m=+47.461728912 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.174982   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.674971171 +0100 CET m=+47.461776963 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.175009   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2\") pod \"4b670339-8e2a-4556-bbff-b6e9b0a720c6\" (UID: \"4b670339-8e2a-4556-bbff-b6e9b0a720c6\") "
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.175030   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.675014062 +0100 CET m=+47.461819853 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.175103   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-eaf1e5ab-9a06-441a-b138-70749798101f\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.175132   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2 podName:4b670339-8e2a-4556-bbff-b6e9b0a720c6 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.675106316 +0100 CET m=+47.461912117 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2") pod "4b670339-8e2a-4556-bbff-b6e9b0a720c6" (UID: "4b670339-8e2a-4556-bbff-b6e9b0a720c6") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.175192   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.675181347 +0100 CET m=+47.461987128 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.175288   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\") pod \"26601002-2b7f-44d7-b247-12092d0a23c8\" (UID: \"26601002-2b7f-44d7-b247-12092d0a23c8\") "
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.175340   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\") pod \"10d85e4d-b0d8-4823-b20d-cda28e641673\" (UID: \"10d85e4d-b0d8-4823-b20d-cda28e641673\") "
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.175381   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName:26601002-2b7f-44d7-b247-12092d0a23c8 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.675368649 +0100 CET m=+47.462174440 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "26601002-2b7f-44d7-b247-12092d0a23c8" (UID: "26601002-2b7f-44d7-b247-12092d0a23c8") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.175412   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName:10d85e4d-b0d8-4823-b20d-cda28e641673 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.675395089 +0100 CET m=+47.462200890 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "10d85e4d-b0d8-4823-b20d-cda28e641673" (UID: "10d85e4d-b0d8-4823-b20d-cda28e641673") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.175877   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.675867207 +0100 CET m=+47.462672998 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "nextcloud-nextcloud-cron-28387640-kvwwv" (UID: "485f2017-fcab-46d8-a0f8-71bf110c1836") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.175896   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.675888577 +0100 CET m=+47.462694368 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "nextcloud-nginx-b674d44dd-vwfct" (UID: "faa24a36-35eb-4764-8ab1-d2630b99b684") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.175908   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.675903455 +0100 CET m=+47.462709246 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.175929   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.675919114 +0100 CET m=+47.462724905 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2") pod "cloudflareddns-56c9c85bb8-h822b" (UID: "fec4f553-b289-49fe-ab03-9284189e5216") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.175947   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.675938831 +0100 CET m=+47.462744622 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.175974   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.675962255 +0100 CET m=+47.462768046 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.176011   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.675999956 +0100 CET m=+47.462805747 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.276396   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.276474   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.276560   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.776505114 +0100 CET m=+47.563310905 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.276599   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.776573523 +0100 CET m=+47.563379314 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.276633   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-80edd197-58be-4fb7-8420-25e736120a31\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.276739   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.776716462 +0100 CET m=+47.563522283 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.276777   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2\") pod \"4b670339-8e2a-4556-bbff-b6e9b0a720c6\" (UID: \"4b670339-8e2a-4556-bbff-b6e9b0a720c6\") "
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.276864   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2 podName:4b670339-8e2a-4556-bbff-b6e9b0a720c6 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.776846717 +0100 CET m=+47.563652508 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2") pod "4b670339-8e2a-4556-bbff-b6e9b0a720c6" (UID: "4b670339-8e2a-4556-bbff-b6e9b0a720c6") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.276890   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-eaf1e5ab-9a06-441a-b138-70749798101f\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.276979   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\") pod \"26601002-2b7f-44d7-b247-12092d0a23c8\" (UID: \"26601002-2b7f-44d7-b247-12092d0a23c8\") "
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.276987   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.776967594 +0100 CET m=+47.563773405 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.277132   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName:26601002-2b7f-44d7-b247-12092d0a23c8 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.777097058 +0100 CET m=+47.563902859 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "26601002-2b7f-44d7-b247-12092d0a23c8" (UID: "26601002-2b7f-44d7-b247-12092d0a23c8") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.277376   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\") pod \"10d85e4d-b0d8-4823-b20d-cda28e641673\" (UID: \"10d85e4d-b0d8-4823-b20d-cda28e641673\") "
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.277465   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName:10d85e4d-b0d8-4823-b20d-cda28e641673 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.777451063 +0100 CET m=+47.564256855 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "10d85e4d-b0d8-4823-b20d-cda28e641673" (UID: "10d85e4d-b0d8-4823-b20d-cda28e641673") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.277878   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.777868849 +0100 CET m=+47.564674630 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "nextcloud-nginx-b674d44dd-vwfct" (UID: "faa24a36-35eb-4764-8ab1-d2630b99b684") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.277973   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.777964579 +0100 CET m=+47.564770370 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.277987   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.777981281 +0100 CET m=+47.564787072 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "nextcloud-c9c757d4f-cwjcs" (UID: "2f7da9b0-b444-4bc5-934c-cf0c339d084e") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.278000   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.777993634 +0100 CET m=+47.564799425 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.278010   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.778004855 +0100 CET m=+47.564810646 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.278020   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.778014834 +0100 CET m=+47.564820625 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.278030   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.778024923 +0100 CET m=+47.564830714 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2") pod "cloudflareddns-56c9c85bb8-h822b" (UID: "fec4f553-b289-49fe-ab03-9284189e5216") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.378368   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.378459   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-80edd197-58be-4fb7-8420-25e736120a31\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.378531   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.87848718 +0100 CET m=+47.665292971 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.378566   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2\") pod \"4b670339-8e2a-4556-bbff-b6e9b0a720c6\" (UID: \"4b670339-8e2a-4556-bbff-b6e9b0a720c6\") "
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.378572   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.878549638 +0100 CET m=+47.665355429 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.378649   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-eaf1e5ab-9a06-441a-b138-70749798101f\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.378658   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2 podName:4b670339-8e2a-4556-bbff-b6e9b0a720c6 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.87863597 +0100 CET m=+47.665441761 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2") pod "4b670339-8e2a-4556-bbff-b6e9b0a720c6" (UID: "4b670339-8e2a-4556-bbff-b6e9b0a720c6") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.378726   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\") pod \"26601002-2b7f-44d7-b247-12092d0a23c8\" (UID: \"26601002-2b7f-44d7-b247-12092d0a23c8\") "
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.378728   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.878714918 +0100 CET m=+47.665520719 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.378821   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName:26601002-2b7f-44d7-b247-12092d0a23c8 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.878805168 +0100 CET m=+47.665610959 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "26601002-2b7f-44d7-b247-12092d0a23c8" (UID: "26601002-2b7f-44d7-b247-12092d0a23c8") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.378936   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\") pod \"10d85e4d-b0d8-4823-b20d-cda28e641673\" (UID: \"10d85e4d-b0d8-4823-b20d-cda28e641673\") "
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.379022   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName:10d85e4d-b0d8-4823-b20d-cda28e641673 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.879010144 +0100 CET m=+47.665815935 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "10d85e4d-b0d8-4823-b20d-cda28e641673" (UID: "10d85e4d-b0d8-4823-b20d-cda28e641673") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.379211   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.379311   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.879302654 +0100 CET m=+47.666108445 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.379642   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.879634338 +0100 CET m=+47.666440139 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "nextcloud-nextcloud-cron-28387640-kvwwv" (UID: "485f2017-fcab-46d8-a0f8-71bf110c1836") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.379667   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.879654826 +0100 CET m=+47.666460627 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.379692   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.879683029 +0100 CET m=+47.666488830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.379710   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.879701153 +0100 CET m=+47.666506944 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "nextcloud-nginx-b674d44dd-vwfct" (UID: "faa24a36-35eb-4764-8ab1-d2630b99b684") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.379723   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.879714629 +0100 CET m=+47.666520420 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.379737   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.87973145 +0100 CET m=+47.666537241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.379748   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.879741609 +0100 CET m=+47.666547400 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2") pod "cloudflareddns-56c9c85bb8-h822b" (UID: "fec4f553-b289-49fe-ab03-9284189e5216") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.480594   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.480676   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.480731   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.980694269 +0100 CET m=+47.767500060 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.480754   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.980739013 +0100 CET m=+47.767544804 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.480773   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-80edd197-58be-4fb7-8420-25e736120a31\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.480836   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2\") pod \"4b670339-8e2a-4556-bbff-b6e9b0a720c6\" (UID: \"4b670339-8e2a-4556-bbff-b6e9b0a720c6\") "
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.480858   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.980844712 +0100 CET m=+47.767650503 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.480882   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-eaf1e5ab-9a06-441a-b138-70749798101f\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.480929   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2 podName:4b670339-8e2a-4556-bbff-b6e9b0a720c6 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.980911438 +0100 CET m=+47.767717229 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2") pod "4b670339-8e2a-4556-bbff-b6e9b0a720c6" (UID: "4b670339-8e2a-4556-bbff-b6e9b0a720c6") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.480949   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\") pod \"26601002-2b7f-44d7-b247-12092d0a23c8\" (UID: \"26601002-2b7f-44d7-b247-12092d0a23c8\") "
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.480956   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.98094458 +0100 CET m=+47.767750371 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.480992   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName:26601002-2b7f-44d7-b247-12092d0a23c8 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.98098174 +0100 CET m=+47.767787531 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "26601002-2b7f-44d7-b247-12092d0a23c8" (UID: "26601002-2b7f-44d7-b247-12092d0a23c8") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.481278   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\") pod \"10d85e4d-b0d8-4823-b20d-cda28e641673\" (UID: \"10d85e4d-b0d8-4823-b20d-cda28e641673\") "
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.481384   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName:10d85e4d-b0d8-4823-b20d-cda28e641673 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.981362877 +0100 CET m=+47.768168678 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "10d85e4d-b0d8-4823-b20d-cda28e641673" (UID: "10d85e4d-b0d8-4823-b20d-cda28e641673") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.481825   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.981817682 +0100 CET m=+47.768623473 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "nextcloud-preview-cron-28387720-dq7tq" (UID: "f625e3e8-e7a8-4439-8d8c-6f10b1c7978a") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.481946   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.981936856 +0100 CET m=+47.768742647 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.481962   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.981954008 +0100 CET m=+47.768759789 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "nextcloud-c9c757d4f-cwjcs" (UID: "2f7da9b0-b444-4bc5-934c-cf0c339d084e") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.481977   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.98197094 +0100 CET m=+47.768776731 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.481987   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.981982512 +0100 CET m=+47.768788303 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2") pod "cloudflareddns-56c9c85bb8-h822b" (UID: "fec4f553-b289-49fe-ab03-9284189e5216") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.481998   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.98199228 +0100 CET m=+47.768798071 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.482008   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:09.98200289 +0100 CET m=+47.768808681 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.582666   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2\") pod \"4b670339-8e2a-4556-bbff-b6e9b0a720c6\" (UID: \"4b670339-8e2a-4556-bbff-b6e9b0a720c6\") "
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.582719   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-eaf1e5ab-9a06-441a-b138-70749798101f\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.582764   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\") pod \"26601002-2b7f-44d7-b247-12092d0a23c8\" (UID: \"26601002-2b7f-44d7-b247-12092d0a23c8\") "
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.582822   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2 podName:4b670339-8e2a-4556-bbff-b6e9b0a720c6 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:10.08277998 +0100 CET m=+47.869585771 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2") pod "4b670339-8e2a-4556-bbff-b6e9b0a720c6" (UID: "4b670339-8e2a-4556-bbff-b6e9b0a720c6") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.582971   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName:26601002-2b7f-44d7-b247-12092d0a23c8 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:10.08295576 +0100 CET m=+47.869761551 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "26601002-2b7f-44d7-b247-12092d0a23c8" (UID: "26601002-2b7f-44d7-b247-12092d0a23c8") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.582988   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:10.082978343 +0100 CET m=+47.869784134 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.583094   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\") pod \"10d85e4d-b0d8-4823-b20d-cda28e641673\" (UID: \"10d85e4d-b0d8-4823-b20d-cda28e641673\") "
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.583196   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName:10d85e4d-b0d8-4823-b20d-cda28e641673 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:10.083180182 +0100 CET m=+47.869985963 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "10d85e4d-b0d8-4823-b20d-cda28e641673" (UID: "10d85e4d-b0d8-4823-b20d-cda28e641673") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.583219   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.583257   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.583296   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-80edd197-58be-4fb7-8420-25e736120a31\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.583318   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:10.083300689 +0100 CET m=+47.870106480 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.583362   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:10.083347377 +0100 CET m=+47.870153168 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.583426   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:10.083416337 +0100 CET m=+47.870222128 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.583616   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:10.083608528 +0100 CET m=+47.870414319 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.583629   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:10.083624779 +0100 CET m=+47.870430570 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2") pod "cloudflareddns-56c9c85bb8-h822b" (UID: "fec4f553-b289-49fe-ab03-9284189e5216") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.583657   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:10.083649235 +0100 CET m=+47.870455026 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.583690   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:10.083676806 +0100 CET m=+47.870482597 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.583709   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:10.083701363 +0100 CET m=+47.870507154 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "nextcloud-preview-cron-28387720-dq7tq" (UID: "f625e3e8-e7a8-4439-8d8c-6f10b1c7978a") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.583719   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:10.083713275 +0100 CET m=+47.870519066 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.583727   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:10.083722482 +0100 CET m=+47.870528273 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "nextcloud-nextcloud-cron-28387640-kvwwv" (UID: "485f2017-fcab-46d8-a0f8-71bf110c1836") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.684846   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.684933   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.685000   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:10.18495579 +0100 CET m=+47.971761581 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.685055   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-80edd197-58be-4fb7-8420-25e736120a31\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.685082   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:10.185057641 +0100 CET m=+47.971863432 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.685112   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName:e8875c9f-4ec0-4bae-a61d-1141b489b4f2 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:10.185092667 +0100 CET m=+47.971898458 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.685146   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2\") pod \"4b670339-8e2a-4556-bbff-b6e9b0a720c6\" (UID: \"4b670339-8e2a-4556-bbff-b6e9b0a720c6\") "
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.685211   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-eaf1e5ab-9a06-441a-b138-70749798101f\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.685268   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2 podName:4b670339-8e2a-4556-bbff-b6e9b0a720c6 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:10.185240976 +0100 CET m=+47.972046767 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2") pod "4b670339-8e2a-4556-bbff-b6e9b0a720c6" (UID: "4b670339-8e2a-4556-bbff-b6e9b0a720c6") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.685308   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\") pod \"26601002-2b7f-44d7-b247-12092d0a23c8\" (UID: \"26601002-2b7f-44d7-b247-12092d0a23c8\") "
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.685333   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName:73073547-35f0-4886-95be-5b91f06ad1ba nodeName:}" failed. No retries permitted until 2023-12-22 17:43:10.185310397 +0100 CET m=+47.972116368 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.685407   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName:26601002-2b7f-44d7-b247-12092d0a23c8 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:10.185389165 +0100 CET m=+47.972194956 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "26601002-2b7f-44d7-b247-12092d0a23c8" (UID: "26601002-2b7f-44d7-b247-12092d0a23c8") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.685689   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\") pod \"10d85e4d-b0d8-4823-b20d-cda28e641673\" (UID: \"10d85e4d-b0d8-4823-b20d-cda28e641673\") "
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.685817   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName:10d85e4d-b0d8-4823-b20d-cda28e641673 nodeName:}" failed. No retries permitted until 2023-12-22 17:43:10.185792724 +0100 CET m=+47.972598515 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "10d85e4d-b0d8-4823-b20d-cda28e641673" (UID: "10d85e4d-b0d8-4823-b20d-cda28e641673") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.686441   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:10.186432026 +0100 CET m=+47.973237817 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.686455   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:10.186450491 +0100 CET m=+47.973256282 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.686465   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:10.1864605 +0100 CET m=+47.973266291 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "nextcloud-nginx-b674d44dd-vwfct" (UID: "faa24a36-35eb-4764-8ab1-d2630b99b684") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.686496   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:10.186485507 +0100 CET m=+47.973291298 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-80edd197-58be-4fb7-8420-25e736120a31" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "scrutiny-597cdf895b-rsnv7" (UID: "47540f18-cb93-4b89-b4c4-a6f01fe502a8") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.686565   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:10.186553875 +0100 CET m=+47.973359666 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2") pod "cloudflareddns-56c9c85bb8-h822b" (UID: "fec4f553-b289-49fe-ab03-9284189e5216") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.686582   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:10.186572641 +0100 CET m=+47.973378432 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "nextcloud-nginx-b674d44dd-vwfct" (UID: "faa24a36-35eb-4764-8ab1-d2630b99b684") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: E1222 17:43:09.686595   16911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36 podName: nodeName:}" failed. No retries permitted until 2023-12-22 17:43:10.186588731 +0100 CET m=+47.973394522 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "adguard-home-57b89dd8b8-xzj2m" (UID: "015c0b2a-db07-4719-b416-3506b105713d") : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.701311   16911 csi_plugin.go:99] kubernetes.io/csi: Trying to validate a new CSI Driver with name: zfs.csi.openebs.io endpoint: /var/lib/kubelet/plugins/zfs-localpv/csi.sock versions: 1.0.0
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.701344   16911 csi_plugin.go:112] kubernetes.io/csi: Register new plugin with name: zfs.csi.openebs.io at endpoint: /var/lib/kubelet/plugins/zfs-localpv/csi.sock
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.786681   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-eaf1e5ab-9a06-441a-b138-70749798101f\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.786773   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\") pod \"26601002-2b7f-44d7-b247-12092d0a23c8\" (UID: \"26601002-2b7f-44d7-b247-12092d0a23c8\") "
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.787017   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\") pod \"10d85e4d-b0d8-4823-b20d-cda28e641673\" (UID: \"10d85e4d-b0d8-4823-b20d-cda28e641673\") "
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.787277   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.787349   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142\") pod \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\" (UID: \"e8875c9f-4ec0-4bae-a61d-1141b489b4f2\") "
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.787426   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-80edd197-58be-4fb7-8420-25e736120a31\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31\") pod \"73073547-35f0-4886-95be-5b91f06ad1ba\" (UID: \"73073547-35f0-4886-95be-5b91f06ad1ba\") "
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.787512   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2\") pod \"4b670339-8e2a-4556-bbff-b6e9b0a720c6\" (UID: \"4b670339-8e2a-4556-bbff-b6e9b0a720c6\") "
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.800677   16911 operation_generator.go:900] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/zfs.csi.openebs.io^pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142" (OuterVolumeSpecName: "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2"). InnerVolumeSpecName "pvc-84aa69e0-a218-4f4e-8ebd-0af31c21d142". PluginName "kubernetes.io/csi", VolumeGidValue ""
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.800712   16911 operation_generator.go:900] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/zfs.csi.openebs.io^pvc-eaf1e5ab-9a06-441a-b138-70749798101f" (OuterVolumeSpecName: "pvc-eaf1e5ab-9a06-441a-b138-70749798101f") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba"). InnerVolumeSpecName "pvc-eaf1e5ab-9a06-441a-b138-70749798101f". PluginName "kubernetes.io/csi", VolumeGidValue ""
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.800778   16911 operation_generator.go:900] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/zfs.csi.openebs.io^pvc-80edd197-58be-4fb7-8420-25e736120a31" (OuterVolumeSpecName: "pvc-80edd197-58be-4fb7-8420-25e736120a31") pod "73073547-35f0-4886-95be-5b91f06ad1ba" (UID: "73073547-35f0-4886-95be-5b91f06ad1ba"). InnerVolumeSpecName "pvc-80edd197-58be-4fb7-8420-25e736120a31". PluginName "kubernetes.io/csi", VolumeGidValue ""
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.800797   16911 operation_generator.go:900] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (OuterVolumeSpecName: "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "10d85e4d-b0d8-4823-b20d-cda28e641673" (UID: "10d85e4d-b0d8-4823-b20d-cda28e641673"). InnerVolumeSpecName "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850". PluginName "kubernetes.io/csi", VolumeGidValue ""
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.800835   16911 operation_generator.go:900] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/zfs.csi.openebs.io^pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2" (OuterVolumeSpecName: "pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2") pod "4b670339-8e2a-4556-bbff-b6e9b0a720c6" (UID: "4b670339-8e2a-4556-bbff-b6e9b0a720c6"). InnerVolumeSpecName "pvc-d8d1caa3-c5b1-445f-b4e6-33f6fffa04b2". PluginName "kubernetes.io/csi", VolumeGidValue ""
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.800881   16911 operation_generator.go:900] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (OuterVolumeSpecName: "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "26601002-2b7f-44d7-b247-12092d0a23c8" (UID: "26601002-2b7f-44d7-b247-12092d0a23c8"). InnerVolumeSpecName "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3". PluginName "kubernetes.io/csi", VolumeGidValue ""
Dec 22 17:43:09 truenas k3s[16911]: I1222 17:43:09.800885   16911 operation_generator.go:900] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/zfs.csi.openebs.io^pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36" (OuterVolumeSpecName: "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36") pod "e8875c9f-4ec0-4bae-a61d-1141b489b4f2" (UID: "e8875c9f-4ec0-4bae-a61d-1141b489b4f2"). InnerVolumeSpecName "pvc-f19324b4-f2b4-4d90-9ab8-4ddea3b01b36". PluginName "kubernetes.io/csi", VolumeGidValue ""
Dec 22 17:43:10 truenas k3s[16911]: I1222 17:43:10.274564   16911 job_controller.go:514] enqueueing job ix-nextcloud/nextcloud-preview-cron-28387720
Dec 22 17:43:12 truenas k3s[16911]: I1222 17:43:12.063653   16911 trace.go:219] Trace[364808929]: "Create" accept:application/vnd.kubernetes.protobuf,application/json,audit-id:3f42ea6f-ff68-442c-9f13-59fd4c131550,client:127.0.0.1,protocol:HTTP/2.0,resource:events,scope:resource,url:/api/v1/namespaces/kube-system/events,user-agent:k3s/v1.26.6+k3s (linux/amd64) kubernetes/e18037a,verb:POST (22-Dec-2023 17:43:11.362) (total time: 701ms):
Dec 22 17:43:12 truenas k3s[16911]: Trace[364808929]: ["Create etcd3" audit-id:3f42ea6f-ff68-442c-9f13-59fd4c131550,key:/events/kube-system/snapshot-controller-546868dfb4-sbhwg.17a3350312d6242b,type:*core.Event,resource:events 700ms (17:43:11.362)
Dec 22 17:43:12 truenas k3s[16911]: Trace[364808929]:  ---"Txn call succeeded" 700ms (17:43:12.063)]
Dec 22 17:43:12 truenas k3s[16911]: Trace[364808929]: [701.32618ms] [701.32618ms] END
Dec 22 17:43:12 truenas k3s[16911]: I1222 17:43:12.080784   16911 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="kube-system/svclb-scrutiny-1973f61e-r29pw" podStartSLOduration=35.080504123 pod.CreationTimestamp="2023-12-22 17:42:37 +0100 CET" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2023-12-22 17:43:12.078887122 +0100 CET m=+49.865692913" watchObservedRunningTime="2023-12-22 17:43:12.080504123 +0100 CET m=+49.867309924"
Dec 22 17:43:12 truenas k3s[16911]: I1222 17:43:12.096789   16911 event.go:294] "Event occurred" object="ix-scrutiny/scrutiny" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="UpdatedLoadBalancer" message="Updated LoadBalancer with new IPs: [] -> [192.168.1.100]"
Dec 22 17:43:12 truenas k3s[16911]: I1222 17:43:12.476380   16911 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="kube-system/svclb-adguard-home-dns-crypt-72069a50-5gs2r" podStartSLOduration=34.476334383 pod.CreationTimestamp="2023-12-22 17:42:38 +0100 CET" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2023-12-22 17:43:12.474458816 +0100 CET m=+50.261264607" watchObservedRunningTime="2023-12-22 17:43:12.476334383 +0100 CET m=+50.263140174"
Dec 22 17:43:12 truenas k3s[16911]: I1222 17:43:12.481962   16911 event.go:294] "Event occurred" object="ix-adguard-home/adguard-home-dns-crypt" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="UpdatedLoadBalancer" message="Updated LoadBalancer with new IPs: [] -> [192.168.1.100]"
Dec 22 17:43:12 truenas k3s[16911]: I1222 17:43:12.874892   16911 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="kube-system/svclb-steam-headless-30ea5efc-v7vq7" podStartSLOduration=34.874857788 pod.CreationTimestamp="2023-12-22 17:42:38 +0100 CET" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2023-12-22 17:43:12.873077289 +0100 CET m=+50.659883080" watchObservedRunningTime="2023-12-22 17:43:12.874857788 +0100 CET m=+50.661663589"
Dec 22 17:43:13 truenas k3s[16911]: I1222 17:43:13.274622   16911 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="kube-system/openebs-zfs-node-59gxn" podStartSLOduration=36.274581701 pod.CreationTimestamp="2023-12-22 17:42:37 +0100 CET" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2023-12-22 17:43:13.274109954 +0100 CET m=+51.060915745" watchObservedRunningTime="2023-12-22 17:43:13.274581701 +0100 CET m=+51.061387512"
Dec 22 17:43:13 truenas k3s[16911]: I1222 17:43:13.941376   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\") pod \"26601002-2b7f-44d7-b247-12092d0a23c8\" (UID: \"26601002-2b7f-44d7-b247-12092d0a23c8\") "
Dec 22 17:43:13 truenas k3s[16911]: I1222 17:43:13.948457   16911 operation_generator.go:900] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (OuterVolumeSpecName: "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "26601002-2b7f-44d7-b247-12092d0a23c8" (UID: "26601002-2b7f-44d7-b247-12092d0a23c8"). InnerVolumeSpecName "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850". PluginName "kubernetes.io/csi", VolumeGidValue ""
Dec 22 17:43:14 truenas k3s[16911]: I1222 17:43:14.467970   16911 request.go:690] Waited for 1.037750061s due to client-side throttling, not priority and fairness, request: GET:https://127.0.0.1:6443/api/v1/namespaces/ix-scrutiny/persistentvolumeclaims/scrutiny-influxdb
Dec 22 17:43:14 truenas k3s[16911]: I1222 17:43:14.750517   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\") pod \"10d85e4d-b0d8-4823-b20d-cda28e641673\" (UID: \"10d85e4d-b0d8-4823-b20d-cda28e641673\") "
Dec 22 17:43:14 truenas k3s[16911]: I1222 17:43:14.760807   16911 operation_generator.go:900] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (OuterVolumeSpecName: "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "10d85e4d-b0d8-4823-b20d-cda28e641673" (UID: "10d85e4d-b0d8-4823-b20d-cda28e641673"). InnerVolumeSpecName "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3". PluginName "kubernetes.io/csi", VolumeGidValue ""
Dec 22 17:43:15 truenas k3s[16911]: I1222 17:43:15.076053   16911 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="kube-system/svclb-adguard-home-759b4b17-7vjvv" podStartSLOduration=38.076012249 pod.CreationTimestamp="2023-12-22 17:42:37 +0100 CET" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2023-12-22 17:43:15.075104171 +0100 CET m=+52.861909962" watchObservedRunningTime="2023-12-22 17:43:15.076012249 +0100 CET m=+52.862818040"
Dec 22 17:43:15 truenas k3s[16911]: I1222 17:43:15.085641   16911 event.go:294] "Event occurred" object="ix-adguard-home/adguard-home" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="UpdatedLoadBalancer" message="Updated LoadBalancer with new IPs: [] -> [192.168.1.100]"
Dec 22 17:43:15 truenas k3s[16911]: I1222 17:43:15.366120   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\") pod \"6a63fe43-d25f-4de3-a374-63d86f8f3603\" (UID: \"6a63fe43-d25f-4de3-a374-63d86f8f3603\") "
Dec 22 17:43:15 truenas k3s[16911]: I1222 17:43:15.376555   16911 operation_generator.go:900] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (OuterVolumeSpecName: "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "6a63fe43-d25f-4de3-a374-63d86f8f3603" (UID: "6a63fe43-d25f-4de3-a374-63d86f8f3603"). InnerVolumeSpecName "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850". PluginName "kubernetes.io/csi", VolumeGidValue ""
Dec 22 17:43:15 truenas k3s[16911]: I1222 17:43:15.567881   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\") pod \"6a63fe43-d25f-4de3-a374-63d86f8f3603\" (UID: \"6a63fe43-d25f-4de3-a374-63d86f8f3603\") "
Dec 22 17:43:15 truenas k3s[16911]: I1222 17:43:15.577555   16911 operation_generator.go:900] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (OuterVolumeSpecName: "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "6a63fe43-d25f-4de3-a374-63d86f8f3603" (UID: "6a63fe43-d25f-4de3-a374-63d86f8f3603"). InnerVolumeSpecName "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3". PluginName "kubernetes.io/csi", VolumeGidValue ""
Dec 22 17:43:16 truenas k3s[16911]: I1222 17:43:16.074260   16911 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="kube-system/amdgpu-device-plugin-daemonset-5z6vv" podStartSLOduration=39.07422578 pod.CreationTimestamp="2023-12-22 17:42:37 +0100 CET" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2023-12-22 17:43:16.072824715 +0100 CET m=+53.859630506" watchObservedRunningTime="2023-12-22 17:43:16.07422578 +0100 CET m=+53.861031571"
Dec 22 17:43:16 truenas k3s[16911]: I1222 17:43:16.273952   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\") pod \"971c5771-2d79-4611-96cf-065af145aa11\" (UID: \"971c5771-2d79-4611-96cf-065af145aa11\") "
Dec 22 17:43:16 truenas k3s[16911]: I1222 17:43:16.280523   16911 operation_generator.go:900] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (OuterVolumeSpecName: "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3") pod "971c5771-2d79-4611-96cf-065af145aa11" (UID: "971c5771-2d79-4611-96cf-065af145aa11"). InnerVolumeSpecName "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3". PluginName "kubernetes.io/csi", VolumeGidValue ""
Dec 22 17:43:16 truenas k3s[16911]: I1222 17:43:16.476046   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\") pod \"971c5771-2d79-4611-96cf-065af145aa11\" (UID: \"971c5771-2d79-4611-96cf-065af145aa11\") "
Dec 22 17:43:16 truenas k3s[16911]: I1222 17:43:16.482399   16911 operation_generator.go:900] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (OuterVolumeSpecName: "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850") pod "971c5771-2d79-4611-96cf-065af145aa11" (UID: "971c5771-2d79-4611-96cf-065af145aa11"). InnerVolumeSpecName "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850". PluginName "kubernetes.io/csi", VolumeGidValue ""
Dec 22 17:43:16 truenas k3s[16911]: I1222 17:43:16.648010   16911 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=10d85e4d-b0d8-4823-b20d-cda28e641673 path="/var/lib/kubelet/pods/10d85e4d-b0d8-4823-b20d-cda28e641673/volumes"
Dec 22 17:43:17 truenas k3s[16911]: I1222 17:43:17.274272   16911 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="kube-system/svclb-blocky-dnstcp-ce68d497-gvc2m" podStartSLOduration=39.274246627 pod.CreationTimestamp="2023-12-22 17:42:38 +0100 CET" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2023-12-22 17:43:17.273361141 +0100 CET m=+55.060166932" watchObservedRunningTime="2023-12-22 17:43:17.274246627 +0100 CET m=+55.061052418"
Dec 22 17:43:17 truenas k3s[16911]: I1222 17:43:17.277998   16911 event.go:294] "Event occurred" object="ix-blocky/blocky-dnstcp" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="UpdatedLoadBalancer" message="Updated LoadBalancer with new IPs: [] -> [192.168.1.100]"
Dec 22 17:43:17 truenas k3s[16911]: I1222 17:43:17.674299   16911 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="kube-system/svclb-adguard-home-dns-https-8820d267-ch5vd" podStartSLOduration=38.674274842 pod.CreationTimestamp="2023-12-22 17:42:39 +0100 CET" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2023-12-22 17:43:17.672760574 +0100 CET m=+55.459566365" watchObservedRunningTime="2023-12-22 17:43:17.674274842 +0100 CET m=+55.461080633"
Dec 22 17:43:18 truenas k3s[16911]: I1222 17:43:18.107674   16911 gc_controller.go:207] "Garbage collecting pods" numPods=5
Dec 22 17:43:18 truenas k3s[16911]: I1222 17:43:18.107708   16911 gc_controller.go:329] "PodGC is force deleting Pod" pod="ix-traefik/traefik-5996c85fd6-2gvqx"
Dec 22 17:43:18 truenas k3s[16911]: I1222 17:43:18.107713   16911 gc_controller.go:329] "PodGC is force deleting Pod" pod="ix-prometheus-operator/prometheus-operator-6ffc985f44-wssv5"
Dec 22 17:43:18 truenas k3s[16911]: I1222 17:43:18.107723   16911 gc_controller.go:329] "PodGC is force deleting Pod" pod="ix-cloudflareddns/cloudflareddns-56c9c85bb8-k4krm"
Dec 22 17:43:18 truenas k3s[16911]: I1222 17:43:18.107734   16911 gc_controller.go:329] "PodGC is force deleting Pod" pod="ix-adguard-home/adguard-home-57b89dd8b8-fl825"
Dec 22 17:43:18 truenas k3s[16911]: I1222 17:43:18.107739   16911 gc_controller.go:329] "PodGC is force deleting Pod" pod="ix-scrutiny/scrutiny-597cdf895b-nrwzp"
Dec 22 17:43:18 truenas k3s[16911]: I1222 17:43:18.472797   16911 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="kube-system/svclb-traefik-5aa01911-bmn92" podStartSLOduration=41.472766193 pod.CreationTimestamp="2023-12-22 17:42:37 +0100 CET" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2023-12-22 17:43:18.472625698 +0100 CET m=+56.259431489" watchObservedRunningTime="2023-12-22 17:43:18.472766193 +0100 CET m=+56.259571984"
Dec 22 17:43:18 truenas k3s[16911]: I1222 17:43:18.476979   16911 event.go:294] "Event occurred" object="ix-traefik/traefik" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="UpdatedLoadBalancer" message="Updated LoadBalancer with new IPs: [] -> [192.168.1.100]"
Dec 22 17:43:18 truenas k3s[16911]: I1222 17:43:18.647155   16911 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=4b670339-8e2a-4556-bbff-b6e9b0a720c6 path="/var/lib/kubelet/pods/4b670339-8e2a-4556-bbff-b6e9b0a720c6/volumes"
Dec 22 17:43:18 truenas k3s[16911]: I1222 17:43:18.648238   16911 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=73073547-35f0-4886-95be-5b91f06ad1ba path="/var/lib/kubelet/pods/73073547-35f0-4886-95be-5b91f06ad1ba/volumes"
Dec 22 17:43:18 truenas k3s[16911]: I1222 17:43:18.649344   16911 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=9be3c85d-e167-4ef5-b617-40608b1afc86 path="/var/lib/kubelet/pods/9be3c85d-e167-4ef5-b617-40608b1afc86/volumes"
Dec 22 17:43:18 truenas k3s[16911]: I1222 17:43:18.650352   16911 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=bf926953-81a8-4a14-b6ec-993883c5ef1b path="/var/lib/kubelet/pods/bf926953-81a8-4a14-b6ec-993883c5ef1b/volumes"
Dec 22 17:43:18 truenas k3s[16911]: I1222 17:43:18.651348   16911 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=e8875c9f-4ec0-4bae-a61d-1141b489b4f2 path="/var/lib/kubelet/pods/e8875c9f-4ec0-4bae-a61d-1141b489b4f2/volumes"
Dec 22 17:43:18 truenas k3s[16911]: I1222 17:43:18.874276   16911 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="kube-system/openebs-zfs-controller-0" podStartSLOduration=40.874248459 pod.CreationTimestamp="2023-12-22 17:42:38 +0100 CET" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2023-12-22 17:43:18.873628283 +0100 CET m=+56.660434094" watchObservedRunningTime="2023-12-22 17:43:18.874248459 +0100 CET m=+56.661054260"
Dec 22 17:43:19 truenas k3s[16911]: I1222 17:43:19.673987   16911 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="kube-system/svclb-adguard-home-dns-crypt-udp-89357ae0-bf2xp" podStartSLOduration=15.673961331 pod.CreationTimestamp="2023-12-22 17:43:04 +0100 CET" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2023-12-22 17:43:19.672608336 +0100 CET m=+57.459414147" watchObservedRunningTime="2023-12-22 17:43:19.673961331 +0100 CET m=+57.460767122"
Dec 22 17:43:20 truenas k3s[16911]: time="2023-12-22T17:43:20+01:00" level=info msg="Slow SQL (started: 2023-12-22 17:43:18.964613973 +0100 CET m=+56.751419764) (total time: 1.339153708s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values(?, ?, ?, ?, ?, ?, ?, ?) : [[/registry/events/ix-nextcloud/nextcloud-c9c757d4f-cwjcs.17a335042e0dce10 0 0 9538367 9538367 3660 [107 56 115 0 10 11 10 2 118 49 18 5 69 118 101 110 116 18 215 5 10 197 2 10 42 110 101 120 116 99 108 111 117 100 45 99 57 99 55 53 55 100 52 102 45 99 119 106 99 115 46 49 55 97 51 51 53 48 52 50 101 48 100 99 101 49 48 18 0 26 12 105 120 45 110 101 120 116 99 108 111 117 100 34 0 42 36 50 101 54 52 54 54 99 101 45 101 57 55 57 45 52 48 56 98 45 56 97 54 102 45 55 54 100 55 97 50 53 52 99 48 48 98 50 0 56 0 66 8 8 166 249 150 172 6 16 0 138 1 207 1 10 3 107 51 115 18 6 85 112 100 97 116 101 26 2 118 49 34 8 8 166 249 150 172 6 16 0 50 8 70 105 101 108 100 115 86 49 58 165 1 10 162 1 123 34 102 58 99 111 117 110 116 34 58 123 125 44 34 102 58 102 105 114 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 105 110 118 111 108 118 101 100 79 98 106 101 99 116 34 58 123 125 44 34 102 58 108 97 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 109 101 115 115 97 103 101 34 58 123 125 44 34 102 58 114 101 97 115 111 110 34 58 123 125 44 34 102 58 115 111 117 114 99 101 34 58 123 34 102 58 99 111 109 112 111 110 101 110 116 34 58 123 125 44 34 102 58 104 111 115 116 34 58 123 125 125 44 34 102 58 116 121 112 101 34 58 123 125 125 66 0 18 99 10 3 80 111 100 18 12 105 120 45 110 101 120 116 99 108 111 117 100 26 25 110 101 120 116 99 108 111 117 100 45 99 57 99 55 53 55 100 52 102 45 99 119 106 99 115 34 36 50 102 55 100 97 57 98 48 45 98 52 52 52 45 52 98 99 53 45 57 51 52 99 45 99 102 48 99 51 51 57 100 48 56 52 101 42 2 118 49 50 7 57 53 51 56 48 53 51 58 0 26 11 70 97 105 108 101 100 77 111 117 110 116 34 220 1 77 111 117 110 116 86 111 108 117 109 101 46 83 101 116 85 112 32 102 97 105 108 101 100 32 102 111 114 32 118 111 108 117 109 101 32 34 112 118 99 45 100 101 53 102 49 50 55 102 45 99 99 54 57 45 52 57 48 99 45 98 51 98 55 45 101 98 98 100 51 52 52 57 50 56 53 48 34 32 58 32 107 117 98 101 114 110 101 116 101 115 46 105 111 47 99 115 105 58 32 109 111 117 110 116 101 114 46 83 101 116 85 112 65 116 32 102 97 105 108 101 100 32 116 111 32 103 101 116 32 67 83 73 32 99 108 105 101 110 116 58 32 100 114 105 118 101 114 32 110 97 109 101 32 122 102 115 46 99 115 105 46 111 112 101 110 101 98 115 46 105 111 32 110 111 116 32 102 111 117 110 100 32 105 110 32 116 104 101 32 108 105 115 116 32 111 102 32 114 101 103 105 115 116 101 114 101 100 32 67 83 73 32 100 114 105 118 101 114 115 42 21 10 7 107 117 98 101 108 101 116 18 10 105 120 45 116 114 117 101 110 97 115 50 8 8 144 249 150 172 6 16 0 58 8 8 144 249 150 172 6 16 0 64 2 74 7 87 97 114 110 105 110 103 82 0 98 0 114 0 122 0 26 0 34 0] [107 56 115 0 10 11 10 2 118 49 18 5 69 118 101 110 116 18 215 5 10 197 2 10 42 110 101 120 116 99 108 111 117 100 45 99 57 99 55 53 55 100 52 102 45 99 119 106 99 115 46 49 55 97 51 51 53 48 52 50 101 48 100 99 101 49 48 18 0 26 12 105 120 45 110 101 120 116 99 108 111 117 100 34 0 42 36 50 101 54 52 54 54 99 101 45 101 57 55 57 45 52 48 56 98 45 56 97 54 102 45 55 54 100 55 97 50 53 52 99 48 48 98 50 0 56 0 66 8 8 166 249 150 172 6 16 0 138 1 207 1 10 3 107 51 115 18 6 85 112 100 97 116 101 26 2 118 49 34 8 8 166 249 150 172 6 16 0 50 8 70 105 101 108 100 115 86 49 58 165 1 10 162 1 123 34 102 58 99 111 117 110 116 34 58 123 125 44 34 102 58 102 105 114 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 105 110 118 111 108 118 101 100 79 98 106 101 99 116 34 58 123 125 44 34 102 58 108 97 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 109 101 115 115 97 103 101 34 58 123 125 44 34 102 58 114 101 97 115 111 110 34 58 123 125 44 34 102 58 115 111 117 114 99 101 34 58 123 34 102 58 99 111 109 112 111 110 101 110 116 34 58 123 125 44 34 102 58 104 111 115 116 34 58 123 125 125 44 34 102 58 116 121 112 101 34 58 123 125 125 66 0 18 99 10 3 80 111 100 18 12 105 120 45 110 101 120 116 99 108 111 117 100 26 25 110 101 120 116 99 108 111 117 100 45 99 57 99 55 53 55 100 52 102 45 99 119 106 99 115 34 36 50 102 55 100 97 57 98 48 45 98 52 52 52 45 52 98 99 53 45 57 51 52 99 45 99 102 48 99 51 51 57 100 48 56 52 101 42 2 118 49 50 7 57 53 51 56 48 53 51 58 0 26 11 70 97 105 108 101 100 77 111 117 110 116 34 220 1 77 111 117 110 116 86 111 108 117 109 101 46 83 101 116 85 112 32 102 97 105 108 101 100 32 102 111 114 32 118 111 108 117 109 101 32 34 112 118 99 45 100 101 53 102 49 50 55 102 45 99 99 54 57 45 52 57 48 99 45 98 51 98 55 45 101 98 98 100 51 52 52 57 50 56 53 48 34 32 58 32 107 117 98 101 114 110 101 116 101 115 46 105 111 47 99 115 105 58 32 109 111 117 110 116 101 114 46 83 101 116 85 112 65 116 32 102 97 105 108 101 100 32 116 111 32 103 101 116 32 67 83 73 32 99 108 105 101 110 116 58 32 100 114 105 118 101 114 32 110 97 109 101 32 122 102 115 46 99 115 105 46 111 112 101 110 101 98 115 46 105 111 32 110 111 116 32 102 111 117 110 100 32 105 110 32 116 104 101 32 108 105 115 116 32 111 102 32 114 101 103 105 115 116 101 114 101 100 32 67 83 73 32 100 114 105 118 101 114 115 42 21 10 7 107 117 98 101 108 101 116 18 10 105 120 45 116 114 117 101 110 97 115 50 8 8 144 249 150 172 6 16 0 58 8 8 144 249 150 172 6 16 0 64 1 74 7 87 97 114 110 105 110 103 82 0 98 0 114 0 122 0 26 0 34 0]]]"
Dec 22 17:43:20 truenas k3s[16911]: I1222 17:43:20.304468   16911 trace.go:219] Trace[223043399]: "Patch" accept:application/vnd.kubernetes.protobuf,application/json,audit-id:9984c02e-9906-4f7b-b3d7-086f22030b41,client:127.0.0.1,protocol:HTTP/2.0,resource:events,scope:resource,url:/api/v1/namespaces/ix-nextcloud/events/nextcloud-c9c757d4f-cwjcs.17a335042e0dce10,user-agent:k3s/v1.26.6+k3s (linux/amd64) kubernetes/e18037a,verb:PATCH (22-Dec-2023 17:43:18.962) (total time: 1341ms):
Dec 22 17:43:20 truenas k3s[16911]: Trace[223043399]: ["GuaranteedUpdate etcd3" audit-id:9984c02e-9906-4f7b-b3d7-086f22030b41,key:/events/ix-nextcloud/nextcloud-c9c757d4f-cwjcs.17a335042e0dce10,type:*core.Event,resource:events 1341ms (17:43:18.962)
Dec 22 17:43:20 truenas k3s[16911]: Trace[223043399]:  ---"Txn call completed" 1340ms (17:43:20.304)]
Dec 22 17:43:20 truenas k3s[16911]: Trace[223043399]: ---"Object stored in database" 1340ms (17:43:20.304)
Dec 22 17:43:20 truenas k3s[16911]: Trace[223043399]: [1.341738508s] [1.341738508s] END
Dec 22 17:43:20 truenas k3s[16911]: I1222 17:43:20.472716   16911 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="kube-system/snapshot-controller-546868dfb4-hm96j" podStartSLOduration=43.472677357 pod.CreationTimestamp="2023-12-22 17:42:37 +0100 CET" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2023-12-22 17:43:20.472323882 +0100 CET m=+58.259129683" watchObservedRunningTime="2023-12-22 17:43:20.472677357 +0100 CET m=+58.259483138"
Dec 22 17:43:20 truenas k3s[16911]: I1222 17:43:20.474173   16911 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="ix-cert-manager/cert-manager-certmanager-cainjector-67d89d8b5b-7q24h" podStartSLOduration=4996.474141681 pod.CreationTimestamp="2023-12-22 16:20:04 +0100 CET" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2023-12-22 17:43:20.072493441 +0100 CET m=+57.859299242" watchObservedRunningTime="2023-12-22 17:43:20.474141681 +0100 CET m=+58.260947472"
Dec 22 17:43:20 truenas k3s[16911]: I1222 17:43:20.873693   16911 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="kube-system/svclb-adguard-home-dns-udp-91694a8a-cxjzk" podStartSLOduration=42.873642553 pod.CreationTimestamp="2023-12-22 17:42:38 +0100 CET" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2023-12-22 17:43:20.872453276 +0100 CET m=+58.659259067" watchObservedRunningTime="2023-12-22 17:43:20.873642553 +0100 CET m=+58.660448344"
Dec 22 17:43:20 truenas k3s[16911]: I1222 17:43:20.878490   16911 event.go:294] "Event occurred" object="ix-adguard-home/adguard-home-dns-udp" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="UpdatedLoadBalancer" message="Updated LoadBalancer with new IPs: [] -> [192.168.1.100]"
Dec 22 17:43:21 truenas k3s[16911]: I1222 17:43:21.274228   16911 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="kube-system/csi-smb-controller-7fbbb8fb6f-s7gbv" podStartSLOduration=31.274189211 pod.CreationTimestamp="2023-12-22 17:42:50 +0100 CET" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2023-12-22 17:43:21.273138756 +0100 CET m=+59.059944547" watchObservedRunningTime="2023-12-22 17:43:21.274189211 +0100 CET m=+59.060994992"
Dec 22 17:43:21 truenas k3s[16911]: I1222 17:43:21.531937   16911 trace.go:219] Trace[342170747]: "Create" accept:application/vnd.kubernetes.protobuf,application/json,audit-id:e45b9f00-5e7a-4a82-a908-e1215aa705cd,client:127.0.0.1,protocol:HTTP/2.0,resource:events,scope:resource,url:/api/v1/namespaces/kube-system/events,user-agent:k3s/v1.26.6+k3s (linux/amd64) kubernetes/e18037a,verb:POST (22-Dec-2023 17:43:20.931) (total time: 600ms):
Dec 22 17:43:21 truenas k3s[16911]: Trace[342170747]: ["Create etcd3" audit-id:e45b9f00-5e7a-4a82-a908-e1215aa705cd,key:/events/kube-system/amdgpu-device-plugin-daemonset-5z6vv.17a335043d2630ca,type:*core.Event,resource:events 599ms (17:43:20.932)
Dec 22 17:43:21 truenas k3s[16911]: Trace[342170747]:  ---"Txn call succeeded" 599ms (17:43:21.531)]
Dec 22 17:43:21 truenas k3s[16911]: Trace[342170747]: [600.035092ms] [600.035092ms] END
Dec 22 17:43:21 truenas k3s[16911]: I1222 17:43:21.674027   16911 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="ix-prometheus-operator/prometheus-operator-6ffc985f44-djq8t" podStartSLOduration=28.673999694 pod.CreationTimestamp="2023-12-22 17:42:53 +0100 CET" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2023-12-22 17:43:21.672792684 +0100 CET m=+59.459598475" watchObservedRunningTime="2023-12-22 17:43:21.673999694 +0100 CET m=+59.460805485"
Dec 22 17:43:22 truenas k3s[16911]: I1222 17:43:22.074197   16911 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="kube-system/snapshot-controller-546868dfb4-sbhwg" podStartSLOduration=45.074152923 pod.CreationTimestamp="2023-12-22 17:42:37 +0100 CET" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2023-12-22 17:43:22.072326378 +0100 CET m=+59.859132179" watchObservedRunningTime="2023-12-22 17:43:22.074152923 +0100 CET m=+59.860958714"
Dec 22 17:43:22 truenas k3s[16911]: I1222 17:43:22.817607   16911 job_controller.go:514] enqueueing job ix-nextcloud/nextcloud-preview-cron-28387720
Dec 22 17:43:22 truenas k3s[16911]: I1222 17:43:22.846400   16911 job_controller.go:514] enqueueing job ix-nextcloud/nextcloud-nextcloud-cron-28387640
Dec 22 17:43:24 truenas k3s[16911]: I1222 17:43:24.076631   16911 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="kube-system/svclb-adguard-home-dns-quic-udp-1-9b503507-h64rw" podStartSLOduration=44.076588696 pod.CreationTimestamp="2023-12-22 17:42:40 +0100 CET" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2023-12-22 17:43:24.076242826 +0100 CET m=+61.863048617" watchObservedRunningTime="2023-12-22 17:43:24.076588696 +0100 CET m=+61.863394487"
Dec 22 17:43:25 truenas k3s[16911]: I1222 17:43:25.274270   16911 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="kube-system/svclb-adguard-home-dns-tcp-36dfd2e8-xdtsg" podStartSLOduration=47.274232106 pod.CreationTimestamp="2023-12-22 17:42:38 +0100 CET" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2023-12-22 17:43:25.272081842 +0100 CET m=+63.058887633" watchObservedRunningTime="2023-12-22 17:43:25.274232106 +0100 CET m=+63.061037907"
Dec 22 17:43:25 truenas k3s[16911]: I1222 17:43:25.277978   16911 event.go:294] "Event occurred" object="ix-adguard-home/adguard-home-dns-tcp" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="UpdatedLoadBalancer" message="Updated LoadBalancer with new IPs: [] -> [192.168.1.100]"
Dec 22 17:43:26 truenas k3s[16911]: I1222 17:43:26.075269   16911 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="ix-cert-manager/cert-manager-certmanager-webhook-688c8b8d4d-wqkjj" podStartSLOduration=4994.07523841 pod.CreationTimestamp="2023-12-22 16:20:12 +0100 CET" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2023-12-22 17:43:26.074262244 +0100 CET m=+63.861068045" watchObservedRunningTime="2023-12-22 17:43:26.07523841 +0100 CET m=+63.862044201"
Dec 22 17:43:26 truenas k3s[16911]: I1222 17:43:26.873593   16911 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="kube-system/svclb-adguard-home-dns-https-udp-98d14874-ck75d" podStartSLOduration=49.873535528 pod.CreationTimestamp="2023-12-22 17:42:37 +0100 CET" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2023-12-22 17:43:26.872854548 +0100 CET m=+64.659660349" watchObservedRunningTime="2023-12-22 17:43:26.873535528 +0100 CET m=+64.660341319"
Dec 22 17:43:26 truenas k3s[16911]: I1222 17:43:26.877454   16911 event.go:294] "Event occurred" object="ix-adguard-home/adguard-home-dns-https-udp" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="UpdatedLoadBalancer" message="Updated LoadBalancer with new IPs: [] -> [192.168.1.100]"
Dec 22 17:43:27 truenas k3s[16911]: I1222 17:43:27.676233   16911 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="ix-syncthing/syncthing-599b64585f-czblk" podStartSLOduration=50.676197314 pod.CreationTimestamp="2023-12-22 17:42:37 +0100 CET" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2023-12-22 17:43:27.675253389 +0100 CET m=+65.462059200" watchObservedRunningTime="2023-12-22 17:43:27.676197314 +0100 CET m=+65.463003115"
Dec 22 17:43:28 truenas k3s[16911]: I1222 17:43:28.475070   16911 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="kube-system/svclb-blocky-fb7abddf-mlc58" podStartSLOduration=50.475044257 pod.CreationTimestamp="2023-12-22 17:42:38 +0100 CET" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2023-12-22 17:43:28.47417929 +0100 CET m=+66.260985131" watchObservedRunningTime="2023-12-22 17:43:28.475044257 +0100 CET m=+66.261850048"
Dec 22 17:43:28 truenas k3s[16911]: I1222 17:43:28.479528   16911 event.go:294] "Event occurred" object="ix-blocky/blocky" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="UpdatedLoadBalancer" message="Updated LoadBalancer with new IPs: [] -> [192.168.1.100]"
Dec 22 17:43:28 truenas k3s[16911]: I1222 17:43:28.566301   16911 scope.go:115] "RemoveContainer" containerID="81af381d24f56b6aa25e85ec1fca058d08741efcd770a2ec58661f420c5a2f66"
Dec 22 17:43:29 truenas k3s[16911]: I1222 17:43:29.273478   16911 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="kube-system/svclb-librespeed-c90d5914-lpf9j" podStartSLOduration=51.273446553 pod.CreationTimestamp="2023-12-22 17:42:38 +0100 CET" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2023-12-22 17:43:29.27196694 +0100 CET m=+67.058772741" watchObservedRunningTime="2023-12-22 17:43:29.273446553 +0100 CET m=+67.060252344"
Dec 22 17:43:29 truenas k3s[16911]: I1222 17:43:29.276385   16911 event.go:294] "Event occurred" object="ix-librespeed/librespeed" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="UpdatedLoadBalancer" message="Updated LoadBalancer with new IPs: [] -> [192.168.1.100]"
Dec 22 17:43:30 truenas k3s[16911]: I1222 17:43:30.076319   16911 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="ix-cloudnative-pg/cloudnative-pg-5f58b6fbcd-jpdrw" podStartSLOduration=5001.076290992 pod.CreationTimestamp="2023-12-22 16:20:09 +0100 CET" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2023-12-22 17:43:30.074749583 +0100 CET m=+67.861555384" watchObservedRunningTime="2023-12-22 17:43:30.076290992 +0100 CET m=+67.863096783"
Dec 22 17:43:30 truenas k3s[16911]: time="2023-12-22T17:43:30+01:00" level=info msg="Slow SQL (started: 2023-12-22 17:43:29.274249974 +0100 CET m=+67.061055765) (total time: 1.107650305s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values(?, ?, ?, ?, ?, ?, ?, ?) : [[/registry/daemonsets/kube-system/svclb-librespeed-c90d5914 0 0 9516568 9537808 0 [107 56 115 0 10 20 10 7 97 112 112 115 47 118 49 18 9 68 97 101 109 111 110 83 101 116 18 197 35 10 181 29 10 25 115 118 99 108 98 45 108 105 98 114 101 115 112 101 101 100 45 99 57 48 100 53 57 49 52 18 0 26 11 107 117 98 101 45 115 121 115 116 101 109 34 0 42 36 98 53 54 98 57 57 48 48 45 100 51 98 53 45 52 52 53 50 45 56 56 98 99 45 53 57 49 51 57 48 54 98 55 49 57 100 50 0 56 1 66 8 8 164 148 150 172 6 16 0 90 72 10 28 111 98 106 101 99 116 115 101 116 46 114 105 111 46 99 97 116 116 108 101 46 105 111 47 104 97 115 104 18 40 56 51 57 54 51 56 52 55 52 52 102 51 52 56 55 99 100 102 99 52 98 54 54 52 48 50 98 51 97 52 101 52 51 48 49 51 56 54 52 48 90 49 10 40 115 118 99 99 111 110 116 114 111 108 108 101 114 46 107 51 115 46 99 97 116 116 108 101 46 105 111 47 110 111 100 101 115 101 108 101 99 116 111 114 18 5 102 97 108 115 101 90 49 10 35 115 118 99 99 111 110 116 114 111 108 108 101 114 46 107 51 115 46 99 97 116 116 108 101 46 105 111 47 115 118 99 110 97 109 101 18 10 108 105 98 114 101 115 112 101 101 100 90 57 10 40 115 118 99 99 111 110 116 114 111 108 108 101 114 46 107 51 115 46 99 97 116 116 108 101 46 105 111 47 115 118 99 110 97 109 101 115 112 97 99 101 18 13 105 120 45 108 105 98 114 101 115 112 101 101 100 98 45 10 40 100 101 112 114 101 99 97 116 101 100 46 100 97 101 109 111 110 115 101 116 46 116 101 109 112 108 97 116 101 46 103 101 110 101 114 97 116 105 111 110 18 1 49 98 207 8 10 31 111 98 106 101 99 116 115 101 116 46 114 105 111 46 99 97 116 116 108 101 46 105 111 47 97 112 112 108 105 101 100 18 171 8 72 52 115 73 65 65 65 65 65 65 65 65 47 55 120 85 119 87 55 106 78 104 68 57 108 87 76 79 115 105 122 70 113 117 77 73 54 67 70 73 103 105 74 111 49 122 70 115 98 121 56 76 73 120 105 82 52 53 103 49 82 82 76 107 83 66 115 106 48 76 56 88 108 74 78 85 81 90 49 78 48 77 80 67 66 48 80 107 101 52 47 107 109 122 102 122 66 79 106 85 88 43 83 68 115 103 90 75 81 79 102 67 117 77 48 104 103 98 48 121 69 107 113 52 82 113 113 116 87 82 70 68 65 106 85 120 83 109 83 69 56 103 110 81 71 77 118 73 121 112 111 81 80 50 51 49 78 119 107 79 120 75 108 88 78 104 88 73 114 67 108 86 100 113 121 105 66 105 84 118 55 116 118 118 104 118 122 111 111 100 49 68 67 101 77 50 84 51 55 53 81 120 110 53 50 52 112 56 113 119 82 57 121 68 78 89 69 53 83 103 86 101 85 112 79 67 76 53 75 85 90 119 75 67 74 78 80 89 52 71 122 67 52 66 52 97 108 47 48 86 114 86 70 66 104 114 66 54 86 112 116 69 53 65 89 48 88 54 104 43 47 99 89 100 104 66 67 98 80 74 120 88 81 121 75 56 54 76 89 106 115 112 90 117 100 67 98 107 86 82 84 97 100 70 100 108 90 78 115 75 66 105 107 117 87 84 50 98 84 73 73 73 72 81 67 109 69 78 101 54 115 49 43 88 81 47 67 81 77 49 89 121 85 70 48 105 84 89 101 105 104 104 105 122 114 81 66 53 84 81 105 108 78 117 102 73 76 121 114 104 51 80 101 113 69 86 117 104 112 115 106 99 82 70 74 110 43 57 121 65 115 52 81 108 55 52 43 54 97 105 85 84 103 69 112 106 113 121 103 121 77 82 72 102 118 51 72 85 57 81 73 52 118 100 110 54 57 109 111 110 77 47 49 79 43 54 66 74 104 113 112 53 71 112 112 119 47 121 57 52 108 97 102 83 84 47 107 119 48 100 101 73 73 78 50 57 111 50 104 112 43 68 102 105 108 69 47 70 114 98 80 82 107 111 43 51 73 110 69 73 57 66 90 99 103 72 75 76 56 57 65 90 109 50 47 51 43 43 49 71 112 53 100 98 43 52 87 54 52 104 103 82 90 49 69 53 102 121 76 77 117 110 48 67 86 118 77 77 118 76 43 101 56 51 113 119 69 113 83 47 118 102 79 66 115 105 114 50 57 87 54 47 118 70 56 109 53 57 78 48 67 117 114 120 98 47 120 88 120 119 90 65 43 54 88 81 119 80 122 77 47 80 48 118 119 56 110 85 122 84 115 55 77 90 100 74 115 69 86 73 48 80 99 99 101 106 69 84 118 121 52 55 49 87 122 112 69 102 54 97 112 115 115 55 82 73 89 50 108 54 122 75 76 82 101 109 71 49 69 103 99 111 52 88 89 55 116 55 122 119 70 77 103 119 118 75 90 84 86 121 77 87 98 110 83 56 83 65 76 79 101 106 55 54 57 87 114 102 119 110 113 71 115 103 99 107 115 76 79 66 51 121 121 56 73 43 77 116 87 50 72 49 105 119 117 98 66 68 119 70 50 51 104 66 77 86 113 120 108 67 81 97 114 47 104 119 90 81 51 84 73 47 101 82 82 73 101 86 48 111 111 86 72 102 77 110 74 90 84 102 89 72 54 122 118 114 43 56 47 110 73 55 104 48 51 88 82 97 85 84 49 72 65 73 103 110 85 89 70 116 103 81 112 56 113 49 82 97 114 99 47 100 98 54 55 43 106 108 48 70 84 111 78 118 48 116 104 118 109 90 68 120 111 87 69 109 67 114 121 98 43 77 54 74 105 103 55 90 89 69 81 119 108 122 117 120 73 55 107 111 50 79 81 50 86 80 48 100 119 52 99 85 98 101 97 107 112 106 71 51 116 68 84 67 70 71 117 115 98 65 53 79 78 89 100 86 71 114 72 48 99 51 106 121 112 119 54 75 118 43 102 121 83 102 50 50 102 107 78 66 112 54 86 47 109 111 99 101 85 86 75 52 72 54 85 107 112 114 119 112 51 82 104 57 79 69 84 101 121 117 120 107 108 107 87 114 70 72 112 111 100 68 116 68 86 50 113 84 73 80 88 47 117 78 52 119 66 54 47 71 113 119 82 97 87 120 48 103 82 108 72 105 102 77 119 85 88 88 108 109 43 119 102 98 99 121 99 116 78 88 85 106 84 101 107 43 70 53 85 49 102 107 88 120 52 113 111 99 119 83 107 66 83 85 74 51 108 113 121 47 82 114 88 49 81 73 74 53 97 88 104 80 73 65 90 100 90 49 47 119 81 65 65 80 47 47 103 87 82 87 43 73 73 72 65 65 65 98 30 10 26 111 98 106 101 99 116 115 101 116 46 114 105 111 46 99 97 116 116 108 101 46 105 111 47 105 100 18 0 98 54 10 33 111 98 106 101 99 116 115 101 116 46 114 105 111 46 99 97 116 116 108 101 46 105 111 47 111 119 110 101 114 45 103 118 107 18 17 47 118 49 44 32 75 105 110 100 61 83 101 114 118 105 99 101 98 48 10 34 111 98 106 101 99 116 115 101 116 46 114 105 111 46 99 97 116 116 108 101 46 105 111 47 111 119 110 101 114 45 110 97 109 101 18 10 108 105 98 114 101 115 112 101 101 100 98 56 10 39 111 98 106 101 99 116 115 101 116 46 114 105 111 46 99 97 116 116 108 101 46 105 111 47 111 119 110 101 114 45 110 97 109 101 115 112 97 99 101 18 13 105 120 45 108 105 98 114 101 115 112 101 101 100 138 1 159 14 10 3 107 51 115 18 6 85 112 100 97 116 101 26 7 97 112 112 115 47 118 49 34 8 8 164 148 150 172 6 16 0 50 8 70 105 101 108 100 115 86 49 58 240 13 10 237 13 123 34 102 58 109 101 116 97 100 97 116 97 34 58 123 34 102 58 97 110 110 111 116 97 116 105 111 110 115 34 58 123 34 46 34 58 123 125 44 34 102 58 100 101 112 114 101 99 97 116 101 100 46 100 97 101 109 111 110 115 101 116 46 116 101 109 112 108 97 116 101 46 103 101 110 101 114 97 116 105 111 110 34 58 123 125 44 34 102 58 111 98 106 101 99 116 115 101 116 46 114 105 111 46 99 97 116 116 108 101 46 105 111 47 97 112 112 108 105 101 100 34 58 123 125 44 34 102 58 111 98 106 101 99 116 115 101 116 46 114 105 111 46 99 97 116 116 108 101 46 105 111 47 105 100 34 58 123 125 44 34 102 58 111 98 106 101 99 116 115 101 116 46 114 105 111 46 99 97 116 116 108 101 46 105 111 47 111 119 110 101 114 45 103 118 107 34 58 123 125 44 34 102 58 111 98 106 101 99 116 115 101 116 46 114 105 111 46 99 97 116 116 108 101 46 105 111 47 111 119 110 101 114 45 110 97 109 101 34 58 123 125 44 34 102 58 111 98 106 101 99 116 115 101 116 46 114 105 111 46 99 97 116 116 108 101 46 105 111 47 111 119 110 101 114 45 110 97 109 101 115 112 97 99 101 34 58 123 125 125 44 34 102 58 108 97 98 101 108 115 34 58 123 34 46 34 58 123 125 44 34 102 58 111 98 106 101 99 116 115 101 116 46 114 105 111 46 99 97 116 116 108 101 46 105 111 47 104 97 115 104 34 58 123 125 44 34 102 58 115 118 99 99 111 110 116 114 111 108 108 101 114 46 107 51 115 46 99 97 116 116 108 101 46 105 111 47 110 111 100 101 115 101 108 101 99 116 111 114 34 58 123 125 44 34 102 58 115 118 99 99 111 110 116 114 111 108 108 101 114 46 107 51 115 46 99 97 116 116 108 101 46 105 111 47 115 118 99 110 97 109 101 34 58 123 125 44 34 102 58 115 118 99 99 111 110 116 114 111 108 108 101 114 46 107 51 115 46 99 97 116 116 108 101 46 105 111 47 115 118 99 110 97 109 101 115 112 97 99 101 34 58 123 125 125 125 44 34 102 58 115 112 101 99 34 58 123 34 102 58 114 101 118 105 115 105 111 110 72 105 115 116 111 114 121 76 105 109 105 116 34 58 123 125 44 34 102 58 115 101 108 101 99 116 111 114 34 58 123 125 44 34 102 58 116 101 109 112 108 97 116 101 34 58 123 34 102 58 109 101 116 97 100 97 116 97 34 58 123 34 102 58 108 97 98 101 108 115 34 58 123 34 46 34 58 123 125 44 34 102 58 97 112 112 34 58 123 125 44 34 102 58 115 118 99 99 111 110 116 114 111 108 108 101 114 46 107 51 115 46 99 97 116 116 108 101 46 105 111 47 115 118 99 110 97 109 101 34 58 123 125 44 34 102 58 115 118 99 99 111 110 116 114 111 108 108 101 114 46 107 51 115 46 99 97 116 116 108 101 46 105 111 47 115 118 99 110 97 109 101 115 112 97 99 101 34 58 123 125 125 125 44 34 102 58 115 112 101 99 34 58 123 34 102 58 97 117 116 111 109 111 117 110 116 83 101 114 118 105 99 101 65 99 99 111 117 110 116 84 111 107 101 110 34 58 123 125 44 34 102 58 99 111 110 116 97 105 110 101 114 115 34 58 123 34 107 58 123 92 34 110 97 109 101 92 34 58 92 34 108 98 45 116 99 112 45 49 48 48 49 54 92 34 125 34 58 123 34 46 34 58 123 125 44 34 102 58 101 110 118 34 58 123 34 46 34 58 123 125 44 34 107 58 123 92 34 110 97 109 101 92 34 58 92 34 68 69 83 84 95 73 80 83 92 34 125 34 58 123 34 46 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 118 97 108 117 101 34 58 123 125 125 44 34 107 58 123 92 34 110 97 109 101 92 34 58 92 34 68 69 83 84 95 80 79 82 84 92 34 125 34 58 123 34 46 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 118 97 108 117 101 34 58 123 125 125 44 34 107 58 123 92 34 110 97 109 101 92 34 58 92 34 68 69 83 84 95 80 82 79 84 79 92 34 125 34 58 123 34 46 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 118 97 108 117 101 34 58 123 125 125 44 34 107 58 123 92 34 110 97 109 101 92 34 58 92 34 83 82 67 95 80 79 82 84 92 34 125 34 58 123 34 46 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 118 97 108 117 101 34 58 123 125 125 44 34 107 58 123 92 34 110 97 109 101 92 34 58 92 34 83 82 67 95 82 65 78 71 69 83 92 34 125 34 58 123 34 46 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 118 97 108 117 101 34 58 123 125 125 125 44 34 102 58 105 109 97 103 101 34 58 123 125 44 34 102 58 105 109 97 103 101 80 117 108 108 80 111 108 105 99 121 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 112 111 114 116 115 34 58 123 34 46 34 58 123 125 44 34 107 58 123 92 34 99 111 110 116 97 105 110 101 114 80 111 114 116 92 34 58 49 48 48 49 54 44 92 34 112 114 111 116 111 99 111 108 92 34 58 92 34 84 67 80 92 34 125 34 58 123 34 46 34 58 123 125 44 34 102 58 99 111 110 116 97 105 110 101 114 80 111 114 116 34 58 123 125 44 34 102 58 104 111 115 116 80 111 114 116 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 112 114 111 116 111 99 111 108 34 58 123 125 125 125 44 34 102 58 114 101 115 111 117 114 99 101 115 34 58 123 125 44 34 102 58 115 101 99 117 114 105 116 121 67 111 110 116 101 120 116 34 58 123 34 46 34 58 123 125 44 34 102 58 99 97 112 97 98 105 108 105 116 105 101 115 34 58 123 34 46 34 58 123 125 44 34 102 58 97 100 100 34 58 123 125 125 125 44 34 102 58 116 101 114 109 105 110 97 116 105 111 110 77 101 115 115 97 103 101 80 97 116 104 34 58 123 125 44 34 102 58 116 101 114 109 105 110 97 116 105 111 110 77 101 115 115 97 103 101 80 111 108 105 99 121 34 58 123 125 125 125 44 34 102 58 100 110 115 80 111 108 105 99 121 34 58 123 125 44 34 102 58 114 101 115 116 97 114 116 80 111 108 105 99 121 34 58 123 125 44 34 102 58 115 99 104 101 100 117 108 101 114 78 97 109 101 34 58 123 125 44 34 102 58 115 101 99 117 114 105 116 121 67 111 110 116 101 120 116 34 58 123 34 46 34 58 123 125 44 34 102 58 115 121 115 99 116 108 115 34 58 123 125 125 44 34 102 58 115 101 114 118 105 99 101 65 99 99 111 117 110 116 34 58 123 125 44 34 102 58 115 101 114 118 105 99 101 65 99 99 111 117 110 116 78 97 109 101 34 58 123 125 44 34 102 58 116 101 114 109 105 110 97 116 105 111 110 71 114 97 99 101 80 101 114 105 111 100 83 101 99 111 110 100 115 34 58 123 125 44 34 102 58 116 111 108 101 114 97 116 105 111 110 115 34 58 123 125 125 125 44 34 102 58 117 112 100 97 116 101 83 116 114 97 116 101 103 121 34 58 123 34 102 58 114 111 108 108 105 110 103 85 112 100 97 116 101 34 58 123 34 46 34 58 123 125 44 34 102 58 109 97 120 83 117 114 103 101 34 58 123 125 44 34 102 58 109 97 120 85 110 97 118 97 105 108 97 98 108 101 34 58 123 125 125 44 34 102 58 116 121 112 101 34 58 123 125 125 125 125 66 0 138 1 254 1 10 3 107 51 115 18 6 85 112 100 97 116 101 26 7 97 112 112 115 47 118 49 34 8 8 177 249 150 172 6 16 0 50 8 70 105 101 108 100 115 86 49 58 201 1 10 198 1 123 34 102 58 115 116 97 116 117 115 34 58 123 34 102 58 99 117 114 114 101 110 116 78 117 109 98 101 114 83 99 104 101 100 117 108 101 100 34 58 123 125 44 34 102 58 100 101 115 105 114 101 100 78 117 109 98 101 114 83 99 104 101 100 117 108 101 100 34 58 123 125 44 34 102 58 110 117 109 98 101 114 65 118 97 105 108 97 98 108 101 34 58 123 125 44 34 102 58 110 117 109 98 101 114 77 105 115 115 99 104 101 100 117 108 101 100 34 58 123 125 44 34 102 58 110 117 109 98 101 114 82 101 97 100 121 34 58 123 125 44 34 102 58 111 98 115 101 114 118 101 100 71 101 110 101 114 97 116 105 111 110 34 58 123 125 44 34 102 58 117 112 100 97 116 101 100 78 117 109 98 101 114 83 99 104 101 100 117 108 101 100 34 58 123 125 125 125 66 6 115 116 97 116 117 115 18 248 5 10 34 10 32 10 3 97 112 112 18 25 115 118 99 108 98 45 108 105 98 114 101 115 112 101 101 100 45 99 57 48 100 53 57 49 52 18 170 5 10 160 1 10 0 18 0 26 0 34 0 42 0 50 0 56 0 66 0 90 32 10 3 97 112 112 18 25 115 118 99 108 98 45 108 105 98 114 101 115 112 101 101 100 45 99 57 48 100 53 57 49 52 90 49 10 35 115 118 99 99 111 110 116 114 111 108 108 101 114 46 107 51 115 46 99 97 116 116 108 101 46 105 111 47 115 118 99 110 97 109 101 18 10 108 105 98 114 101 115 112 101 101 100 90 57 10 40 115 118 99 99 111 110 116 114 111 108 108 101 114 46 107 51 115 46 99 97 116 116 108 101 46 105 111 47 115 118 99 110 97 109 101 115 112 97 99 101 18 13 105 120 45 108 105 98 114 101 115 112 101 101 100 18 132 4 18 251 1 10 12 108 98 45 116 99 112 45 49 48 48 49 54 18 25 114 97 110 99 104 101 114 47 107 108 105 112 112 101 114 45 108 98 58 118 48 46 52 46 52 42 0 50 27 10 12 108 98 45 116 99 112 45 49 48 48 49 54 16 160 78 24 160 78 34 3 84 67 80 42 0 58 17 10 8 83 82 67 95 80 79 82 84 18 5 49 48 48 49 54 58 23 10 10 83 82 67 95 82 65 78 71 69 83 18 9 48 46 48 46 48 46 48 47 48 58 17 10 10 68 69 83 84 95 80 82 79 84 79 18 3 84 67 80 58 18 10 9 68 69 83 84 95 80 79 82 84 18 5 49 48 48 49 54 58 25 10 8 68 69 83 84 95 73 80 83 18 13 49 55 50 46 49 55 46 51 54 46 50 50 56 66 0 106 20 47 100 101 118 47 116 101 114 109 105 110 97 116 105 111 110 45 108 111 103 114 12 73 102 78 111 116 80 114 101 115 101 110 116 122 13 10 11 10 9 78 69 84 95 65 68 77 73 78 128 1 0 136 1 0 144 1 0 162 1 4 70 105 108 101 26 6 65 108 119 97 121 115 32 30 50 12 67 108 117 115 116 101 114 70 105 114 115 116 66 5 115 118 99 108 98 74 5 115 118 99 108 98 82 0 88 0 96 0 104 0 114 26 58 24 10 19 110 101 116 46 105 112 118 52 46 105 112 95 102 111 114 119 97 114 100 18 1 49 130 1 0 138 1 0 154 1 17 100 101 102 97 117 108 116 45 115 99 104 101 100 117 108 101 114 168 1 0 178 1 54 10 30 110 111 100 101 45 114 111 108 101 46 107 117 98 101 114 110 101 116 101 115 46 105 111 47 109 97 115 116 101 114 18 6 69 120 105 115 116 115 26 0 34 10 78 111 83 99 104 101 100 117 108 101 178 1 61 10 37 110 111 100 101 45 114 111 108 101 46 107 117 98 101 114 110 101 116 101 115 46 105 111 47 99 111 110 116 114 111 108 45 112 108 97 110 101 18 6 69 120 105 115 116 115 26 0 34 10 78 111 83 99 104 101 100 117 108 101 178 1 32 10 18 67 114 105 116 105 99 97 108 65 100 100 111 110 115 79 110 108 121 18 6 69 120 105 115 116 115 26 0 34 0 194 1 0 26 33 10 13 82 111 108 108 105 110 103 85 112 100 97 116 101 18 16 10 6 8 0 16 1 26 0 18 6 8 0 16 0 26 0 32 0 48 10 26 16 8 1 16 0 24 1 32 1 40 1 48 1 56 1 64 0 26 0 34 0] [107 56 115 0 10 20 10 7 97 112 112 115 47 118 49 18 9 68 97 101 109 111 110 83 101 116 18 199 35 10 183 29 10 25 115 118 99 108 98 45 108 105 98 114 101 115 112 101 101 100 45 99 57 48 100 53 57 49 52 18 0 26 11 107 117 98 101 45 115 121 115 116 101 109 34 0 42 36 98 53 54 98 57 57 48 48 45 100 51 98 53 45 52 52 53 50 45 56 56 98 99 45 53 57 49 51 57 48 54 98 55 49 57 100 50 0 56 1 66 8 8 164 148 150 172 6 16 0 90 72 10 28 111 98 106 101 99 116 115 101 116 46 114 105 111 46 99 97 116 116 108 101 46 105 111 47 104 97 115 104 18 40 56 51 57 54 51 56 52 55 52 52 102 51 52 56 55 99 100 102 99 52 98 54 54 52 48 50 98 51 97 52 101 52 51 48 49 51 56 54 52 48 90 49 10 40 115 118 99 99 111 110 116 114 111 108 108 101 114 46 107 51 115 46 99 97 116 116 108 101 46 105 111 47 110 111 100 101 115 101 108 101 99 116 111 114 18 5 102 97 108 115 101 90 49 10 35 115 118 99 99 111 110 116 114 111 108 108 101 114 46 107 51 115 46 99 97 116 116 108 101 46 105 111 47 115 118 99 110 97 109 101 18 10 108 105 98 114 101 115 112 101 101 100 90 57 10 40 115 118 99 99 111 110 116 114 111 108 108 101 114 46 107 51 115 46 99 97 116 116 108 101 46 105 111 47 115 118 99 110 97 109 101 115 112 97 99 101 18 13 105 120 45 108 105 98 114 101 115 112 101 101 100 98 45 10 40 100 101 112 114 101 99 97 116 101 100 46 100 97 101 109 111 110 115 101 116 46 116 101 109 112 108 97 116 101 46 103 101 110 101 114 97 116 105 111 110 18 1 49 98 207 8 10 31 111 98 106 101 99 116 115 101 116 46 114 105 111 46 99 97 116 116 108 101 46 105 111 47 97 112 112 108 105 101 100 18 171 8 72 52 115 73 65 65 65 65 65 65 65 65 47 55 120 85 119 87 55 106 78 104 68 57 108 87 76 79 115 105 122 70 113 117 77 73 54 67 70 73 103 105 74 111 49 122 70 115 98 121 56 76 73 120 105 82 52 53 103 49 82 82 76 107 83 66 115 106 48 76 56 88 108 74 78 85 81 90 49 78 48 77 80 67 66 48 80 107 101 52 47 107 109 122 102 122 66 79 106 85 88 43 83 68 115 103 90 75 81 79 102 67 117 77 48 104 103 98 48 121 69 107 113 52 82 113 113 116 87 82 70 68 65 106 85 120 83 109 83 69 56 103 110 81 71 77 118 73 121 112 111 81 80 50 51 49 78 119 107 79 120 75 108 88 78 104 88 73 114 67 108 86 100 113 121 105 66 105 84 118 55 116 118 118 104 118 122 111 111 100 49 68 67 101 77 50 84 51 55 53 81 120 110 53 50 52 112 56 113 119 82 57 121 68 78 89 69 53 83 103 86 101 85 112 79 67 76 53 75 85 90 119 75 67 74 78 80 89 52 71 122 67 52 66 52 97 108 47 48 86 114 86 70 66 104 114 66 54 86 112 116 69 53 65 89 48 88 54 104 43 47 99 89 100 104 66 67 98 80 74 120 88 81 121 75 56 54 76 89 106 115 112 90 117 100 67 98 107 86 82 84 97 100 70 100 108 90 78 115 75 66 105 107 117 87 84 50 98 84 73 73 73 72 81 67 109 69 78 101 54 115 49 43 88 81 47 67 81 77 49 89 121 85 70 48 105 84 89 101 105 104 104 105 122 114 81 66 53 84 81 105 108 78 117 102 73 76 121 114 104 51 80 101 113 69 86 117 104 112 115 106 99 82 70 74 110 43 57 121 65 115 52 81 108 55 52 43 54 97 105 85 84 103 69 112 106 113 121 103 121 77 82 72 102 118 51 72 85 57 81 73 52 118 100 110 54 57 109 111 110 77 47 49 79 43 54 66 74 104 113 112 53 71 112 112 119 47 121 57 52 108 97 102 83 84 47 107 119 48 100 101 73 73 78 50 57 111 50 104 112 43 68 102 105 108 69 47 70 114 98 80 82 107 111 43 51 73 110 69 73 57 66 90 99 103 72 75 76 56 57 65 90 109 50 47 51 43 43 49 71 112 53 100 98 43 52 87 54 52 104 103 82 90 49 69 53 102 121 76 77 117 110 48 67 86 118 77 77 118 76 43 101 56 51 113 119 69 113 83 47 118 102 79 66 115 105 114 50 57 87 54 47 118 70 56 109 53 57 78 48 67 117 114 120 98 47 120 88 120 119 90 65 43 54 88 81 119 80 122 77 47 80 48 118 119 56 110 85 122 84 115 55 77 90 100 74 115 69 86 73 48 80 99 99 101 106 69 84 118 121 52 55 49 87 122 112 69 102 54 97 112 115 115 55 82 73 89 50 108 54 122 75 76 82 101 109 71 49 69 103 99 111 52 88 89 55 116 55 122 119 70 77 103 119 118 75 90 84 86 121 77 87 98 110 83 56 83 65 76 79 101 106 55 54 57 87 114 102 119 110 113 71 115 103 99 107 115 76 79 66 51 121 121 56 73 43 77 116 87 50 72 49 105 119 117 98 66 68 119 70 50 51 104 66 77 86 113 120 108 67 81 97 114 47 104 119 90 81 51 84 73 47 101 82 82 73 101 86 48 111 111 86 72 102 77 110 74 90 84 102 89 72 54 122 118 114 43 56 47 110 73 55 104 48 51 88 82 97 85 84 49 72 65 73 103 110 85 89 70 116 103 81 112 56 113 49 82 97 114 99 47 100 98 54 55 43 106 108 48 70 84 111 78 118 48 116 104 118 109 90 68 120 111 87 69 109 67 114 121 98 43 77 54 74 105 103 55 90 89 69 81 119 108 122 117 120 73 55 107 111 50 79 81 50 86 80 48 100 119 52 99 85 98 101 97 107 112 106 71 51 116 68 84 67 70 71 117 115 98 65 53 79 78 89 100 86 71 114 72 48 99 51 106 121 112 119 54 75 118 43 102 121 83 102 50 50 102 107 78 66 112 54 86 47 109 111 99 101 85 86 75 52 72 54 85 107 112 114 119 112 51 82 104 57 79 69 84 101 121 117 120 107 108 107 87 114 70 72 112 111 100 68 116 68 86 50 113 84 73 80 88 47 117 78 52 119 66 54 47 71 113 119 82 97 87 120 48 103 82 108 72 105 102 77 119 85 88 88 108 109 43 119 102 98 99 121 99 116 78 88 85 106 84 101 107 43 70 53 85 49 102 107 88 120 52 113 111 99 119 83 107 66 83 85 74 51 108 113 121 47 82 114 88 49 81 73 74 53 97 88 104 80 73 65 90 100 90 49 47 119 81 65 65 80 47 47 103 87 82 87 43 73 73 72 65 65 65 98 30 10 26 111 98 106 101 99 116 115 101 116 46 114 105 111 46 99 97 116 116 108 101 46 105 111 47 105 100 18 0 98 54 10 33 111 98 106 101 99 116 115 101 116 46 114 105 111 46 99 97 116 116 108 101 46 105 111 47 111 119 110 101 114 45 103 118 107 18 17 47 118 49 44 32 75 105 110 100 61 83 101 114 118 105 99 101 98 48 10 34 111 98 106 101 99 116 115 101 116 46 114 105 111 46 99 97 116 116 108 101 46 105 111 47 111 119 110 101 114 45 110 97 109 101 18 10 108 105 98 114 101 115 112 101 101 100 98 56 10 39 111 98 106 101 99 116 115 101 116 46 114 105 111 46 99 97 116 116 108 101 46 105 111 47 111 119 110 101 114 45 110 97 109 101 115 112 97 99 101 18 13 105 120 45 108 105 98 114 101 115 112 101 101 100 138 1 159 14 10 3 107 51 115 18 6 85 112 100 97 116 101 26 7 97 112 112 115 47 118 49 34 8 8 164 148 150 172 6 16 0 50 8 70 105 101 108 100 115 86 49 58 240 13 10 237 13 123 34 102 58 109 101 116 97 100 97 116 97 34 58 123 34 102 58 97 110 110 111 116 97 116 105 111 110 115 34 58 123 34 46 34 58 123 125 44 34 102 58 100 101 112 114 101 99 97 116 101 100 46 100 97 101 109 111 110 115 101 116 46 116 101 109 112 108 97 116 101 46 103 101 110 101 114 97 116 105 111 110 34 58 123 125 44 34 102 58 111 98 106 101 99 116 115 101 116 46 114 105 111 46 99 97 116 116 108 101 46 105 111 47 97 112 112 108 105 101 100 34 58 123 125 44 34 102 58 111 98 106 101 99 116 115 101 116 46 114 105 111 46 99 97 116 116 108 101 46 105 111 47 105 100 34 58 123 125 44 34 102 58 111 98 106 101 99 116 115 101 116 46 114 105 111 46 99 97 116 116 108 101 46 105 111 47 111 119 110 101 114 45 103 118 107 34 58 123 125 44 34 102 58 111 98 106 101 99 116 115 101 116 46 114 105 111 46 99 97 116 116 108 101 46 105 111 47 111 119 110 101 114 45 110 97 109 101 34 58 123 125 44 34 102 58 111 98 106 101 99 116 115 101 116 46 114 105 111 46 99 97 116 116 108 101 46 105 111 47 111 119 110 101 114 45 110 97 109 101 115 112 97 99 101 34 58 123 125 125 44 34 102 58 108 97 98 101 108 115 34 58 123 34 46 34 58 123 125 44 34 102 58 111 98 106 101 99 116 115 101 116 46 114 105 111 46 99 97 116 116 108 101 46 105 111 47 104 97 115 104 34 58 123 125 44 34 102 58 115 118 99 99 111 110 116 114 111 108 108 101 114 46 107 51 115 46 99 97 116 116 108 101 46 105 111 47 110 111 100 101 115 101 108 101 99 116 111 114 34 58 123 125 44 34 102 58 115 118 99 99 111 110 116 114 111 108 108 101 114 46 107 51 115 46 99 97 116 116 108 101 46 105 111 47 115 118 99 110 97 109 101 34 58 123 125 44 34 102 58 115 118 99 99 111 110 116 114 111 108 108 101 114 46 107 51 115 46 99 97 116 116 108 101 46 105 111 47 115 118 99 110 97 109 101 115 112 97 99 101 34 58 123 125 125 125 44 34 102 58 115 112 101 99 34 58 123 34 102 58 114 101 118 105 115 105 111 110 72 105 115 116 111 114 121 76 105 109 105 116 34 58 123 125 44 34 102 58 115 101 108 101 99 116 111 114 34 58 123 125 44 34 102 58 116 101 109 112 108 97 116 101 34 58 123 34 102 58 109 101 116 97 100 97 116 97 34 58 123 34 102 58 108 97 98 101 108 115 34 58 123 34 46 34 58 123 125 44 34 102 58 97 112 112 34 58 123 125 44 34 102 58 115 118 99 99 111 110 116 114 111 108 108 101 114 46 107 51 115 46 99 97 116 116 108 101 46 105 111 47 115 118 99 110 97 109 101 34 58 123 125 44 34 102 58 115 118 99 99 111 110 116 114 111 108 108 101 114 46 107 51 115 46 99 97 116 116 108 101 46 105 111 47 115 118 99 110 97 109 101 115 112 97 99 101 34 58 123 125 125 125 44 34 102 58 115 112 101 99 34 58 123 34 102 58 97 117 116 111 109 111 117 110 116 83 101 114 118 105 99 101 65 99 99 111 117 110 116 84 111 107 101 110 34 58 123 125 44 34 102 58 99 111 110 116 97 105 110 101 114 115 34 58 123 34 107 58 123 92 34 110 97 109 101 92 34 58 92 34 108 98 45 116 99 112 45 49 48 48 49 54 92 34 125 34 58 123 34 46 34 58 123 125 44 34 102 58 101 110 118 34 58 123 34 46 34 58 123 125 44 34 107 58 123 92 34 110 97 109 101 92 34 58 92 34 68 69 83 84 95 73 80 83 92 34 125 34 58 123 34 46 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 118 97 108 117 101 34 58 123 125 125 44 34 107 58 123 92 34 110 97 109 101 92 34 58 92 34 68 69 83 84 95 80 79 82 84 92 34 125 34 58 123 34 46 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 118 97 108 117 101 34 58 123 125 125 44 34 107 58 123 92 34 110 97 109 101 92 34 58 92 34 68 69 83 84 95 80 82 79 84 79 92 34 125 34 58 123 34 46 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 118 97 108 117 101 34 58 123 125 125 44 34 107 58 123 92 34 110 97 109 101 92 34 58 92 34 83 82 67 95 80 79 82 84 92 34 125 34 58 123 34 46 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 118 97 108 117 101 34 58 123 125 125 44 34 107 58 123 92 34 110 97 109 101 92 34 58 92 34 83 82 67 95 82 65 78 71 69 83 92 34 125 34 58 123 34 46 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 118 97 108 117 101 34 58 123 125 125 125 44 34 102 58 105 109 97 103 101 34 58 123 125 44 34 102 58 105 109 97 103 101 80 117 108 108 80 111 108 105 99 121 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 112 111 114 116 115 34 58 123 34 46 34 58 123 125 44 34 107 58 123 92 34 99 111 110 116 97 105 110 101 114 80 111 114 116 92 34 58 49 48 48 49 54 44 92 34 112 114 111 116 111 99 111 108 92 34 58 92 34 84 67 80 92 34 125 34 58 123 34 46 34 58 123 125 44 34 102 58 99 111 110 116 97 105 110 101 114 80 111 114 116 34 58 123 125 44 34 102 58 104 111 115 116 80 111 114 116 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 112 114 111 116 111 99 111 108 34 58 123 125 125 125 44 34 102 58 114 101 115 111 117 114 99 101 115 34 58 123 125 44 34 102 58 115 101 99 117 114 105 116 121 67 111 110 116 101 120 116 34 58 123 34 46 34 58 123 125 44 34 102 58 99 97 112 97 98 105 108 105 116 105 101 115 34 58 123 34 46 34 58 123 125 44 34 102 58 97 100 100 34 58 123 125 125 125 44 34 102 58 116 101 114 109 105 110 97 116 105 111 110 77 101 115 115 97 103 101 80 97 116 104 34 58 123 125 44 34 102 58 116 101 114 109 105 110 97 116 105 111 110 77 101 115 115 97 103 101 80 111 108 105 99 121 34 58 123 125 125 125 44 34 102 58 100 110 115 80 111 108 105 99 121 34 58 123 125 44 34 102 58 114 101 115 116 97 114 116 80 111 108 105 99 121 34 58 123 125 44 34 102 58 115 99 104 101 100 117 108 101 114 78 97 109 101 34 58 123 125 44 34 102 58 115 101 99 117 114 105 116 121 67 111 110 116 101 120 116 34 58 123 34 46 34 58 123 125 44 34 102 58 115 121 115 99 116 108 115 34 58 123 125 125 44 34 102 58 115 101 114 118 105 99 101 65 99 99 111 117 110 116 34 58 123 125 44 34 102 58 115 101 114 118 105 99 101 65 99 99 111 117 110 116 78 97 109 101 34 58 123 125 44 34 102 58 116 101 114 109 105 110 97 116 105 111 110 71 114 97 99 101 80 101 114 105 111 100 83 101 99 111 110 100 115 34 58 123 125 44 34 102 58 116 111 108 101 114 97 116 105 111 110 115 34 58 123 125 125 125 44 34 102 58 117 112 100 97 116 101 83 116 114 97 116 101 103 121 34 58 123 34 102 58 114 111 108 108 105 110 103 85 112 100 97 116 101 34 58 123 34 46 34 58 123 125 44 34 102 58 109 97 120 83 117 114 103 101 34 58 123 125 44 34 102 58 109 97 120 85 110 97 118 97 105 108 97 98 108 101 34 58 123 125 125 44 34 102 58 116 121 112 101 34 58 123 125 125 125 125 66 0 138 1 128 2 10 3 107 51 115 18 6 85 112 100 97 116 101 26 7 97 112 112 115 47 118 49 34 8 8 254 248 150 172 6 16 0 50 8 70 105 101 108 100 115 86 49 58 203 1 10 200 1 123 34 102 58 115 116 97 116 117 115 34 58 123 34 102 58 99 117 114 114 101 110 116 78 117 109 98 101 114 83 99 104 101 100 117 108 101 100 34 58 123 125 44 34 102 58 100 101 115 105 114 101 100 78 117 109 98 101 114 83 99 104 101 100 117 108 101 100 34 58 123 125 44 34 102 58 110 117 109 98 101 114 77 105 115 115 99 104 101 100 117 108 101 100 34 58 123 125 44 34 102 58 110 117 109 98 101 114 82 101 97 100 121 34 58 123 125 44 34 102 58 110 117 109 98 101 114 85 110 97 118 97 105 108 97 98 108 101 34 58 123 125 44 34 102 58 111 98 115 101 114 118 101 100 71 101 110 101 114 97 116 105 111 110 34 58 123 125 44 34 102 58 117 112 100 97 116 101 100 78 117 109 98 101 114 83 99 104 101 100 117 108 101 100 34 58 123 125 125 125 66 6 115 116 97 116 117 115 18 248 5 10 34 10 32 10 3 97 112 112 18 25 115 118 99 108 98 45 108 105 98 114 101 115 112 101 101 100 45 99 57 48 100 53 57 49 52 18 170 5 10 160 1 10 0 18 0 26 0 34 0 42 0 50 0 56 0 66 0 90 32 10 3 97 112 112 18 25 115 118 99 108 98 45 108 105 98 114 101 115 112 101 101 100 45 99 57 48 100 53 57 49 52 90 49 10 35 115 118 99 99 111 110 116 114 111 108 108 101 114 46 107 51 115 46 99 97 116 116 108 101 46 105 111 47 115 118 99 110 97 109 101 18 10 108 105 98 114 101 115 112 101 101 100 90 57 10 40 115 118 99 99 111 110 116 114 111 108 108 101 114 46 107 51 115 46 99 97 116 116 108 101 46 105 111 47 115 118 99 110 97 109 101 115 112 97 99 101 18 13 105 120 45 108 105 98 114 101 115 112 101 101 100 18 132 4 18 251 1 10 12 108 98 45 116 99 112 45 49 48 48 49 54 18 25 114 97 110 99 104 101 114 47 107 108 105 112 112 101 114 45 108 98 58 118 48 46 52 46 52 42 0 50 27 10 12 108 98 45 116 99 112 45 49 48 48 49 54 16 160 78 24 160 78 34 3 84 67 80 42 0 58 17 10 8 83 82 67 95 80 79 82 84 18 5 49 48 48 49 54 58 23 10 10 83 82 67 95 82 65 78 71 69 83 18 9 48 46 48 46 48 46 48 47 48 58 17 10 10 68 69 83 84 95 80 82 79 84 79 18 3 84 67 80 58 18 10 9 68 69 83 84 95 80 79 82 84 18 5 49 48 48 49 54 58 25 10 8 68 69 83 84 95 73 80 83 18 13 49 55 50 46 49 55 46 51 54 46 50 50 56 66 0 106 20 47 100 101 118 47 116 101 114 109 105 110 97 116 105 111 110 45 108 111 103 114 12 73 102 78 111 116 80 114 101 115 101 110 116 122 13 10 11 10 9 78 69 84 95 65 68 77 73 78 128 1 0 136 1 0 144 1 0 162 1 4 70 105 108 101 26 6 65 108 119 97 121 115 32 30 50 12 67 108 117 115 116 101 114 70 105 114 115 116 66 5 115 118 99 108 98 74 5 115 118 99 108 98 82 0 88 0 96 0 104 0 114 26 58 24 10 19 110 101 116 46 105 112 118 52 46 105 112 95 102 111 114 119 97 114 100 18 1 49 130 1 0 138 1 0 154 1 17 100 101 102 97 117 108 116 45 115 99 104 101 100 117 108 101 114 168 1 0 178 1 54 10 30 110 111 100 101 45 114 111 108 101 46 107 117 98 101 114 110 101 116 101 115 46 105 111 47 109 97 115 116 101 114 18 6 69 120 105 115 116 115 26 0 34 10 78 111 83 99 104 101 100 117 108 101 178 1 61 10 37 110 111 100 101 45 114 111 108 101 46 107 117 98 101 114 110 101 116 101 115 46 105 111 47 99 111 110 116 114 111 108 45 112 108 97 110 101 18 6 69 120 105 115 116 115 26 0 34 10 78 111 83 99 104 101 100 117 108 101 178 1 32 10 18 67 114 105 116 105 99 97 108 65 100 100 111 110 115 79 110 108 121 18 6 69 120 105 115 116 115 26 0 34 0 194 1 0 26 33 10 13 82 111 108 108 105 110 103 85 112 100 97 116 101 18 16 10 6 8 0 16 1 26 0 18 6 8 0 16 0 26 0 32 0 48 10 26 16 8 1 16 0 24 1 32 0 40 1 48 1 56 0 64 1 26 0 34 0]]]"
Dec 22 17:43:30 truenas k3s[16911]: I1222 17:43:30.383165   16911 trace.go:219] Trace[86856513]: "Update" accept:application/vnd.kubernetes.protobuf, */*,audit-id:bc08eab8-cbcb-456f-a464-2e8e4fa5fb18,client:127.0.0.1,protocol:HTTP/2.0,resource:daemonsets,scope:resource,url:/apis/apps/v1/namespaces/kube-system/daemonsets/svclb-librespeed-c90d5914/status,user-agent:k3s/v1.26.6+k3s (linux/amd64) kubernetes/e18037a/system:serviceaccount:kube-system:daemon-set-controller,verb:PUT (22-Dec-2023 17:43:29.272) (total time: 1110ms):
Dec 22 17:43:30 truenas k3s[16911]: Trace[86856513]: ["GuaranteedUpdate etcd3" audit-id:bc08eab8-cbcb-456f-a464-2e8e4fa5fb18,key:/daemonsets/kube-system/svclb-librespeed-c90d5914,type:*apps.DaemonSet,resource:daemonsets.apps 1110ms (17:43:29.272)
Dec 22 17:43:30 truenas k3s[16911]: Trace[86856513]:  ---"Txn call completed" 1109ms (17:43:30.382)]
Dec 22 17:43:30 truenas k3s[16911]: Trace[86856513]: [1.110501257s] [1.110501257s] END
Dec 22 17:43:30 truenas k3s[16911]: I1222 17:43:30.528649   16911 scope.go:115] "RemoveContainer" containerID="05156a4f15de7e22ba6695e543846acb41a06a49723b0eaccb9abf3e421450e6"
Dec 22 17:43:30 truenas k3s[16911]: I1222 17:43:30.970009   16911 event.go:294] "Event occurred" object="ix-traefik/traefik-tcp" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="UpdatedLoadBalancer" message="Updated LoadBalancer with new IPs: [] -> [192.168.1.100]"
Dec 22 17:43:31 truenas k3s[16911]: I1222 17:43:31.391670   16911 scope.go:115] "RemoveContainer" containerID="6a51587a549be6118cdb80911d5252cf238a64508b9a541dee51313b940a64cd"
Dec 22 17:43:31 truenas k3s[16911]: I1222 17:43:31.874439   16911 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="kube-system/svclb-traefik-tcp-f0275a1c-px89t" podStartSLOduration=53.874414782 pod.CreationTimestamp="2023-12-22 17:42:38 +0100 CET" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2023-12-22 17:43:31.181274533 +0100 CET m=+68.968080324" watchObservedRunningTime="2023-12-22 17:43:31.874414782 +0100 CET m=+69.661220573"
Dec 22 17:43:32 truenas k3s[16911]: I1222 17:43:32.055002   16911 trace.go:219] Trace[50696909]: "Create" accept:application/vnd.kubernetes.protobuf,application/json,audit-id:8665599d-07c1-44ff-8c47-0405b2575f5f,client:127.0.0.1,protocol:HTTP/2.0,resource:events,scope:resource,url:/api/v1/namespaces/kube-system/events,user-agent:k3s/v1.26.6+k3s (linux/amd64) kubernetes/e18037a,verb:POST (22-Dec-2023 17:43:31.361) (total time: 693ms):
Dec 22 17:43:32 truenas k3s[16911]: Trace[50696909]: ["Create etcd3" audit-id:8665599d-07c1-44ff-8c47-0405b2575f5f,key:/events/kube-system/openebs-zfs-controller-0.17a335072e1d5191,type:*core.Event,resource:events 692ms (17:43:31.362)
Dec 22 17:43:32 truenas k3s[16911]: Trace[50696909]:  ---"Txn call succeeded" 692ms (17:43:32.054)]
Dec 22 17:43:32 truenas k3s[16911]: Trace[50696909]: [693.235788ms] [693.235788ms] END
Dec 22 17:43:32 truenas k3s[16911]: I1222 17:43:32.281322   16911 scope.go:115] "RemoveContainer" containerID="384dccf815118e10d50afecc78abf1e7547f75728ffdcd55f3a66fecbdb20ae0"
Dec 22 17:43:32 truenas k3s[16911]: I1222 17:43:32.673650   16911 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="ix-cert-manager/cert-manager-certmanager-557965ccc8-tzws2" podStartSLOduration=5007.673626088 pod.CreationTimestamp="2023-12-22 16:20:05 +0100 CET" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2023-12-22 17:43:32.673138431 +0100 CET m=+70.459944222" watchObservedRunningTime="2023-12-22 17:43:32.673626088 +0100 CET m=+70.460431879"
Dec 22 17:43:32 truenas k3s[16911]: I1222 17:43:32.727719   16911 scope.go:115] "RemoveContainer" containerID="886b9b054f9929d41e176da9ad106c7b03fc8488b7b54d85feeb1cd2aac9211f"
Dec 22 17:43:33 truenas k3s[16911]: I1222 17:43:33.224720   16911 scope.go:115] "RemoveContainer" containerID="2f9e517ad699fb4b63b9a14dda5bae70d4b1d63ab6b2849e3466ccd4dc262a8c"
Dec 22 17:43:33 truenas k3s[16911]: I1222 17:43:33.474714   16911 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="ix-nextcloud/nextcloud-redis-0" podStartSLOduration=5009.474691984 pod.CreationTimestamp="2023-12-22 16:20:04 +0100 CET" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2023-12-22 17:43:33.473413169 +0100 CET m=+71.260218970" watchObservedRunningTime="2023-12-22 17:43:33.474691984 +0100 CET m=+71.261497775"
Dec 22 17:43:33 truenas k3s[16911]: I1222 17:43:33.812130   16911 scope.go:115] "RemoveContainer" containerID="56a34362b0cd50ac76920b71a893f1829bda352d10fe28114b148c9f5735fba7"
Dec 22 17:43:34 truenas k3s[16911]: I1222 17:43:34.247826   16911 scope.go:115] "RemoveContainer" containerID="fc590133ea54abdc1d5b6dda33e60a6d12166ac3fd4937441c99dd5e9ff6aa04"
Dec 22 17:43:34 truenas k3s[16911]: I1222 17:43:34.273724   16911 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="kube-system/coredns-59b4f5bbd5-qlxsh" podStartSLOduration=57.273695902 pod.CreationTimestamp="2023-12-22 17:42:37 +0100 CET" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2023-12-22 17:43:34.273340213 +0100 CET m=+72.060146014" watchObservedRunningTime="2023-12-22 17:43:34.273695902 +0100 CET m=+72.060501703"
Dec 22 17:43:34 truenas k3s[16911]: I1222 17:43:34.344136   16911 scope.go:115] "RemoveContainer" containerID="178c528858743b69632d4bb5016ba2e814f0fd747a3c2e931a79ab5233eb4d40"
Dec 22 17:43:34 truenas k3s[16911]: I1222 17:43:34.464188   16911 scope.go:115] "RemoveContainer" containerID="7e899da2cf551d8b2e7fd042314964212c7f9f86eb18d96b33d77f8531276451"
Dec 22 17:43:34 truenas k3s[16911]: I1222 17:43:34.552107   16911 scope.go:115] "RemoveContainer" containerID="45d8daabaadf9b10dd0000dcac655a29728600c7569182e448567292cb33c990"
Dec 22 17:43:34 truenas k3s[16911]: I1222 17:43:34.656022   16911 scope.go:115] "RemoveContainer" containerID="1fad5d309bfa76b06c5fc72ab0691045661623b220e3a6af5385756dbd9d68af"
Dec 22 17:43:34 truenas k3s[16911]: I1222 17:43:34.735801   16911 scope.go:115] "RemoveContainer" containerID="f02c1ba8851b923c2f87f76b13eca6e6e44b5615e757a9a5c0bd1cc0c54d4ae9"
Dec 22 17:43:34 truenas k3s[16911]: I1222 17:43:34.889424   16911 scope.go:115] "RemoveContainer" containerID="23198624fff5ed05713bd755cc89ad1dc36b3b2000650eeb67a13553f84be966"
Dec 22 17:43:34 truenas k3s[16911]: I1222 17:43:34.991325   16911 scope.go:115] "RemoveContainer" containerID="4f4449a297857c06317e10bf806a3a5a0e9acc40e16a229434fccb312d63ef26"
Dec 22 17:43:35 truenas k3s[16911]: I1222 17:43:35.076012   16911 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="kube-system/svclb-blocky-dnsudp-40a34fec-m9rq7" podStartSLOduration=57.075977703 pod.CreationTimestamp="2023-12-22 17:42:38 +0100 CET" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2023-12-22 17:43:35.074111414 +0100 CET m=+72.860917205" watchObservedRunningTime="2023-12-22 17:43:35.075977703 +0100 CET m=+72.862783524"
Dec 22 17:43:35 truenas k3s[16911]: I1222 17:43:35.078671   16911 event.go:294] "Event occurred" object="ix-blocky/blocky-dnsudp" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="UpdatedLoadBalancer" message="Updated LoadBalancer with new IPs: [] -> [192.168.1.100]"
Dec 22 17:43:35 truenas k3s[16911]: I1222 17:43:35.096851   16911 scope.go:115] "RemoveContainer" containerID="b0acba208fc0283ae28a5e80794ef70d8d7fabd74e57314ef04b1ede341d3a11"
Dec 22 17:43:35 truenas k3s[16911]: I1222 17:43:35.187738   16911 scope.go:115] "RemoveContainer" containerID="b19bd1bc8cb08cfb750be90a872bb0663d4dbf4ed19a1303a1696d053846fcf8"
Dec 22 17:43:35 truenas k3s[16911]: I1222 17:43:35.254430   16911 scope.go:115] "RemoveContainer" containerID="13a78362fb9aab70d47b65edf0dfba7dd5b266797589e226b66e02a8bc5a71ae"
Dec 22 17:43:35 truenas k3s[16911]: I1222 17:43:35.361356   16911 scope.go:115] "RemoveContainer" containerID="ae3593d82ca889cf729f13e7b5f51714731d9484f93dfa81749fcc32847345c1"
Dec 22 17:43:35 truenas k3s[16911]: I1222 17:43:35.455846   16911 scope.go:115] "RemoveContainer" containerID="4ff1beaf5fb208d58934f0a0f2570c3a237fd9ec0ab52afc22f7fec3e0018ab3"
Dec 22 17:43:35 truenas k3s[16911]: I1222 17:43:35.473070   16911 job_controller.go:514] enqueueing job ix-nextcloud/nextcloud-nextcloud-cron-28387640
Dec 22 17:43:35 truenas k3s[16911]: I1222 17:43:35.558884   16911 scope.go:115] "RemoveContainer" containerID="fa2336dd9f2992d4d264f12217d709ffdb25d3ed9043b3a38845cf1d2705a0c1"
Dec 22 17:43:35 truenas k3s[16911]: I1222 17:43:35.668311   16911 scope.go:115] "RemoveContainer" containerID="6df17a6fc118550b27b7263700740c14c9bf384d4dacbb4318dce9b445df9af5"
Dec 22 17:43:35 truenas k3s[16911]: I1222 17:43:35.785929   16911 scope.go:115] "RemoveContainer" containerID="b9afd36e62eca84723c614af43242bb5af0f5871b9e8256af4098ae47f6525fc"
Dec 22 17:43:35 truenas k3s[16911]: I1222 17:43:35.866636   16911 scope.go:115] "RemoveContainer" containerID="6d0e2a104a32c11d571a2c2f8ad5353fd8dd9b99d30c2a4d41c110c59dc44e91"
Dec 22 17:43:35 truenas k3s[16911]: I1222 17:43:35.874896   16911 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="kube-system/csi-nfs-controller-7b74694749-g6k8q" podStartSLOduration=37.874871173 pod.CreationTimestamp="2023-12-22 17:42:58 +0100 CET" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2023-12-22 17:43:35.873625911 +0100 CET m=+73.660431712" watchObservedRunningTime="2023-12-22 17:43:35.874871173 +0100 CET m=+73.661676964"
Dec 22 17:43:35 truenas k3s[16911]: I1222 17:43:35.963594   16911 scope.go:115] "RemoveContainer" containerID="8c16333189a1767dbc4170ee04265234114940cdcce6580e90da45be2a98ef6a"
Dec 22 17:43:36 truenas k3s[16911]: I1222 17:43:36.067494   16911 scope.go:115] "RemoveContainer" containerID="bc22b46c9109bbde6cb2935fe457934b07a7652c36cd47084aa8bce1e0d9e21b"
Dec 22 17:43:36 truenas k3s[16911]: I1222 17:43:36.161670   16911 scope.go:115] "RemoveContainer" containerID="1ec66c7dee4b956d34346e6fca29eb363defb92cd756ad97a78057b15f44af49"
Dec 22 17:43:36 truenas k3s[16911]: I1222 17:43:36.242478   16911 scope.go:115] "RemoveContainer" containerID="292bd40824e1b148ed9d66ac61a6ae3df9e7a93cc66756a2d23de381acdbca96"
Dec 22 17:43:36 truenas k3s[16911]: I1222 17:43:36.277529   16911 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="ix-traefik/traefik-5996c85fd6-n64lg" podStartSLOduration=28.277493926 pod.CreationTimestamp="2023-12-22 17:43:08 +0100 CET" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2023-12-22 17:43:36.276093883 +0100 CET m=+74.062899674" watchObservedRunningTime="2023-12-22 17:43:36.277493926 +0100 CET m=+74.064299717"
Dec 22 17:43:36 truenas k3s[16911]: I1222 17:43:36.336289   16911 scope.go:115] "RemoveContainer" containerID="613ec6cf0f5786af384da48e86b3eaf331f680e6e07f173558a2970323084699"
Dec 22 17:43:36 truenas k3s[16911]: I1222 17:43:36.404468   16911 scope.go:115] "RemoveContainer" containerID="78c3788032e1ac560e6394fe2d0cf4edaa4b3959cc80a3026c7e23c8b8738cfa"
Dec 22 17:43:36 truenas k3s[16911]: I1222 17:43:36.499494   16911 scope.go:115] "RemoveContainer" containerID="55cdd7e0d51f4a8faa95bf3f850453597ab77fd12f178c6c7e558c59f6a94e19"
Dec 22 17:43:36 truenas k3s[16911]: I1222 17:43:36.577151   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-a8dd155b-b0ab-42ff-975f-7c411496e41c\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-a8dd155b-b0ab-42ff-975f-7c411496e41c\") pod \"f71d37a0-295a-4e20-850d-a30130211b6a\" (UID: \"f71d37a0-295a-4e20-850d-a30130211b6a\") "
Dec 22 17:43:36 truenas k3s[16911]: I1222 17:43:36.577213   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-346c83fb-a449-41eb-bc74-080578ba1326\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-346c83fb-a449-41eb-bc74-080578ba1326\") pod \"f71d37a0-295a-4e20-850d-a30130211b6a\" (UID: \"f71d37a0-295a-4e20-850d-a30130211b6a\") "
Dec 22 17:43:36 truenas k3s[16911]: I1222 17:43:36.585216   16911 operation_generator.go:900] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/zfs.csi.openebs.io^pvc-346c83fb-a449-41eb-bc74-080578ba1326" (OuterVolumeSpecName: "pvc-346c83fb-a449-41eb-bc74-080578ba1326") pod "f71d37a0-295a-4e20-850d-a30130211b6a" (UID: "f71d37a0-295a-4e20-850d-a30130211b6a"). InnerVolumeSpecName "pvc-346c83fb-a449-41eb-bc74-080578ba1326". PluginName "kubernetes.io/csi", VolumeGidValue ""
Dec 22 17:43:36 truenas k3s[16911]: I1222 17:43:36.585262   16911 operation_generator.go:900] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/zfs.csi.openebs.io^pvc-a8dd155b-b0ab-42ff-975f-7c411496e41c" (OuterVolumeSpecName: "pvc-a8dd155b-b0ab-42ff-975f-7c411496e41c") pod "f71d37a0-295a-4e20-850d-a30130211b6a" (UID: "f71d37a0-295a-4e20-850d-a30130211b6a"). InnerVolumeSpecName "pvc-a8dd155b-b0ab-42ff-975f-7c411496e41c". PluginName "kubernetes.io/csi", VolumeGidValue ""
Dec 22 17:43:36 truenas k3s[16911]: I1222 17:43:36.599130   16911 scope.go:115] "RemoveContainer" containerID="dcbce716cede941929ebf7ef1dd93222aa5e47650c39994629079fb6616fca54"
Dec 22 17:43:36 truenas k3s[16911]: I1222 17:43:36.677723   16911 reconciler_common.go:288] "operationExecutor.UnmountDevice started for volume \"pvc-a8dd155b-b0ab-42ff-975f-7c411496e41c\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-a8dd155b-b0ab-42ff-975f-7c411496e41c\") on node \"ix-truenas\" "
Dec 22 17:43:36 truenas k3s[16911]: I1222 17:43:36.677753   16911 reconciler_common.go:288] "operationExecutor.UnmountDevice started for volume \"pvc-346c83fb-a449-41eb-bc74-080578ba1326\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-346c83fb-a449-41eb-bc74-080578ba1326\") on node \"ix-truenas\" "
Dec 22 17:43:36 truenas k3s[16911]: I1222 17:43:36.712734   16911 scope.go:115] "RemoveContainer" containerID="b36c473f3c41ef7c43d7fa99e139fe2d53ca456551b8d8dece78a494d7763116"
Dec 22 17:43:36 truenas k3s[16911]: I1222 17:43:36.714660   16911 csi_attacher.go:624] kubernetes.io/csi: attacher.UnmountDevice STAGE_UNSTAGE_VOLUME capability not set. Skipping UnmountDevice...
Dec 22 17:43:36 truenas k3s[16911]: I1222 17:43:36.714661   16911 csi_attacher.go:624] kubernetes.io/csi: attacher.UnmountDevice STAGE_UNSTAGE_VOLUME capability not set. Skipping UnmountDevice...
Dec 22 17:43:36 truenas k3s[16911]: I1222 17:43:36.714870   16911 operation_generator.go:1014] UnmountDevice succeeded for volume "pvc-346c83fb-a449-41eb-bc74-080578ba1326" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-346c83fb-a449-41eb-bc74-080578ba1326") on node "ix-truenas"
Dec 22 17:43:36 truenas k3s[16911]: I1222 17:43:36.714885   16911 operation_generator.go:1014] UnmountDevice succeeded for volume "pvc-a8dd155b-b0ab-42ff-975f-7c411496e41c" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-a8dd155b-b0ab-42ff-975f-7c411496e41c") on node "ix-truenas"
Dec 22 17:43:36 truenas k3s[16911]: I1222 17:43:36.778672   16911 reconciler_common.go:295] "Volume detached for volume \"pvc-a8dd155b-b0ab-42ff-975f-7c411496e41c\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-a8dd155b-b0ab-42ff-975f-7c411496e41c\") on node \"ix-truenas\" DevicePath \"\""
Dec 22 17:43:36 truenas k3s[16911]: I1222 17:43:36.778692   16911 reconciler_common.go:295] "Volume detached for volume \"pvc-346c83fb-a449-41eb-bc74-080578ba1326\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-346c83fb-a449-41eb-bc74-080578ba1326\") on node \"ix-truenas\" DevicePath \"\""
Dec 22 17:43:36 truenas k3s[16911]: I1222 17:43:36.814915   16911 scope.go:115] "RemoveContainer" containerID="9fc202454822ec6f11948356b5988b615ea45d43be9ed8b75e72f8f950c14b5e"
Dec 22 17:43:36 truenas k3s[16911]: I1222 17:43:36.912573   16911 scope.go:115] "RemoveContainer" containerID="5e8bbcaf40a4439118c433ba5f6c533b5ed5aec72ca6a523c12ec91460361a4a"
Dec 22 17:43:37 truenas k3s[16911]: I1222 17:43:37.019122   16911 scope.go:115] "RemoveContainer" containerID="d64cf07fedf6e7103e82d22c24851e1b3f5df953eba57bb14095e2c32abc43dd"
Dec 22 17:43:37 truenas k3s[16911]: I1222 17:43:37.073728   16911 job_controller.go:514] enqueueing job ix-nextcloud/nextcloud-nextcloud-cron-28387640
Dec 22 17:43:37 truenas k3s[16911]: I1222 17:43:37.074528   16911 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="ix-nextcloud/nextcloud-nextcloud-cron-28387640-kvwwv" podStartSLOduration=35.074504385 pod.CreationTimestamp="2023-12-22 17:43:02 +0100 CET" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2023-12-22 17:43:37.073469869 +0100 CET m=+74.860275660" watchObservedRunningTime="2023-12-22 17:43:37.074504385 +0100 CET m=+74.861310176"
Dec 22 17:43:37 truenas k3s[16911]: I1222 17:43:37.114643   16911 scope.go:115] "RemoveContainer" containerID="5c90f051743958444d05faee3f4073702e59749efe1676448e2b7399668a829d"
Dec 22 17:43:37 truenas k3s[16911]: I1222 17:43:37.214888   16911 scope.go:115] "RemoveContainer" containerID="0cc7f843fd68ab38d4f74427e5b0d14ffacc25e327d08c195f17573e080aba94"
Dec 22 17:43:37 truenas k3s[16911]: I1222 17:43:37.302017   16911 scope.go:115] "RemoveContainer" containerID="5a83d7018c57b279ff2c50a30ea3775156cc7d210143af4ed37bfa77dae83dc3"
Dec 22 17:43:37 truenas k3s[16911]: I1222 17:43:37.384546   16911 scope.go:115] "RemoveContainer" containerID="d61594d32e32adbf57664b26dd767d290cbab6a9310cfa936d67863d7c92128e"
Dec 22 17:43:37 truenas k3s[16911]: I1222 17:43:37.473400   16911 scope.go:115] "RemoveContainer" containerID="1546c3adcba4b01d8dabb5784a76b9d1d6907b66822286bcfb94281f64866850"
Dec 22 17:43:37 truenas k3s[16911]: I1222 17:43:37.475763   16911 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="ix-adguard-home/adguard-home-57b89dd8b8-xzj2m" podStartSLOduration=5004.475738346 pod.CreationTimestamp="2023-12-22 16:20:13 +0100 CET" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2023-12-22 17:43:37.474210543 +0100 CET m=+75.261016334" watchObservedRunningTime="2023-12-22 17:43:37.475738346 +0100 CET m=+75.262544137"
Dec 22 17:43:37 truenas k3s[16911]: I1222 17:43:37.747797   16911 scope.go:115] "RemoveContainer" containerID="2503850cf52fbf213739e12bf31070e098d011146adfa04f9d318caa5b103250"
Dec 22 17:43:37 truenas k3s[16911]: I1222 17:43:37.917728   16911 scope.go:115] "RemoveContainer" containerID="7bf522051981781ffc494c5213f6a8a108cca37c02b808e9eb4556327cddd22e"
Dec 22 17:43:38 truenas k3s[16911]: I1222 17:43:38.062272   16911 scope.go:115] "RemoveContainer" containerID="7245508741d36382d8a93e3635c3c82c1f1a33417291053f9e04ca740673412c"
Dec 22 17:43:38 truenas k3s[16911]: I1222 17:43:38.079462   16911 job_controller.go:514] enqueueing job ix-nextcloud/nextcloud-nextcloud-cron-28387640
Dec 22 17:43:38 truenas k3s[16911]: I1222 17:43:38.079665   16911 event.go:294] "Event occurred" object="ix-nextcloud/nextcloud-nextcloud-cron" fieldPath="" kind="CronJob" apiVersion="batch/v1" type="Normal" reason="JobAlreadyActive" message="Not starting job because prior execution is running and concurrency policy is Forbid"
Dec 22 17:43:38 truenas k3s[16911]: I1222 17:43:38.182602   16911 scope.go:115] "RemoveContainer" containerID="aadd8c9ffd950fe64f623da133e3bc4c5d9280a9512de31f6eb26c9842c7e2b5"
Dec 22 17:43:38 truenas k3s[16911]: I1222 17:43:38.283441   16911 scope.go:115] "RemoveContainer" containerID="6e89454031d1f410b15736e590561af046f55e6957eb05762409669d49e12732"
Dec 22 17:43:38 truenas k3s[16911]: I1222 17:43:38.395892   16911 scope.go:115] "RemoveContainer" containerID="6de8ec50260bd86688ae6ef299a6ce4e1fdd71b8177ab0b63cbc0ad777a8a746"
Dec 22 17:43:38 truenas k3s[16911]: I1222 17:43:38.481153   16911 scope.go:115] "RemoveContainer" containerID="5620bb6e24782ce4520785846774b3668f66f582170b8b5fd16ffe680ec523c9"
Dec 22 17:43:38 truenas k3s[16911]: I1222 17:43:38.566242   16911 scope.go:115] "RemoveContainer" containerID="be56b939350260a4dc24d67ed074c09d2cfc85529af811e2cec9482613953aed"
Dec 22 17:43:38 truenas k3s[16911]: I1222 17:43:38.674454   16911 job_controller.go:514] enqueueing job ix-nextcloud/nextcloud-preview-cron-28387720
Dec 22 17:43:38 truenas k3s[16911]: I1222 17:43:38.674551   16911 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="ix-nextcloud/nextcloud-preview-cron-28387720-dq7tq" podStartSLOduration=60.674520426 pod.CreationTimestamp="2023-12-22 17:42:38 +0100 CET" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2023-12-22 17:43:38.674244437 +0100 CET m=+76.461050228" watchObservedRunningTime="2023-12-22 17:43:38.674520426 +0100 CET m=+76.461326227"
Dec 22 17:43:38 truenas k3s[16911]: I1222 17:43:38.676973   16911 scope.go:115] "RemoveContainer" containerID="860776794d2c5b00f105cbe160d6017cd9b6bc386729a5a5515947cbf1a00a0c"
Dec 22 17:43:38 truenas k3s[16911]: I1222 17:43:38.798325   16911 scope.go:115] "RemoveContainer" containerID="edb936809a0db159fc616e589cd92d830455d509980bc7f3535ca24ce38ed14e"
Dec 22 17:43:38 truenas k3s[16911]: I1222 17:43:38.887910   16911 scope.go:115] "RemoveContainer" containerID="b3c0899034d5bf1593c56753cc4050fd30fa633f36e7a2be92cfa2b614c91e8f"
Dec 22 17:43:38 truenas k3s[16911]: I1222 17:43:38.996774   16911 scope.go:115] "RemoveContainer" containerID="021bffe09281b0d4083582f3597a3ba5355b014fe3aa366dcf7a9106ee8baecb"
Dec 22 17:43:39 truenas k3s[16911]: I1222 17:43:39.075836   16911 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="ix-cloudflareddns/cloudflareddns-56c9c85bb8-h822b" podStartSLOduration=32.075801787 pod.CreationTimestamp="2023-12-22 17:43:07 +0100 CET" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2023-12-22 17:43:39.075285976 +0100 CET m=+76.862091767" watchObservedRunningTime="2023-12-22 17:43:39.075801787 +0100 CET m=+76.862607588"
Dec 22 17:43:39 truenas k3s[16911]: I1222 17:43:39.106634   16911 scope.go:115] "RemoveContainer" containerID="e630fd7f43663bd8d634c37faaf06fa89420b203707754dc671fc5842ee0687b"
Dec 22 17:43:39 truenas k3s[16911]: I1222 17:43:39.192137   16911 scope.go:115] "RemoveContainer" containerID="024c4771fa376374a5c55caf9781e8756889f12dd1891119db2785ba400a2db2"
Dec 22 17:43:39 truenas k3s[16911]: I1222 17:43:39.308594   16911 scope.go:115] "RemoveContainer" containerID="e450431ce45d28e855d0d75422f100ca8f357b8deaaf6303155c70c97023c8cb"
Dec 22 17:43:39 truenas k3s[16911]: I1222 17:43:39.387832   16911 scope.go:115] "RemoveContainer" containerID="9cbf2de12f8102c37c9cb3757e87d87c920d8f9a953b93e01559df97084aa79e"
Dec 22 17:43:39 truenas k3s[16911]: I1222 17:43:39.450617   16911 scope.go:115] "RemoveContainer" containerID="1f18651ef70f16c144ea2c7361c951c0e3cae5c174d293aae4fec1589ced6af6"
Dec 22 17:43:39 truenas k3s[16911]: I1222 17:43:39.475900   16911 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="ix-nextcloud/nextcloud-cnpg-main-pooler-rw-78cc84d8cb-bqj9x" podStartSLOduration=5013.47587226 pod.CreationTimestamp="2023-12-22 16:20:06 +0100 CET" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2023-12-22 17:43:39.475052217 +0100 CET m=+77.261858008" watchObservedRunningTime="2023-12-22 17:43:39.47587226 +0100 CET m=+77.262678051"
Dec 22 17:43:39 truenas k3s[16911]: I1222 17:43:39.521848   16911 scope.go:115] "RemoveContainer" containerID="0beaef85d7443fc1c9b9f2b063a277722ec721463442717e578c93fa03c2ab6f"
Dec 22 17:43:39 truenas k3s[16911]: I1222 17:43:39.631508   16911 scope.go:115] "RemoveContainer" containerID="0a159607a835c73f779c190ded15090b0fd463560d8d5ae51d3592b6a74e2005"
Dec 22 17:43:39 truenas k3s[16911]: I1222 17:43:39.679751   16911 job_controller.go:514] enqueueing job ix-nextcloud/nextcloud-preview-cron-28387720
Dec 22 17:43:39 truenas k3s[16911]: I1222 17:43:39.718138   16911 scope.go:115] "RemoveContainer" containerID="f84ea4fa7f0bf3493e296c4e78b3e18d063ca4100bf7e8de2e5d564fa5a7000a"
Dec 22 17:43:39 truenas k3s[16911]: I1222 17:43:39.802512   16911 scope.go:115] "RemoveContainer" containerID="610b201c73a3ac0347844aeb108bb9b4fb07401d8829c9ad59d20a72277e833a"
Dec 22 17:43:39 truenas k3s[16911]: I1222 17:43:39.874692   16911 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="ix-scrutiny/scrutiny-597cdf895b-rsnv7" podStartSLOduration=5004.874657476 pod.CreationTimestamp="2023-12-22 16:20:15 +0100 CET" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2023-12-22 17:43:39.873364384 +0100 CET m=+77.660170175" watchObservedRunningTime="2023-12-22 17:43:39.874657476 +0100 CET m=+77.661463267"
Dec 22 17:43:39 truenas k3s[16911]: I1222 17:43:39.878365   16911 scope.go:115] "RemoveContainer" containerID="3a56f1aefca77db95e27aa97b285f155da9ad71f17be53edf2ea3e9a8382389e"
Dec 22 17:43:39 truenas k3s[16911]: I1222 17:43:39.972645   16911 scope.go:115] "RemoveContainer" containerID="b57932f000aa7758a4062cdf0f9b2a57905b1c20864792bfe8d244164c78b02b"
Dec 22 17:43:43 truenas k3s[16911]: {"level":"warn","ts":"2023-12-22T17:43:43.089+0100","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ac81c0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 22 17:43:43 truenas k3s[16911]: I1222 17:43:43.269754   16911 scope.go:115] "RemoveContainer" containerID="39c9fbc58c4469221529c74b90cb4047b172f2ef147358cfa4a9779a0e30027b"
Dec 22 17:43:44 truenas k3s[16911]: I1222 17:43:44.279883   16911 job_controller.go:514] enqueueing job ix-nextcloud/nextcloud-nextcloud-cron-28387640
Dec 22 17:43:45 truenas k3s[16911]: I1222 17:43:45.276213   16911 scope.go:115] "RemoveContainer" containerID="39c9fbc58c4469221529c74b90cb4047b172f2ef147358cfa4a9779a0e30027b"
Dec 22 17:43:45 truenas k3s[16911]: I1222 17:43:45.276511   16911 scope.go:115] "RemoveContainer" containerID="55603d0068fb10b249fe3be2411da13f2da73f2569c7a3b2a1e48f49174c669d"
Dec 22 17:43:45 truenas k3s[16911]: E1222 17:43:45.277106   16911 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"adguard-home\" with CrashLoopBackOff: \"back-off 10s restarting failed container=adguard-home pod=adguard-home-57b89dd8b8-xzj2m_ix-adguard-home(015c0b2a-db07-4719-b416-3506b105713d)\"" pod="ix-adguard-home/adguard-home-57b89dd8b8-xzj2m" podUID=015c0b2a-db07-4719-b416-3506b105713d
Dec 22 17:43:45 truenas k3s[16911]: I1222 17:43:45.286294   16911 job_controller.go:514] enqueueing job ix-nextcloud/nextcloud-nextcloud-cron-28387640
Dec 22 17:43:45 truenas k3s[16911]: I1222 17:43:45.286542   16911 event.go:294] "Event occurred" object="ix-nextcloud/nextcloud-nextcloud-cron" fieldPath="" kind="CronJob" apiVersion="batch/v1" type="Normal" reason="JobAlreadyActive" message="Not starting job because prior execution is running and concurrency policy is Forbid"
Dec 22 17:43:45 truenas k3s[16911]: I1222 17:43:45.292589   16911 job_controller.go:514] enqueueing job ix-nextcloud/nextcloud-preview-cron-28387720
Dec 22 17:43:45 truenas k3s[16911]: I1222 17:43:45.751908   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\") pod \"485f2017-fcab-46d8-a0f8-71bf110c1836\" (UID: \"485f2017-fcab-46d8-a0f8-71bf110c1836\") "
Dec 22 17:43:45 truenas k3s[16911]: I1222 17:43:45.751948   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/host-path/485f2017-fcab-46d8-a0f8-71bf110c1836-data\") pod \"485f2017-fcab-46d8-a0f8-71bf110c1836\" (UID: \"485f2017-fcab-46d8-a0f8-71bf110c1836\") "
Dec 22 17:43:45 truenas k3s[16911]: I1222 17:43:45.751981   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"varlogs\" (UniqueName: \"kubernetes.io/empty-dir/485f2017-fcab-46d8-a0f8-71bf110c1836-varlogs\") pod \"485f2017-fcab-46d8-a0f8-71bf110c1836\" (UID: \"485f2017-fcab-46d8-a0f8-71bf110c1836\") "
Dec 22 17:43:45 truenas k3s[16911]: I1222 17:43:45.751997   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/485f2017-fcab-46d8-a0f8-71bf110c1836-tmp\") pod \"485f2017-fcab-46d8-a0f8-71bf110c1836\" (UID: \"485f2017-fcab-46d8-a0f8-71bf110c1836\") "
Dec 22 17:43:45 truenas k3s[16911]: I1222 17:43:45.752025   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"devshm\" (UniqueName: \"kubernetes.io/empty-dir/485f2017-fcab-46d8-a0f8-71bf110c1836-devshm\") pod \"485f2017-fcab-46d8-a0f8-71bf110c1836\" (UID: \"485f2017-fcab-46d8-a0f8-71bf110c1836\") "
Dec 22 17:43:45 truenas k3s[16911]: I1222 17:43:45.752054   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"shared\" (UniqueName: \"kubernetes.io/empty-dir/485f2017-fcab-46d8-a0f8-71bf110c1836-shared\") pod \"485f2017-fcab-46d8-a0f8-71bf110c1836\" (UID: \"485f2017-fcab-46d8-a0f8-71bf110c1836\") "
Dec 22 17:43:45 truenas k3s[16911]: I1222 17:43:45.752055   16911 operation_generator.go:900] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/485f2017-fcab-46d8-a0f8-71bf110c1836-data" (OuterVolumeSpecName: "data") pod "485f2017-fcab-46d8-a0f8-71bf110c1836" (UID: "485f2017-fcab-46d8-a0f8-71bf110c1836"). InnerVolumeSpecName "data". PluginName "kubernetes.io/host-path", VolumeGidValue ""
Dec 22 17:43:45 truenas k3s[16911]: I1222 17:43:45.752087   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"varrun\" (UniqueName: \"kubernetes.io/empty-dir/485f2017-fcab-46d8-a0f8-71bf110c1836-varrun\") pod \"485f2017-fcab-46d8-a0f8-71bf110c1836\" (UID: \"485f2017-fcab-46d8-a0f8-71bf110c1836\") "
Dec 22 17:43:45 truenas k3s[16911]: I1222 17:43:45.752157   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"html\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\") pod \"485f2017-fcab-46d8-a0f8-71bf110c1836\" (UID: \"485f2017-fcab-46d8-a0f8-71bf110c1836\") "
Dec 22 17:43:45 truenas k3s[16911]: I1222 17:43:45.752289   16911 reconciler_common.go:295] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/host-path/485f2017-fcab-46d8-a0f8-71bf110c1836-data\") on node \"ix-truenas\" DevicePath \"\""
Dec 22 17:43:45 truenas k3s[16911]: W1222 17:43:45.752545   16911 empty_dir.go:525] Warning: Failed to clear quota on /var/lib/kubelet/pods/485f2017-fcab-46d8-a0f8-71bf110c1836/volumes/kubernetes.io~empty-dir/shared: clearQuota called, but quotas disabled
Dec 22 17:43:45 truenas k3s[16911]: I1222 17:43:45.752649   16911 operation_generator.go:900] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/485f2017-fcab-46d8-a0f8-71bf110c1836-shared" (OuterVolumeSpecName: "shared") pod "485f2017-fcab-46d8-a0f8-71bf110c1836" (UID: "485f2017-fcab-46d8-a0f8-71bf110c1836"). InnerVolumeSpecName "shared". PluginName "kubernetes.io/empty-dir", VolumeGidValue ""
Dec 22 17:43:45 truenas k3s[16911]: I1222 17:43:45.753788   16911 operation_generator.go:900] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/485f2017-fcab-46d8-a0f8-71bf110c1836-devshm" (OuterVolumeSpecName: "devshm") pod "485f2017-fcab-46d8-a0f8-71bf110c1836" (UID: "485f2017-fcab-46d8-a0f8-71bf110c1836"). InnerVolumeSpecName "devshm". PluginName "kubernetes.io/empty-dir", VolumeGidValue ""
Dec 22 17:43:45 truenas k3s[16911]: I1222 17:43:45.753856   16911 operation_generator.go:900] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/485f2017-fcab-46d8-a0f8-71bf110c1836-tmp" (OuterVolumeSpecName: "tmp") pod "485f2017-fcab-46d8-a0f8-71bf110c1836" (UID: "485f2017-fcab-46d8-a0f8-71bf110c1836"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGidValue ""
Dec 22 17:43:45 truenas k3s[16911]: I1222 17:43:45.753916   16911 operation_generator.go:900] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/485f2017-fcab-46d8-a0f8-71bf110c1836-varrun" (OuterVolumeSpecName: "varrun") pod "485f2017-fcab-46d8-a0f8-71bf110c1836" (UID: "485f2017-fcab-46d8-a0f8-71bf110c1836"). InnerVolumeSpecName "varrun". PluginName "kubernetes.io/empty-dir", VolumeGidValue ""
Dec 22 17:43:45 truenas k3s[16911]: I1222 17:43:45.754175   16911 operation_generator.go:900] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/485f2017-fcab-46d8-a0f8-71bf110c1836-varlogs" (OuterVolumeSpecName: "varlogs") pod "485f2017-fcab-46d8-a0f8-71bf110c1836" (UID: "485f2017-fcab-46d8-a0f8-71bf110c1836"). InnerVolumeSpecName "varlogs". PluginName "kubernetes.io/empty-dir", VolumeGidValue ""
Dec 22 17:43:45 truenas k3s[16911]: I1222 17:43:45.773403   16911 operation_generator.go:900] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (OuterVolumeSpecName: "config") pod "485f2017-fcab-46d8-a0f8-71bf110c1836" (UID: "485f2017-fcab-46d8-a0f8-71bf110c1836"). InnerVolumeSpecName "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850". PluginName "kubernetes.io/csi", VolumeGidValue ""
Dec 22 17:43:45 truenas k3s[16911]: I1222 17:43:45.773425   16911 operation_generator.go:900] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (OuterVolumeSpecName: "html") pod "485f2017-fcab-46d8-a0f8-71bf110c1836" (UID: "485f2017-fcab-46d8-a0f8-71bf110c1836"). InnerVolumeSpecName "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3". PluginName "kubernetes.io/csi", VolumeGidValue ""
Dec 22 17:43:45 truenas k3s[16911]: I1222 17:43:45.852870   16911 reconciler_common.go:295] "Volume detached for volume \"varlogs\" (UniqueName: \"kubernetes.io/empty-dir/485f2017-fcab-46d8-a0f8-71bf110c1836-varlogs\") on node \"ix-truenas\" DevicePath \"\""
Dec 22 17:43:45 truenas k3s[16911]: I1222 17:43:45.852896   16911 reconciler_common.go:295] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/485f2017-fcab-46d8-a0f8-71bf110c1836-tmp\") on node \"ix-truenas\" DevicePath \"\""
Dec 22 17:43:45 truenas k3s[16911]: I1222 17:43:45.852907   16911 reconciler_common.go:295] "Volume detached for volume \"devshm\" (UniqueName: \"kubernetes.io/empty-dir/485f2017-fcab-46d8-a0f8-71bf110c1836-devshm\") on node \"ix-truenas\" DevicePath \"\""
Dec 22 17:43:45 truenas k3s[16911]: I1222 17:43:45.852917   16911 reconciler_common.go:295] "Volume detached for volume \"shared\" (UniqueName: \"kubernetes.io/empty-dir/485f2017-fcab-46d8-a0f8-71bf110c1836-shared\") on node \"ix-truenas\" DevicePath \"\""
Dec 22 17:43:45 truenas k3s[16911]: I1222 17:43:45.852927   16911 reconciler_common.go:295] "Volume detached for volume \"varrun\" (UniqueName: \"kubernetes.io/empty-dir/485f2017-fcab-46d8-a0f8-71bf110c1836-varrun\") on node \"ix-truenas\" DevicePath \"\""
Dec 22 17:43:46 truenas k3s[16911]: I1222 17:43:46.281984   16911 scope.go:115] "RemoveContainer" containerID="55603d0068fb10b249fe3be2411da13f2da73f2569c7a3b2a1e48f49174c669d"
Dec 22 17:43:46 truenas k3s[16911]: I1222 17:43:46.282395   16911 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f715f15af1cdded591e4ca6c9c3cca8f1a4f1fd541f64e5b0b8104588c1cf8e3"
Dec 22 17:43:46 truenas k3s[16911]: E1222 17:43:46.282402   16911 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"adguard-home\" with CrashLoopBackOff: \"back-off 10s restarting failed container=adguard-home pod=adguard-home-57b89dd8b8-xzj2m_ix-adguard-home(015c0b2a-db07-4719-b416-3506b105713d)\"" pod="ix-adguard-home/adguard-home-57b89dd8b8-xzj2m" podUID=015c0b2a-db07-4719-b416-3506b105713d
Dec 22 17:43:46 truenas k3s[16911]: I1222 17:43:46.297228   16911 job_controller.go:514] enqueueing job ix-nextcloud/nextcloud-preview-cron-28387720
Dec 22 17:43:46 truenas k3s[16911]: I1222 17:43:46.298785   16911 job_controller.go:514] enqueueing job ix-nextcloud/nextcloud-nextcloud-cron-28387640
Dec 22 17:43:46 truenas k3s[16911]: I1222 17:43:46.760000   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/host-path/f625e3e8-e7a8-4439-8d8c-6f10b1c7978a-data\") pod \"f625e3e8-e7a8-4439-8d8c-6f10b1c7978a\" (UID: \"f625e3e8-e7a8-4439-8d8c-6f10b1c7978a\") "
Dec 22 17:43:46 truenas k3s[16911]: I1222 17:43:46.760107   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"html\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\") pod \"f625e3e8-e7a8-4439-8d8c-6f10b1c7978a\" (UID: \"f625e3e8-e7a8-4439-8d8c-6f10b1c7978a\") "
Dec 22 17:43:46 truenas k3s[16911]: I1222 17:43:46.760104   16911 operation_generator.go:900] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/f625e3e8-e7a8-4439-8d8c-6f10b1c7978a-data" (OuterVolumeSpecName: "data") pod "f625e3e8-e7a8-4439-8d8c-6f10b1c7978a" (UID: "f625e3e8-e7a8-4439-8d8c-6f10b1c7978a"). InnerVolumeSpecName "data". PluginName "kubernetes.io/host-path", VolumeGidValue ""
Dec 22 17:43:46 truenas k3s[16911]: I1222 17:43:46.760130   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"devshm\" (UniqueName: \"kubernetes.io/empty-dir/f625e3e8-e7a8-4439-8d8c-6f10b1c7978a-devshm\") pod \"f625e3e8-e7a8-4439-8d8c-6f10b1c7978a\" (UID: \"f625e3e8-e7a8-4439-8d8c-6f10b1c7978a\") "
Dec 22 17:43:46 truenas k3s[16911]: I1222 17:43:46.760183   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\") pod \"f625e3e8-e7a8-4439-8d8c-6f10b1c7978a\" (UID: \"f625e3e8-e7a8-4439-8d8c-6f10b1c7978a\") "
Dec 22 17:43:46 truenas k3s[16911]: I1222 17:43:46.760208   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"varlogs\" (UniqueName: \"kubernetes.io/empty-dir/f625e3e8-e7a8-4439-8d8c-6f10b1c7978a-varlogs\") pod \"f625e3e8-e7a8-4439-8d8c-6f10b1c7978a\" (UID: \"f625e3e8-e7a8-4439-8d8c-6f10b1c7978a\") "
Dec 22 17:43:46 truenas k3s[16911]: I1222 17:43:46.760225   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f625e3e8-e7a8-4439-8d8c-6f10b1c7978a-tmp\") pod \"f625e3e8-e7a8-4439-8d8c-6f10b1c7978a\" (UID: \"f625e3e8-e7a8-4439-8d8c-6f10b1c7978a\") "
Dec 22 17:43:46 truenas k3s[16911]: I1222 17:43:46.760240   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"shared\" (UniqueName: \"kubernetes.io/empty-dir/f625e3e8-e7a8-4439-8d8c-6f10b1c7978a-shared\") pod \"f625e3e8-e7a8-4439-8d8c-6f10b1c7978a\" (UID: \"f625e3e8-e7a8-4439-8d8c-6f10b1c7978a\") "
Dec 22 17:43:46 truenas k3s[16911]: I1222 17:43:46.760269   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"varrun\" (UniqueName: \"kubernetes.io/empty-dir/f625e3e8-e7a8-4439-8d8c-6f10b1c7978a-varrun\") pod \"f625e3e8-e7a8-4439-8d8c-6f10b1c7978a\" (UID: \"f625e3e8-e7a8-4439-8d8c-6f10b1c7978a\") "
Dec 22 17:43:46 truenas k3s[16911]: I1222 17:43:46.760368   16911 reconciler_common.go:295] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/host-path/f625e3e8-e7a8-4439-8d8c-6f10b1c7978a-data\") on node \"ix-truenas\" DevicePath \"\""
Dec 22 17:43:46 truenas k3s[16911]: W1222 17:43:46.760621   16911 empty_dir.go:525] Warning: Failed to clear quota on /var/lib/kubelet/pods/f625e3e8-e7a8-4439-8d8c-6f10b1c7978a/volumes/kubernetes.io~empty-dir/shared: clearQuota called, but quotas disabled
Dec 22 17:43:46 truenas k3s[16911]: I1222 17:43:46.760699   16911 operation_generator.go:900] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f625e3e8-e7a8-4439-8d8c-6f10b1c7978a-shared" (OuterVolumeSpecName: "shared") pod "f625e3e8-e7a8-4439-8d8c-6f10b1c7978a" (UID: "f625e3e8-e7a8-4439-8d8c-6f10b1c7978a"). InnerVolumeSpecName "shared". PluginName "kubernetes.io/empty-dir", VolumeGidValue ""
Dec 22 17:43:46 truenas k3s[16911]: I1222 17:43:46.761769   16911 operation_generator.go:900] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f625e3e8-e7a8-4439-8d8c-6f10b1c7978a-devshm" (OuterVolumeSpecName: "devshm") pod "f625e3e8-e7a8-4439-8d8c-6f10b1c7978a" (UID: "f625e3e8-e7a8-4439-8d8c-6f10b1c7978a"). InnerVolumeSpecName "devshm". PluginName "kubernetes.io/empty-dir", VolumeGidValue ""
Dec 22 17:43:46 truenas k3s[16911]: I1222 17:43:46.761840   16911 operation_generator.go:900] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f625e3e8-e7a8-4439-8d8c-6f10b1c7978a-varlogs" (OuterVolumeSpecName: "varlogs") pod "f625e3e8-e7a8-4439-8d8c-6f10b1c7978a" (UID: "f625e3e8-e7a8-4439-8d8c-6f10b1c7978a"). InnerVolumeSpecName "varlogs". PluginName "kubernetes.io/empty-dir", VolumeGidValue ""
Dec 22 17:43:46 truenas k3s[16911]: I1222 17:43:46.761886   16911 operation_generator.go:900] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f625e3e8-e7a8-4439-8d8c-6f10b1c7978a-tmp" (OuterVolumeSpecName: "tmp") pod "f625e3e8-e7a8-4439-8d8c-6f10b1c7978a" (UID: "f625e3e8-e7a8-4439-8d8c-6f10b1c7978a"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGidValue ""
Dec 22 17:43:46 truenas k3s[16911]: I1222 17:43:46.761963   16911 operation_generator.go:900] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f625e3e8-e7a8-4439-8d8c-6f10b1c7978a-varrun" (OuterVolumeSpecName: "varrun") pod "f625e3e8-e7a8-4439-8d8c-6f10b1c7978a" (UID: "f625e3e8-e7a8-4439-8d8c-6f10b1c7978a"). InnerVolumeSpecName "varrun". PluginName "kubernetes.io/empty-dir", VolumeGidValue ""
Dec 22 17:43:46 truenas k3s[16911]: I1222 17:43:46.775092   16911 operation_generator.go:900] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (OuterVolumeSpecName: "html") pod "f625e3e8-e7a8-4439-8d8c-6f10b1c7978a" (UID: "f625e3e8-e7a8-4439-8d8c-6f10b1c7978a"). InnerVolumeSpecName "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3". PluginName "kubernetes.io/csi", VolumeGidValue ""
Dec 22 17:43:46 truenas k3s[16911]: I1222 17:43:46.775105   16911 operation_generator.go:900] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (OuterVolumeSpecName: "config") pod "f625e3e8-e7a8-4439-8d8c-6f10b1c7978a" (UID: "f625e3e8-e7a8-4439-8d8c-6f10b1c7978a"). InnerVolumeSpecName "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850". PluginName "kubernetes.io/csi", VolumeGidValue ""
Dec 22 17:43:46 truenas k3s[16911]: I1222 17:43:46.861391   16911 reconciler_common.go:295] "Volume detached for volume \"varrun\" (UniqueName: \"kubernetes.io/empty-dir/f625e3e8-e7a8-4439-8d8c-6f10b1c7978a-varrun\") on node \"ix-truenas\" DevicePath \"\""
Dec 22 17:43:46 truenas k3s[16911]: I1222 17:43:46.861409   16911 reconciler_common.go:295] "Volume detached for volume \"devshm\" (UniqueName: \"kubernetes.io/empty-dir/f625e3e8-e7a8-4439-8d8c-6f10b1c7978a-devshm\") on node \"ix-truenas\" DevicePath \"\""
Dec 22 17:43:46 truenas k3s[16911]: I1222 17:43:46.861417   16911 reconciler_common.go:295] "Volume detached for volume \"varlogs\" (UniqueName: \"kubernetes.io/empty-dir/f625e3e8-e7a8-4439-8d8c-6f10b1c7978a-varlogs\") on node \"ix-truenas\" DevicePath \"\""
Dec 22 17:43:46 truenas k3s[16911]: I1222 17:43:46.861425   16911 reconciler_common.go:295] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f625e3e8-e7a8-4439-8d8c-6f10b1c7978a-tmp\") on node \"ix-truenas\" DevicePath \"\""
Dec 22 17:43:46 truenas k3s[16911]: I1222 17:43:46.861431   16911 reconciler_common.go:295] "Volume detached for volume \"shared\" (UniqueName: \"kubernetes.io/empty-dir/f625e3e8-e7a8-4439-8d8c-6f10b1c7978a-shared\") on node \"ix-truenas\" DevicePath \"\""
Dec 22 17:43:47 truenas k3s[16911]: I1222 17:43:47.285787   16911 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="175b61b8d69a1046535b4e1e783caa7f5fb84b52d57a064ee94a97443026c559"
Dec 22 17:43:47 truenas k3s[16911]: I1222 17:43:47.297514   16911 job_controller.go:514] enqueueing job ix-nextcloud/nextcloud-preview-cron-28387720
Dec 22 17:43:47 truenas k3s[16911]: I1222 17:43:47.305526   16911 job_controller.go:514] enqueueing job ix-nextcloud/nextcloud-nextcloud-cron-28387640
Dec 22 17:43:47 truenas k3s[16911]: I1222 17:43:47.305696   16911 event.go:294] "Event occurred" object="ix-nextcloud/nextcloud-nextcloud-cron" fieldPath="" kind="CronJob" apiVersion="batch/v1" type="Normal" reason="JobAlreadyActive" message="Not starting job because prior execution is running and concurrency policy is Forbid"
Dec 22 17:43:47 truenas k3s[16911]: I1222 17:43:47.310119   16911 job_controller.go:514] enqueueing job ix-nextcloud/nextcloud-nextcloud-cron-28387640
Dec 22 17:43:47 truenas k3s[16911]: I1222 17:43:47.315608   16911 job_controller.go:514] enqueueing job ix-nextcloud/nextcloud-nextcloud-cron-28387640
Dec 22 17:43:47 truenas k3s[16911]: I1222 17:43:47.315606   16911 event.go:294] "Event occurred" object="ix-nextcloud/nextcloud-nextcloud-cron-28387640" fieldPath="" kind="Job" apiVersion="batch/v1" type="Normal" reason="Completed" message="Job completed"
Dec 22 17:43:47 truenas k3s[16911]: I1222 17:43:47.315737   16911 event.go:294] "Event occurred" object="ix-nextcloud/nextcloud-nextcloud-cron" fieldPath="" kind="CronJob" apiVersion="batch/v1" type="Normal" reason="SawCompletedJob" message="Saw completed job: nextcloud-nextcloud-cron-28387640, status: Complete"
Dec 22 17:43:47 truenas k3s[16911]: I1222 17:43:47.319628   16911 job_controller.go:514] enqueueing job ix-nextcloud/nextcloud-nextcloud-cron-28387640
Dec 22 17:43:47 truenas k3s[16911]: I1222 17:43:47.319629   16911 job_controller.go:514] enqueueing job ix-nextcloud/nextcloud-nextcloud-cron-28387720
Dec 22 17:43:47 truenas k3s[16911]: I1222 17:43:47.319715   16911 event.go:294] "Event occurred" object="ix-nextcloud/nextcloud-nextcloud-cron" fieldPath="" kind="CronJob" apiVersion="batch/v1" type="Normal" reason="SuccessfulCreate" message="Created job nextcloud-nextcloud-cron-28387720"
Dec 22 17:43:47 truenas k3s[16911]: I1222 17:43:47.325103   16911 event.go:294] "Event occurred" object="ix-nextcloud/nextcloud-nextcloud-cron-28387720" fieldPath="" kind="Job" apiVersion="batch/v1" type="Normal" reason="SuccessfulCreate" message="Created pod: nextcloud-nextcloud-cron-28387720-n2shl"
Dec 22 17:43:47 truenas k3s[16911]: I1222 17:43:47.325259   16911 job_controller.go:514] enqueueing job ix-nextcloud/nextcloud-nextcloud-cron-28387720
Dec 22 17:43:47 truenas k3s[16911]: I1222 17:43:47.328894   16911 job_controller.go:514] enqueueing job ix-nextcloud/nextcloud-nextcloud-cron-28387720
Dec 22 17:43:47 truenas k3s[16911]: I1222 17:43:47.330602   16911 job_controller.go:514] enqueueing job ix-nextcloud/nextcloud-nextcloud-cron-28387720
Dec 22 17:43:47 truenas k3s[16911]: I1222 17:43:47.330697   16911 topology_manager.go:210] "Topology Admit Handler"
Dec 22 17:43:47 truenas k3s[16911]: E1222 17:43:47.330792   16911 cpu_manager.go:395] "RemoveStaleState: removing container" podUID="485f2017-fcab-46d8-a0f8-71bf110c1836" containerName="nextcloud"
Dec 22 17:43:47 truenas k3s[16911]: E1222 17:43:47.330803   16911 cpu_manager.go:395] "RemoveStaleState: removing container" podUID="f625e3e8-e7a8-4439-8d8c-6f10b1c7978a" containerName="nextcloud"
Dec 22 17:43:47 truenas k3s[16911]: I1222 17:43:47.330872   16911 memory_manager.go:346] "RemoveStaleState removing state" podUID="f625e3e8-e7a8-4439-8d8c-6f10b1c7978a" containerName="nextcloud"
Dec 22 17:43:47 truenas k3s[16911]: I1222 17:43:47.330880   16911 memory_manager.go:346] "RemoveStaleState removing state" podUID="485f2017-fcab-46d8-a0f8-71bf110c1836" containerName="nextcloud"
Dec 22 17:43:47 truenas k3s[16911]: E1222 17:43:47.332272   16911 cronjob_controllerv2.go:166] error syncing CronJobController ix-nextcloud/nextcloud-nextcloud-cron, requeuing: Operation cannot be fulfilled on cronjobs.batch "nextcloud-nextcloud-cron": the object has been modified; please apply your changes to the latest version and try again
Dec 22 17:43:47 truenas k3s[16911]: I1222 17:43:47.338275   16911 job_controller.go:514] enqueueing job ix-nextcloud/nextcloud-nextcloud-cron-28387720
Dec 22 17:43:47 truenas k3s[16911]: I1222 17:43:47.769305   16911 reconciler_common.go:253] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/host-path/47956676-adbc-44a0-bc6e-daad63f700a8-data\") pod \"nextcloud-nextcloud-cron-28387720-n2shl\" (UID: \"47956676-adbc-44a0-bc6e-daad63f700a8\") " pod="ix-nextcloud/nextcloud-nextcloud-cron-28387720-n2shl"
Dec 22 17:43:47 truenas k3s[16911]: I1222 17:43:47.769350   16911 reconciler_common.go:253] "operationExecutor.VerifyControllerAttachedVolume started for volume \"devshm\" (UniqueName: \"kubernetes.io/empty-dir/47956676-adbc-44a0-bc6e-daad63f700a8-devshm\") pod \"nextcloud-nextcloud-cron-28387720-n2shl\" (UID: \"47956676-adbc-44a0-bc6e-daad63f700a8\") " pod="ix-nextcloud/nextcloud-nextcloud-cron-28387720-n2shl"
Dec 22 17:43:48 truenas k3s[16911]: I1222 17:43:48.375620   16911 reconciler_common.go:253] "operationExecutor.VerifyControllerAttachedVolume started for volume \"shared\" (UniqueName: \"kubernetes.io/empty-dir/47956676-adbc-44a0-bc6e-daad63f700a8-shared\") pod \"nextcloud-nextcloud-cron-28387720-n2shl\" (UID: \"47956676-adbc-44a0-bc6e-daad63f700a8\") " pod="ix-nextcloud/nextcloud-nextcloud-cron-28387720-n2shl"
Dec 22 17:43:48 truenas k3s[16911]: I1222 17:43:48.375667   16911 reconciler_common.go:253] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47956676-adbc-44a0-bc6e-daad63f700a8-tmp\") pod \"nextcloud-nextcloud-cron-28387720-n2shl\" (UID: \"47956676-adbc-44a0-bc6e-daad63f700a8\") " pod="ix-nextcloud/nextcloud-nextcloud-cron-28387720-n2shl"
Dec 22 17:43:48 truenas k3s[16911]: I1222 17:43:48.375754   16911 reconciler_common.go:253] "operationExecutor.VerifyControllerAttachedVolume started for volume \"varlogs\" (UniqueName: \"kubernetes.io/empty-dir/47956676-adbc-44a0-bc6e-daad63f700a8-varlogs\") pod \"nextcloud-nextcloud-cron-28387720-n2shl\" (UID: \"47956676-adbc-44a0-bc6e-daad63f700a8\") " pod="ix-nextcloud/nextcloud-nextcloud-cron-28387720-n2shl"
Dec 22 17:43:48 truenas k3s[16911]: I1222 17:43:48.375789   16911 reconciler_common.go:253] "operationExecutor.VerifyControllerAttachedVolume started for volume \"varrun\" (UniqueName: \"kubernetes.io/empty-dir/47956676-adbc-44a0-bc6e-daad63f700a8-varrun\") pod \"nextcloud-nextcloud-cron-28387720-n2shl\" (UID: \"47956676-adbc-44a0-bc6e-daad63f700a8\") " pod="ix-nextcloud/nextcloud-nextcloud-cron-28387720-n2shl"
Dec 22 17:43:48 truenas k3s[16911]: I1222 17:43:48.380227   16911 job_controller.go:514] enqueueing job ix-nextcloud/nextcloud-preview-cron-28387720
Dec 22 17:43:48 truenas k3s[16911]: I1222 17:43:48.539112   16911 job_controller.go:514] enqueueing job ix-nextcloud/nextcloud-preview-cron-28387720
Dec 22 17:43:48 truenas k3s[16911]: I1222 17:43:48.543239   16911 job_controller.go:514] enqueueing job ix-nextcloud/nextcloud-preview-cron-28387720
Dec 22 17:43:48 truenas k3s[16911]: I1222 17:43:48.543241   16911 event.go:294] "Event occurred" object="ix-nextcloud/nextcloud-preview-cron-28387720" fieldPath="" kind="Job" apiVersion="batch/v1" type="Normal" reason="Completed" message="Job completed"
Dec 22 17:43:48 truenas k3s[16911]: I1222 17:43:48.543289   16911 event.go:294] "Event occurred" object="ix-nextcloud/nextcloud-preview-cron" fieldPath="" kind="CronJob" apiVersion="batch/v1" type="Normal" reason="SawCompletedJob" message="Saw completed job: nextcloud-preview-cron-28387720, status: Complete"
Dec 22 17:43:49 truenas k3s[16911]: I1222 17:43:49.011244   16911 job_controller.go:514] enqueueing job ix-nextcloud/nextcloud-nextcloud-cron-28387720
Dec 22 17:43:49 truenas k3s[16911]: I1222 17:43:49.561881   16911 scope.go:115] "RemoveContainer" containerID="55603d0068fb10b249fe3be2411da13f2da73f2569c7a3b2a1e48f49174c669d"
Dec 22 17:43:49 truenas k3s[16911]: E1222 17:43:49.562265   16911 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"adguard-home\" with CrashLoopBackOff: \"back-off 10s restarting failed container=adguard-home pod=adguard-home-57b89dd8b8-xzj2m_ix-adguard-home(015c0b2a-db07-4719-b416-3506b105713d)\"" pod="ix-adguard-home/adguard-home-57b89dd8b8-xzj2m" podUID=015c0b2a-db07-4719-b416-3506b105713d
Dec 22 17:43:50 truenas k3s[16911]: I1222 17:43:50.292084   16911 scope.go:115] "RemoveContainer" containerID="55603d0068fb10b249fe3be2411da13f2da73f2569c7a3b2a1e48f49174c669d"
Dec 22 17:43:50 truenas k3s[16911]: E1222 17:43:50.292409   16911 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"adguard-home\" with CrashLoopBackOff: \"back-off 10s restarting failed container=adguard-home pod=adguard-home-57b89dd8b8-xzj2m_ix-adguard-home(015c0b2a-db07-4719-b416-3506b105713d)\"" pod="ix-adguard-home/adguard-home-57b89dd8b8-xzj2m" podUID=015c0b2a-db07-4719-b416-3506b105713d
Dec 22 17:43:52 truenas k3s[16911]: I1222 17:43:52.304093   16911 job_controller.go:514] enqueueing job ix-nextcloud/nextcloud-nextcloud-cron-28387720
Dec 22 17:43:52 truenas k3s[16911]: I1222 17:43:52.304707   16911 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="ix-nextcloud/nextcloud-nextcloud-cron-28387720-n2shl" podStartSLOduration=5.304681673 pod.CreationTimestamp="2023-12-22 17:43:47 +0100 CET" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2023-12-22 17:43:52.30380311 +0100 CET m=+90.090608911" watchObservedRunningTime="2023-12-22 17:43:52.304681673 +0100 CET m=+90.091487464"
Dec 22 17:43:53 truenas k3s[16911]: I1222 17:43:53.309846   16911 job_controller.go:514] enqueueing job ix-nextcloud/nextcloud-nextcloud-cron-28387720
Dec 22 17:43:53 truenas k3s[16911]: I1222 17:43:53.310895   16911 job_controller.go:514] enqueueing job ix-nextcloud/nextcloud-nextcloud-cron-28387720
Dec 22 17:43:54 truenas k3s[16911]: I1222 17:43:54.315913   16911 job_controller.go:514] enqueueing job ix-nextcloud/nextcloud-nextcloud-cron-28387720
Dec 22 17:43:54 truenas k3s[16911]: I1222 17:43:54.561978   16911 scope.go:115] "RemoveContainer" containerID="55603d0068fb10b249fe3be2411da13f2da73f2569c7a3b2a1e48f49174c669d"
Dec 22 17:43:55 truenas k3s[16911]: I1222 17:43:55.306833   16911 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d34d065aada0cbe9080d52163d68e29637640eb5b61211db69a142bd74d99ca"
Dec 22 17:43:55 truenas k3s[16911]: I1222 17:43:55.332151   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"varrun\" (UniqueName: \"kubernetes.io/empty-dir/47956676-adbc-44a0-bc6e-daad63f700a8-varrun\") pod \"47956676-adbc-44a0-bc6e-daad63f700a8\" (UID: \"47956676-adbc-44a0-bc6e-daad63f700a8\") "
Dec 22 17:43:55 truenas k3s[16911]: I1222 17:43:55.332219   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"varlogs\" (UniqueName: \"kubernetes.io/empty-dir/47956676-adbc-44a0-bc6e-daad63f700a8-varlogs\") pod \"47956676-adbc-44a0-bc6e-daad63f700a8\" (UID: \"47956676-adbc-44a0-bc6e-daad63f700a8\") "
Dec 22 17:43:55 truenas k3s[16911]: I1222 17:43:55.332248   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/host-path/47956676-adbc-44a0-bc6e-daad63f700a8-data\") pod \"47956676-adbc-44a0-bc6e-daad63f700a8\" (UID: \"47956676-adbc-44a0-bc6e-daad63f700a8\") "
Dec 22 17:43:55 truenas k3s[16911]: I1222 17:43:55.332319   16911 operation_generator.go:900] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/47956676-adbc-44a0-bc6e-daad63f700a8-data" (OuterVolumeSpecName: "data") pod "47956676-adbc-44a0-bc6e-daad63f700a8" (UID: "47956676-adbc-44a0-bc6e-daad63f700a8"). InnerVolumeSpecName "data". PluginName "kubernetes.io/host-path", VolumeGidValue ""
Dec 22 17:43:55 truenas k3s[16911]: I1222 17:43:55.332404   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"html\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3\") pod \"47956676-adbc-44a0-bc6e-daad63f700a8\" (UID: \"47956676-adbc-44a0-bc6e-daad63f700a8\") "
Dec 22 17:43:55 truenas k3s[16911]: I1222 17:43:55.332469   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850\") pod \"47956676-adbc-44a0-bc6e-daad63f700a8\" (UID: \"47956676-adbc-44a0-bc6e-daad63f700a8\") "
Dec 22 17:43:55 truenas k3s[16911]: I1222 17:43:55.332506   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"shared\" (UniqueName: \"kubernetes.io/empty-dir/47956676-adbc-44a0-bc6e-daad63f700a8-shared\") pod \"47956676-adbc-44a0-bc6e-daad63f700a8\" (UID: \"47956676-adbc-44a0-bc6e-daad63f700a8\") "
Dec 22 17:43:55 truenas k3s[16911]: I1222 17:43:55.332530   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47956676-adbc-44a0-bc6e-daad63f700a8-tmp\") pod \"47956676-adbc-44a0-bc6e-daad63f700a8\" (UID: \"47956676-adbc-44a0-bc6e-daad63f700a8\") "
Dec 22 17:43:55 truenas k3s[16911]: I1222 17:43:55.332558   16911 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"devshm\" (UniqueName: \"kubernetes.io/empty-dir/47956676-adbc-44a0-bc6e-daad63f700a8-devshm\") pod \"47956676-adbc-44a0-bc6e-daad63f700a8\" (UID: \"47956676-adbc-44a0-bc6e-daad63f700a8\") "
Dec 22 17:43:55 truenas k3s[16911]: I1222 17:43:55.332703   16911 reconciler_common.go:295] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/host-path/47956676-adbc-44a0-bc6e-daad63f700a8-data\") on node \"ix-truenas\" DevicePath \"\""
Dec 22 17:43:55 truenas k3s[16911]: W1222 17:43:55.332779   16911 empty_dir.go:525] Warning: Failed to clear quota on /var/lib/kubelet/pods/47956676-adbc-44a0-bc6e-daad63f700a8/volumes/kubernetes.io~empty-dir/shared: clearQuota called, but quotas disabled
Dec 22 17:43:55 truenas k3s[16911]: I1222 17:43:55.332850   16911 operation_generator.go:900] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47956676-adbc-44a0-bc6e-daad63f700a8-shared" (OuterVolumeSpecName: "shared") pod "47956676-adbc-44a0-bc6e-daad63f700a8" (UID: "47956676-adbc-44a0-bc6e-daad63f700a8"). InnerVolumeSpecName "shared". PluginName "kubernetes.io/empty-dir", VolumeGidValue ""
Dec 22 17:43:55 truenas k3s[16911]: I1222 17:43:55.333849   16911 operation_generator.go:900] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47956676-adbc-44a0-bc6e-daad63f700a8-varrun" (OuterVolumeSpecName: "varrun") pod "47956676-adbc-44a0-bc6e-daad63f700a8" (UID: "47956676-adbc-44a0-bc6e-daad63f700a8"). InnerVolumeSpecName "varrun". PluginName "kubernetes.io/empty-dir", VolumeGidValue ""
Dec 22 17:43:55 truenas k3s[16911]: I1222 17:43:55.333930   16911 operation_generator.go:900] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47956676-adbc-44a0-bc6e-daad63f700a8-tmp" (OuterVolumeSpecName: "tmp") pod "47956676-adbc-44a0-bc6e-daad63f700a8" (UID: "47956676-adbc-44a0-bc6e-daad63f700a8"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGidValue ""
Dec 22 17:43:55 truenas k3s[16911]: I1222 17:43:55.333973   16911 operation_generator.go:900] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47956676-adbc-44a0-bc6e-daad63f700a8-varlogs" (OuterVolumeSpecName: "varlogs") pod "47956676-adbc-44a0-bc6e-daad63f700a8" (UID: "47956676-adbc-44a0-bc6e-daad63f700a8"). InnerVolumeSpecName "varlogs". PluginName "kubernetes.io/empty-dir", VolumeGidValue ""
Dec 22 17:43:55 truenas k3s[16911]: I1222 17:43:55.334043   16911 operation_generator.go:900] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47956676-adbc-44a0-bc6e-daad63f700a8-devshm" (OuterVolumeSpecName: "devshm") pod "47956676-adbc-44a0-bc6e-daad63f700a8" (UID: "47956676-adbc-44a0-bc6e-daad63f700a8"). InnerVolumeSpecName "devshm". PluginName "kubernetes.io/empty-dir", VolumeGidValue ""
Dec 22 17:43:55 truenas k3s[16911]: I1222 17:43:55.346360   16911 operation_generator.go:900] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/zfs.csi.openebs.io^pvc-de5f127f-cc69-490c-b3b7-ebbd34492850" (OuterVolumeSpecName: "config") pod "47956676-adbc-44a0-bc6e-daad63f700a8" (UID: "47956676-adbc-44a0-bc6e-daad63f700a8"). InnerVolumeSpecName "pvc-de5f127f-cc69-490c-b3b7-ebbd34492850". PluginName "kubernetes.io/csi", VolumeGidValue ""
Dec 22 17:43:55 truenas k3s[16911]: I1222 17:43:55.347266   16911 operation_generator.go:900] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3" (OuterVolumeSpecName: "html") pod "47956676-adbc-44a0-bc6e-daad63f700a8" (UID: "47956676-adbc-44a0-bc6e-daad63f700a8"). InnerVolumeSpecName "pvc-9e3dc448-974f-4b18-bc2a-794ce2c6a3d3". PluginName "kubernetes.io/csi", VolumeGidValue ""
Dec 22 17:43:55 truenas k3s[16911]: I1222 17:43:55.433734   16911 reconciler_common.go:295] "Volume detached for volume \"varlogs\" (UniqueName: \"kubernetes.io/empty-dir/47956676-adbc-44a0-bc6e-daad63f700a8-varlogs\") on node \"ix-truenas\" DevicePath \"\""
Dec 22 17:43:55 truenas k3s[16911]: I1222 17:43:55.433753   16911 reconciler_common.go:295] "Volume detached for volume \"shared\" (UniqueName: \"kubernetes.io/empty-dir/47956676-adbc-44a0-bc6e-daad63f700a8-shared\") on node \"ix-truenas\" DevicePath \"\""
Dec 22 17:43:55 truenas k3s[16911]: I1222 17:43:55.433761   16911 reconciler_common.go:295] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47956676-adbc-44a0-bc6e-daad63f700a8-tmp\") on node \"ix-truenas\" DevicePath \"\""
Dec 22 17:43:55 truenas k3s[16911]: I1222 17:43:55.433773   16911 reconciler_common.go:295] "Volume detached for volume \"devshm\" (UniqueName: \"kubernetes.io/empty-dir/47956676-adbc-44a0-bc6e-daad63f700a8-devshm\") on node \"ix-truenas\" DevicePath \"\""
Dec 22 17:43:55 truenas k3s[16911]: I1222 17:43:55.433786   16911 reconciler_common.go:295] "Volume detached for volume \"varrun\" (UniqueName: \"kubernetes.io/empty-dir/47956676-adbc-44a0-bc6e-daad63f700a8-varrun\") on node \"ix-truenas\" DevicePath \"\""
Dec 22 17:43:55 truenas k3s[16911]: I1222 17:43:55.616910   16911 job_controller.go:514] enqueueing job ix-nextcloud/nextcloud-nextcloud-cron-28387720
Dec 22 17:43:56 truenas k3s[16911]: I1222 17:43:56.621977   16911 job_controller.go:514] enqueueing job ix-nextcloud/nextcloud-nextcloud-cron-28387720
Dec 22 17:43:56 truenas k3s[16911]: I1222 17:43:56.627022   16911 job_controller.go:514] enqueueing job ix-nextcloud/nextcloud-nextcloud-cron-28387720
Dec 22 17:43:56 truenas k3s[16911]: I1222 17:43:56.630405   16911 event.go:294] "Event occurred" object="ix-nextcloud/nextcloud-nextcloud-cron-28387720" fieldPath="" kind="Job" apiVersion="batch/v1" type="Normal" reason="Completed" message="Job completed"
Dec 22 17:43:56 truenas k3s[16911]: I1222 17:43:56.630450   16911 job_controller.go:514] enqueueing job ix-nextcloud/nextcloud-nextcloud-cron-28387720
Dec 22 17:43:56 truenas k3s[16911]: I1222 17:43:56.630599   16911 event.go:294] "Event occurred" object="ix-nextcloud/nextcloud-nextcloud-cron" fieldPath="" kind="CronJob" apiVersion="batch/v1" type="Normal" reason="SawCompletedJob" message="Saw completed job: nextcloud-nextcloud-cron-28387720, status: Complete"
Dec 22 17:43:56 truenas k3s[16911]: I1222 17:43:56.634028   16911 job_controller.go:514] enqueueing job ix-nextcloud/nextcloud-nextcloud-cron-28387720
Dec 22 17:43:58 truenas k3s[16911]: I1222 17:43:58.123678   16911 gc_controller.go:207] "Garbage collecting pods" numPods=3
Dec 22 17:43:58 truenas k3s[16911]: I1222 17:43:58.123712   16911 gc_controller.go:329] "PodGC is force deleting Pod" pod="ix-nextcloud/nextcloud-imaginary-6fdbd88855-bvsb9"
Dec 22 17:43:58 truenas k3s[16911]: I1222 17:43:58.123732   16911 gc_controller.go:329] "PodGC is force deleting Pod" pod="ix-nextcloud/nextcloud-c9c757d4f-6mpz8"
Dec 22 17:43:58 truenas k3s[16911]: I1222 17:43:58.123740   16911 gc_controller.go:329] "PodGC is force deleting Pod" pod="ix-nextcloud/nextcloud-cnpg-main-1"
Dec 22 17:43:58 truenas k3s[16911]: I1222 17:43:58.167917   16911 topology_manager.go:210] "Topology Admit Handler"
Dec 22 17:43:58 truenas k3s[16911]: E1222 17:43:58.167995   16911 cpu_manager.go:395] "RemoveStaleState: removing container" podUID="47956676-adbc-44a0-bc6e-daad63f700a8" containerName="nextcloud"
Dec 22 17:43:58 truenas k3s[16911]: I1222 17:43:58.168045   16911 memory_manager.go:346] "RemoveStaleState removing state" podUID="47956676-adbc-44a0-bc6e-daad63f700a8" containerName="nextcloud"
Dec 22 17:43:58 truenas k3s[16911]: I1222 17:43:58.258491   16911 reconciler_common.go:253] "operationExecutor.VerifyControllerAttachedVolume started for volume \"scratch-data\" (UniqueName: \"kubernetes.io/empty-dir/0370e6ed-21ab-4a40-a325-556c03675a3a-scratch-data\") pod \"nextcloud-cnpg-main-1\" (UID: \"0370e6ed-21ab-4a40-a325-556c03675a3a\") " pod="ix-nextcloud/nextcloud-cnpg-main-1"
Dec 22 17:43:58 truenas k3s[16911]: I1222 17:43:58.258534   16911 reconciler_common.go:253] "operationExecutor.VerifyControllerAttachedVolume started for volume \"shm\" (UniqueName: \"kubernetes.io/empty-dir/0370e6ed-21ab-4a40-a325-556c03675a3a-shm\") pod \"nextcloud-cnpg-main-1\" (UID: \"0370e6ed-21ab-4a40-a325-556c03675a3a\") " pod="ix-nextcloud/nextcloud-cnpg-main-1"
Dec 22 17:43:58 truenas k3s[16911]: I1222 17:43:58.258557   16911 reconciler_common.go:253] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-346c83fb-a449-41eb-bc74-080578ba1326\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-346c83fb-a449-41eb-bc74-080578ba1326\") pod \"nextcloud-cnpg-main-1\" (UID: \"0370e6ed-21ab-4a40-a325-556c03675a3a\") " pod="ix-nextcloud/nextcloud-cnpg-main-1"
Dec 22 17:43:58 truenas k3s[16911]: I1222 17:43:58.258637   16911 reconciler_common.go:253] "operationExecutor.VerifyControllerAttachedVolume started for volume \"superuser-secret\" (UniqueName: \"kubernetes.io/secret/0370e6ed-21ab-4a40-a325-556c03675a3a-superuser-secret\") pod \"nextcloud-cnpg-main-1\" (UID: \"0370e6ed-21ab-4a40-a325-556c03675a3a\") " pod="ix-nextcloud/nextcloud-cnpg-main-1"
Dec 22 17:43:58 truenas k3s[16911]: I1222 17:43:58.258666   16911 reconciler_common.go:253] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-a8dd155b-b0ab-42ff-975f-7c411496e41c\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-a8dd155b-b0ab-42ff-975f-7c411496e41c\") pod \"nextcloud-cnpg-main-1\" (UID: \"0370e6ed-21ab-4a40-a325-556c03675a3a\") " pod="ix-nextcloud/nextcloud-cnpg-main-1"
Dec 22 17:43:58 truenas k3s[16911]: I1222 17:43:58.258688   16911 reconciler_common.go:253] "operationExecutor.VerifyControllerAttachedVolume started for volume \"app-secret\" (UniqueName: \"kubernetes.io/secret/0370e6ed-21ab-4a40-a325-556c03675a3a-app-secret\") pod \"nextcloud-cnpg-main-1\" (UID: \"0370e6ed-21ab-4a40-a325-556c03675a3a\") " pod="ix-nextcloud/nextcloud-cnpg-main-1"
Dec 22 17:43:58 truenas k3s[16911]: I1222 17:43:58.258770   16911 reconciler_common.go:253] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klrzf\" (UniqueName: \"kubernetes.io/projected/0370e6ed-21ab-4a40-a325-556c03675a3a-kube-api-access-klrzf\") pod \"nextcloud-cnpg-main-1\" (UID: \"0370e6ed-21ab-4a40-a325-556c03675a3a\") " pod="ix-nextcloud/nextcloud-cnpg-main-1"
Dec 22 17:43:58 truenas k3s[16911]: I1222 17:43:58.361451   16911 csi_attacher.go:376] kubernetes.io/csi: attacher.MountDevice STAGE_UNSTAGE_VOLUME capability not set. Skipping MountDevice...
Dec 22 17:43:58 truenas k3s[16911]: I1222 17:43:58.361486   16911 operation_generator.go:658] "MountVolume.MountDevice succeeded for volume \"pvc-a8dd155b-b0ab-42ff-975f-7c411496e41c\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-a8dd155b-b0ab-42ff-975f-7c411496e41c\") pod \"nextcloud-cnpg-main-1\" (UID: \"0370e6ed-21ab-4a40-a325-556c03675a3a\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/zfs.csi.openebs.io/b65489a53a63e3cd3e77eed1ae85b0796225072c4c0d1836cca80729f255eaba/globalmount\"" pod="ix-nextcloud/nextcloud-cnpg-main-1"
Dec 22 17:43:58 truenas k3s[16911]: I1222 17:43:58.361510   16911 csi_attacher.go:376] kubernetes.io/csi: attacher.MountDevice STAGE_UNSTAGE_VOLUME capability not set. Skipping MountDevice...
Dec 22 17:43:58 truenas k3s[16911]: I1222 17:43:58.361545   16911 operation_generator.go:658] "MountVolume.MountDevice succeeded for volume \"pvc-346c83fb-a449-41eb-bc74-080578ba1326\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-346c83fb-a449-41eb-bc74-080578ba1326\") pod \"nextcloud-cnpg-main-1\" (UID: \"0370e6ed-21ab-4a40-a325-556c03675a3a\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/zfs.csi.openebs.io/3ea5c32d7e10f0510707222463657498b85563157b11ddbcb552fa6f775e5fa8/globalmount\"" pod="ix-nextcloud/nextcloud-cnpg-main-1"
Dec 22 17:43:58 truenas k3s[16911]: I1222 17:43:58.648711   16911 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=26601002-2b7f-44d7-b247-12092d0a23c8 path="/var/lib/kubelet/pods/26601002-2b7f-44d7-b247-12092d0a23c8/volumes"
Dec 22 17:43:58 truenas k3s[16911]: I1222 17:43:58.650350   16911 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=4695e46e-d3a5-41b0-83b9-ee7eff0b8044 path="/var/lib/kubelet/pods/4695e46e-d3a5-41b0-83b9-ee7eff0b8044/volumes"
Dec 22 17:43:58 truenas k3s[16911]: I1222 17:43:58.652172   16911 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=f71d37a0-295a-4e20-850d-a30130211b6a path="/var/lib/kubelet/pods/f71d37a0-295a-4e20-850d-a30130211b6a/volumes"
Dec 22 17:44:02 truenas k3s[16911]: I1222 17:44:02.325195   16911 scope.go:115] "RemoveContainer" containerID="55603d0068fb10b249fe3be2411da13f2da73f2569c7a3b2a1e48f49174c669d"
Dec 22 17:44:02 truenas k3s[16911]: I1222 17:44:02.325491   16911 scope.go:115] "RemoveContainer" containerID="9cb280d1ef5ba704dc6e17b400b0e60cffd08b400b827f8bb256a86099e42a7c"
Dec 22 17:44:02 truenas k3s[16911]: E1222 17:44:02.325802   16911 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"adguard-home\" with CrashLoopBackOff: \"back-off 20s restarting failed container=adguard-home pod=adguard-home-57b89dd8b8-xzj2m_ix-adguard-home(015c0b2a-db07-4719-b416-3506b105713d)\"" pod="ix-adguard-home/adguard-home-57b89dd8b8-xzj2m" podUID=015c0b2a-db07-4719-b416-3506b105713d
Dec 22 17:44:04 truenas k3s[16911]: I1222 17:44:04.479619   16911 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="ix-nextcloud/nextcloud-cnpg-main-1" podStartSLOduration=6.479581244 pod.CreationTimestamp="2023-12-22 17:43:58 +0100 CET" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2023-12-22 17:44:04.156360345 +0100 CET m=+101.943166136" watchObservedRunningTime="2023-12-22 17:44:04.479581244 +0100 CET m=+102.266387035"
Dec 22 17:44:04 truenas k3s[16911]: I1222 17:44:04.562285   16911 scope.go:115] "RemoveContainer" containerID="9cb280d1ef5ba704dc6e17b400b0e60cffd08b400b827f8bb256a86099e42a7c"
Dec 22 17:44:04 truenas k3s[16911]: E1222 17:44:04.562632   16911 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"adguard-home\" with CrashLoopBackOff: \"back-off 20s restarting failed container=adguard-home pod=adguard-home-57b89dd8b8-xzj2m_ix-adguard-home(015c0b2a-db07-4719-b416-3506b105713d)\"" pod="ix-adguard-home/adguard-home-57b89dd8b8-xzj2m" podUID=015c0b2a-db07-4719-b416-3506b105713d
Dec 22 17:44:05 truenas k3s[16911]: I1222 17:44:05.335237   16911 scope.go:115] "RemoveContainer" containerID="9cb280d1ef5ba704dc6e17b400b0e60cffd08b400b827f8bb256a86099e42a7c"
Dec 22 17:44:05 truenas k3s[16911]: E1222 17:44:05.335697   16911 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"adguard-home\" with CrashLoopBackOff: \"back-off 20s restarting failed container=adguard-home pod=adguard-home-57b89dd8b8-xzj2m_ix-adguard-home(015c0b2a-db07-4719-b416-3506b105713d)\"" pod="ix-adguard-home/adguard-home-57b89dd8b8-xzj2m" podUID=015c0b2a-db07-4719-b416-3506b105713d
Dec 22 17:44:16 truenas k3s[16911]: I1222 17:44:16.643728   16911 scope.go:115] "RemoveContainer" containerID="9cb280d1ef5ba704dc6e17b400b0e60cffd08b400b827f8bb256a86099e42a7c"
Dec 22 17:44:18 truenas k3s[16911]: I1222 17:44:18.382577   16911 scope.go:115] "RemoveContainer" containerID="9cb280d1ef5ba704dc6e17b400b0e60cffd08b400b827f8bb256a86099e42a7c"
Dec 22 17:44:18 truenas k3s[16911]: I1222 17:44:18.382943   16911 scope.go:115] "RemoveContainer" containerID="ec92d0d760fd7fcb39aed77460f1f9c7fb3cde08f7c4bd9f256b952de7ef6d94"
Dec 22 17:44:18 truenas k3s[16911]: E1222 17:44:18.383350   16911 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"adguard-home\" with CrashLoopBackOff: \"back-off 40s restarting failed container=adguard-home pod=adguard-home-57b89dd8b8-xzj2m_ix-adguard-home(015c0b2a-db07-4719-b416-3506b105713d)\"" pod="ix-adguard-home/adguard-home-57b89dd8b8-xzj2m" podUID=015c0b2a-db07-4719-b416-3506b105713d
Dec 22 17:44:19 truenas k3s[16911]: I1222 17:44:19.561941   16911 scope.go:115] "RemoveContainer" containerID="ec92d0d760fd7fcb39aed77460f1f9c7fb3cde08f7c4bd9f256b952de7ef6d94"
Dec 22 17:44:19 truenas k3s[16911]: E1222 17:44:19.562369   16911 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"adguard-home\" with CrashLoopBackOff: \"back-off 40s restarting failed container=adguard-home pod=adguard-home-57b89dd8b8-xzj2m_ix-adguard-home(015c0b2a-db07-4719-b416-3506b105713d)\"" pod="ix-adguard-home/adguard-home-57b89dd8b8-xzj2m" podUID=015c0b2a-db07-4719-b416-3506b105713d
Dec 22 17:44:20 truenas k3s[16911]: I1222 17:44:20.392976   16911 scope.go:115] "RemoveContainer" containerID="ec92d0d760fd7fcb39aed77460f1f9c7fb3cde08f7c4bd9f256b952de7ef6d94"
Dec 22 17:44:20 truenas k3s[16911]: E1222 17:44:20.393313   16911 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"adguard-home\" with CrashLoopBackOff: \"back-off 40s restarting failed container=adguard-home pod=adguard-home-57b89dd8b8-xzj2m_ix-adguard-home(015c0b2a-db07-4719-b416-3506b105713d)\"" pod="ix-adguard-home/adguard-home-57b89dd8b8-xzj2m" podUID=015c0b2a-db07-4719-b416-3506b105713d
Dec 22 17:44:23 truenas k3s[16911]: I1222 17:44:23.691087   16911 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="ix-nextcloud/nextcloud-imaginary-6fdbd88855-25hcq" podStartSLOduration=82.691053149 pod.CreationTimestamp="2023-12-22 17:43:01 +0100 CET" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2023-12-22 17:44:23.689480815 +0100 CET m=+121.476286606" watchObservedRunningTime="2023-12-22 17:44:23.691053149 +0100 CET m=+121.477858940"
Dec 22 17:44:24 truenas k3s[16911]: I1222 17:44:24.562047   16911 scope.go:115] "RemoveContainer" containerID="ec92d0d760fd7fcb39aed77460f1f9c7fb3cde08f7c4bd9f256b952de7ef6d94"
Dec 22 17:44:24 truenas k3s[16911]: E1222 17:44:24.562507   16911 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"adguard-home\" with CrashLoopBackOff: \"back-off 40s restarting failed container=adguard-home pod=adguard-home-57b89dd8b8-xzj2m_ix-adguard-home(015c0b2a-db07-4719-b416-3506b105713d)\"" pod="ix-adguard-home/adguard-home-57b89dd8b8-xzj2m" podUID=015c0b2a-db07-4719-b416-3506b105713d
Dec 22 17:44:25 truenas k3s[16911]: I1222 17:44:25.406561   16911 scope.go:115] "RemoveContainer" containerID="ec92d0d760fd7fcb39aed77460f1f9c7fb3cde08f7c4bd9f256b952de7ef6d94"
Dec 22 17:44:25 truenas k3s[16911]: E1222 17:44:25.406989   16911 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"adguard-home\" with CrashLoopBackOff: \"back-off 40s restarting failed container=adguard-home pod=adguard-home-57b89dd8b8-xzj2m_ix-adguard-home(015c0b2a-db07-4719-b416-3506b105713d)\"" pod="ix-adguard-home/adguard-home-57b89dd8b8-xzj2m" podUID=015c0b2a-db07-4719-b416-3506b105713d
Dec 22 17:44:26 truenas k3s[16911]: {"level":"warn","ts":"2023-12-22T17:44:26.289+0100","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ac81c0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 22 17:44:27 truenas k3s[16911]: time="2023-12-22T17:44:27+01:00" level=info msg="Slow SQL (started: 2023-12-22 17:44:25.856109499 +0100 CET m=+123.642915290) (total time: 1.875989761s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values(?, ?, ?, ?, ?, ?, ?, ?) : [[/registry/leases/kube-system/external-resizer-zfs-csi-openebs-io 0 0 779 9538889 0 [107 56 115 0 10 31 10 22 99 111 111 114 100 105 110 97 116 105 111 110 46 107 56 115 46 105 111 47 118 49 18 5 76 101 97 115 101 18 238 2 10 175 2 10 35 101 120 116 101 114 110 97 108 45 114 101 115 105 122 101 114 45 122 102 115 45 99 115 105 45 111 112 101 110 101 98 115 45 105 111 18 0 26 11 107 117 98 101 45 115 121 115 116 101 109 34 0 42 36 53 100 49 99 54 100 50 48 45 54 53 54 56 45 52 102 99 50 45 98 100 48 48 45 56 99 56 53 98 51 57 102 50 51 100 48 50 0 56 0 66 8 8 218 246 137 170 6 16 0 138 1 193 1 10 11 99 115 105 45 114 101 115 105 122 101 114 18 6 85 112 100 97 116 101 26 22 99 111 111 114 100 105 110 97 116 105 111 110 46 107 56 115 46 105 111 47 118 49 34 8 8 233 249 150 172 6 16 0 50 8 70 105 101 108 100 115 86 49 58 124 10 122 123 34 102 58 115 112 101 99 34 58 123 34 102 58 97 99 113 117 105 114 101 84 105 109 101 34 58 123 125 44 34 102 58 104 111 108 100 101 114 73 100 101 110 116 105 116 121 34 58 123 125 44 34 102 58 108 101 97 115 101 68 117 114 97 116 105 111 110 83 101 99 111 110 100 115 34 58 123 125 44 34 102 58 108 101 97 115 101 84 114 97 110 115 105 116 105 111 110 115 34 58 123 125 44 34 102 58 114 101 110 101 119 84 105 109 101 34 58 123 125 125 125 66 0 18 58 10 24 111 112 101 110 101 98 115 45 122 102 115 45 99 111 110 116 114 111 108 108 101 114 45 48 16 15 26 12 8 218 246 137 170 6 16 144 167 134 169 1 34 12 8 233 249 150 172 6 16 152 201 133 150 3 40 0 26 0 34 0] [107 56 115 0 10 31 10 22 99 111 111 114 100 105 110 97 116 105 111 110 46 107 56 115 46 105 111 47 118 49 18 5 76 101 97 115 101 18 238 2 10 175 2 10 35 101 120 116 101 114 110 97 108 45 114 101 115 105 122 101 114 45 122 102 115 45 99 115 105 45 111 112 101 110 101 98 115 45 105 111 18 0 26 11 107 117 98 101 45 115 121 115 116 101 109 34 0 42 36 53 100 49 99 54 100 50 48 45 54 53 54 56 45 52 102 99 50 45 98 100 48 48 45 56 99 56 53 98 51 57 102 50 51 100 48 50 0 56 0 66 8 8 218 246 137 170 6 16 0 138 1 193 1 10 11 99 115 105 45 114 101 115 105 122 101 114 18 6 85 112 100 97 116 101 26 22 99 111 111 114 100 105 110 97 116 105 111 110 46 107 56 115 46 105 111 47 118 49 34 8 8 228 249 150 172 6 16 0 50 8 70 105 101 108 100 115 86 49 58 124 10 122 123 34 102 58 115 112 101 99 34 58 123 34 102 58 97 99 113 117 105 114 101 84 105 109 101 34 58 123 125 44 34 102 58 104 111 108 100 101 114 73 100 101 110 116 105 116 121 34 58 123 125 44 34 102 58 108 101 97 115 101 68 117 114 97 116 105 111 110 83 101 99 111 110 100 115 34 58 123 125 44 34 102 58 108 101 97 115 101 84 114 97 110 115 105 116 105 111 110 115 34 58 123 125 44 34 102 58 114 101 110 101 119 84 105 109 101 34 58 123 125 125 125 66 0 18 58 10 24 111 112 101 110 101 98 115 45 122 102 115 45 99 111 110 116 114 111 108 108 101 114 45 48 16 15 26 12 8 218 246 137 170 6 16 144 167 134 169 1 34 12 8 228 249 150 172 6 16 128 174 237 147 3 40 0 26 0 34 0]]]"
Dec 22 17:44:27 truenas k3s[16911]: I1222 17:44:27.732507   16911 trace.go:219] Trace[209520218]: "Update" accept:application/json, */*,audit-id:1a9d4483-b624-4d12-9afb-d62cabbbf4e7,client:172.16.3.77,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/external-resizer-zfs-csi-openebs-io,user-agent:csi-resizer/v0.0.0 (linux/amd64) kubernetes/$Format,verb:PUT (22-Dec-2023 17:44:25.854) (total time: 1877ms):
Dec 22 17:44:27 truenas k3s[16911]: Trace[209520218]: ["GuaranteedUpdate etcd3" audit-id:1a9d4483-b624-4d12-9afb-d62cabbbf4e7,key:/leases/kube-system/external-resizer-zfs-csi-openebs-io,type:*coordination.Lease,resource:leases.coordination.k8s.io 1877ms (17:44:25.854)
Dec 22 17:44:27 truenas k3s[16911]: Trace[209520218]:  ---"Txn call completed" 1877ms (17:44:27.732)]
Dec 22 17:44:27 truenas k3s[16911]: Trace[209520218]: [1.877978918s] [1.877978918s] END
Dec 22 17:44:28 truenas k3s[16911]: I1222 17:44:28.434172   16911 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="ix-nextcloud/nextcloud-c9c757d4f-cwjcs" podStartSLOduration=94.434060403 pod.CreationTimestamp="2023-12-22 17:42:54 +0100 CET" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2023-12-22 17:44:28.432850109 +0100 CET m=+126.219655900" watchObservedRunningTime="2023-12-22 17:44:28.434060403 +0100 CET m=+126.220866194"
Dec 22 17:44:37 truenas k3s[16911]: I1222 17:44:37.643761   16911 scope.go:115] "RemoveContainer" containerID="ec92d0d760fd7fcb39aed77460f1f9c7fb3cde08f7c4bd9f256b952de7ef6d94"
Dec 22 17:44:37 truenas k3s[16911]: E1222 17:44:37.644272   16911 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"adguard-home\" with CrashLoopBackOff: \"back-off 40s restarting failed container=adguard-home pod=adguard-home-57b89dd8b8-xzj2m_ix-adguard-home(015c0b2a-db07-4719-b416-3506b105713d)\"" pod="ix-adguard-home/adguard-home-57b89dd8b8-xzj2m" podUID=015c0b2a-db07-4719-b416-3506b105713d
Dec 22 17:44:52 truenas k3s[16911]: I1222 17:44:52.643407   16911 scope.go:115] "RemoveContainer" containerID="ec92d0d760fd7fcb39aed77460f1f9c7fb3cde08f7c4bd9f256b952de7ef6d94"
Dec 22 17:44:52 truenas k3s[16911]: E1222 17:44:52.643863   16911 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"adguard-home\" with CrashLoopBackOff: \"back-off 40s restarting failed container=adguard-home pod=adguard-home-57b89dd8b8-xzj2m_ix-adguard-home(015c0b2a-db07-4719-b416-3506b105713d)\"" pod="ix-adguard-home/adguard-home-57b89dd8b8-xzj2m" podUID=015c0b2a-db07-4719-b416-3506b105713d
Dec 22 17:45:00 truenas k3s[16911]: I1222 17:45:00.106828   16911 event.go:294] "Event occurred" object="ix-nextcloud/nextcloud-nextcloud-cron" fieldPath="" kind="CronJob" apiVersion="batch/v1" type="Normal" reason="SuccessfulCreate" message="Created job nextcloud-nextcloud-cron-28387725"
Dec 22 17:45:00 truenas k3s[16911]: I1222 17:45:00.106865   16911 job_controller.go:514] enqueueing job ix-nextcloud/nextcloud-nextcloud-cron-28387725
Dec 22 17:45:00 truenas k3s[16911]: I1222 17:45:00.111834   16911 event.go:294] "Event occurred" object="ix-nextcloud/nextcloud-nextcloud-cron-28387725" fieldPath="" kind="Job" apiVersion="batch/v1" type="Normal" reason="SuccessfulCreate" message="Created pod: nextcloud-nextcloud-cron-28387725-cmpmw"
Dec 22 17:45:00 truenas k3s[16911]: I1222 17:45:00.112145   16911 job_controller.go:514] enqueueing job ix-nextcloud/nextcloud-nextcloud-cron-28387725
Dec 22 17:45:00 truenas k3s[16911]: I1222 17:45:00.115014   16911 job_controller.go:514] enqueueing job ix-nextcloud/nextcloud-nextcloud-cron-28387725
Dec 22 17:45:00 truenas k3s[16911]: I1222 17:45:00.115775   16911 job_controller.go:514] enqueueing job ix-nextcloud/nextcloud-nextcloud-cron-28387725
Dec 22 17:45:00 truenas k3s[16911]: I1222 17:45:00.116551   16911 topology_manager.go:210] "Topology Admit Handler"
Dec 22 17:45:00 truenas k3s[16911]: I1222 17:45:00.125048   16911 job_controller.go:514] enqueueing job ix-nextcloud/nextcloud-nextcloud-cron-28387725
Dec 22 17:45:00 truenas k3s[16911]: I1222 17:45:00.171768   16911 reconciler_common.go:253] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/host-path/a21c0609-a098-473e-9b2b-5446745bc1fd-data\") pod \"nextcloud-nextcloud-cron-28387725-cmpmw\" (UID: \"a21c0609-a098-473e-9b2b-5446745bc1fd\") " pod="ix-nextcloud/nextcloud-nextcloud-cron-28387725-cmpmw"
Dec 22 17:45:00 truenas k3s[16911]: I1222 17:45:00.171828   16911 reconciler_common.go:253] "operationExecutor.VerifyControllerAttachedVolume started for volume \"varrun\" (UniqueName: \"kubernetes.io/empty-dir/a21c0609-a098-473e-9b2b-5446745bc1fd-varrun\") pod \"nextcloud-nextcloud-cron-28387725-cmpmw\" (UID: \"a21c0609-a098-473e-9b2b-5446745bc1fd\") " pod="ix-nextcloud/nextcloud-nextcloud-cron-28387725-cmpmw"
Dec 22 17:45:00 truenas k3s[16911]: I1222 17:45:00.171941   16911 reconciler_common.go:253] "operationExecutor.VerifyControllerAttachedVolume started for volume \"shared\" (UniqueName: \"kubernetes.io/empty-dir/a21c0609-a098-473e-9b2b-5446745bc1fd-shared\") pod \"nextcloud-nextcloud-cron-28387725-cmpmw\" (UID: \"a21c0609-a098-473e-9b2b-5446745bc1fd\") " pod="ix-nextcloud/nextcloud-nextcloud-cron-28387725-cmpmw"
Dec 22 17:45:00 truenas k3s[16911]: I1222 17:45:00.172061   16911 reconciler_common.go:253] "operationExecutor.VerifyControllerAttachedVolume started for volume \"devshm\" (UniqueName: \"kubernetes.io/empty-dir/a21c0609-a098-473e-9b2b-5446745bc1fd-devshm\") pod \"nextcloud-nextcloud-cron-28387725-cmpmw\" (UID: \"a21c0609-a098-473e-9b2b-5446745bc1fd\") " pod="ix-nextcloud/nextcloud-nextcloud-cron-28387725-cmpmw"
Dec 22 17:45:00 truenas k3s[16911]: I1222 17:45:00.172110   16911 reconciler_common.go:253] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a21c0609-a098-473e-9b2b-5446745bc1fd-tmp\") pod \"nextcloud-nextcloud-cron-28387725-cmpmw\" (UID: \"a21c0609-a098-473e-9b2b-5446745bc1fd\") " pod="ix-nextcloud/nextcloud-nextcloud-cron-28387725-cmpmw"
Dec 22 17:45:00 truenas k3s[16911]: I1222 17:45:00.172156   16911 reconciler_common.go:253] "operationExecutor.VerifyControllerAttachedVolume started for volume \"varlogs\" (UniqueName: \"kubernetes.io/empty-dir/a21c0609-a098-473e-9b2b-5446745bc1fd-varlogs\") pod \"nextcloud-nextcloud-cron-28387725-cmpmw\" (UID: \"a21c0609-a098-473e-9b2b-5446745bc1fd\") " pod="ix-nextcloud/nextcloud-nextcloud-cron-28387725-cmpmw"
Dec 22 17:45:00 truenas k3s[16911]: I1222 17:45:00.574094   16911 job_controller.go:514] enqueueing job ix-nextcloud/nextcloud-nextcloud-cron-28387725
Dec 22 17:45:02 truenas k3s[16911]: I1222 17:45:02.506173   16911 job_controller.go:514] enqueueing job ix-nextcloud/nextcloud-nextcloud-cron-28387725
Dec 22 17:45:02 truenas k3s[16911]: I1222 17:45:02.507717   16911 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="ix-nextcloud/nextcloud-nextcloud-cron-28387725-cmpmw" podStartSLOduration=2.507692828 pod.CreationTimestamp="2023-12-22 17:45:00 +0100 CET" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2023-12-22 17:45:02.505706536 +0100 CET m=+160.292512337" watchObservedRunningTime="2023-12-22 17:45:02.507692828 +0100 CET m=+160.294498629"
Dec 22 17:45:03 truenas k3s[16911]: {"level":"warn","ts":"2023-12-22T17:45:03.009+0100","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ac81c0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 22 17:45:03 truenas k3s[16911]: I1222 17:45:03.510836   16911 job_controller.go:514] enqueueing job ix-nextcloud/nextcloud-nextcloud-cron-28387725
Dec 22 17:45:05 truenas k3s[16911]: I1222 17:45:05.524722   16911 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="ix-nextcloud/nextcloud-nginx-b674d44dd-vwfct" podStartSLOduration=5090.524683924 pod.CreationTimestamp="2023-12-22 16:20:15 +0100 CET" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2023-12-22 17:45:05.522943674 +0100 CET m=+163.309749475" watchObservedRunningTime="2023-12-22 17:45:05.524683924 +0100 CET m=+163.311489715"
Dec 22 17:45:05 truenas k3s[16911]: I1222 17:45:05.644521   16911 scope.go:115] "RemoveContainer" containerID="ec92d0d760fd7fcb39aed77460f1f9c7fb3cde08f7c4bd9f256b952de7ef6d94"
Dec 22 17:45:06 truenas k3s[16911]: I1222 17:45:06.512898   16911 scope.go:115] "RemoveContainer" containerID="ec92d0d760fd7fcb39aed77460f1f9c7fb3cde08f7c4bd9f256b952de7ef6d94"
Dec 22 17:45:06 truenas k3s[16911]: I1222 17:45:06.513257   16911 scope.go:115] "RemoveContainer" containerID="693b7a00d6b74be2c0256e5d7fc76b415faaedc7ee348f68db8e995db26873dd"
Dec 22 17:45:06 truenas k3s[16911]: E1222 17:45:06.513613   16911 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"adguard-home\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=adguard-home pod=adguard-home-57b89dd8b8-xzj2m_ix-adguard-home(015c0b2a-db07-4719-b416-3506b105713d)\"" pod="ix-adguard-home/adguard-home-57b89dd8b8-xzj2m" podUID=015c0b2a-db07-4719-b416-3506b105713d
Dec 22 17:45:06 truenas k3s[16911]: I1222 17:45:06.526927   16911 job_controller.go:514] enqueueing job ix-nextcloud/nextcloud-nextcloud-cron-28387725
Dec 22 17:45:07 truenas k3s[16911]: I1222 17:45:07.531546   16911 job_controller.go:514] enqueueing job ix-nextcloud/nextcloud-nextcloud-cron-28387725
Dec 22 17:45:09 truenas k3s[16911]: I1222 17:45:09.562811   16911 scope.go:115] "RemoveContainer" containerID="693b7a00d6b74be2c0256e5d7fc76b415faaedc7ee348f68db8e995db26873dd"
Dec 22 17:45:09 truenas k3s[16911]: E1222 17:45:09.563307   16911 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"adguard-home\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=adguard-home pod=adguard-home-57b89dd8b8-xzj2m_ix-adguard-home(015c0b2a-db07-4719-b416-3506b105713d)\"" pod="ix-adguard-home/adguard-home-57b89dd8b8-xzj2m" podUID=015c0b2a-db07-4719-b416-3506b105713d
Dec 22 17:45:10 truenas k3s[16911]: I1222 17:45:10.523996   16911 scope.go:115] "RemoveContainer" containerID="693b7a00d6b74be2c0256e5d7fc76b415faaedc7ee348f68db8e995db26873dd"
Dec 22 17:45:10 truenas k3s[16911]: E1222 17:45:10.524411   16911 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"adguard-home\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=adguard-home pod=adguard-home-57b89dd8b8-xzj2m_ix-adguard-home(015c0b2a-db07-4719-b416-3506b105713d)\"" pod="ix-adguard-home/adguard-home-57b89dd8b8-xzj2m" podUID=015c0b2a-db07-4719-b416-3506b105713d
