
-------------
1) K3s logs
-------------

Dec 23 21:59:59 freenas k3s[16024]: time="2023-12-23T21:59:59+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:00:04 freenas k3s[16024]: time="2023-12-23T22:00:04+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 22:00:33 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:00:33.790+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:01:14 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:01:14.398+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:01:56 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:01:56.594+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:01:57 freenas k3s[16024]: I1223 22:01:57.381051   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 22:01:57 freenas k3s[16024]: E1223 22:01:57.385769   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 22:03:59.385739208 +0200 EET m=+110479.778349420 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 22:01:57 freenas k3s[16024]: I1223 22:01:57.482839   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 22:01:57 freenas k3s[16024]: E1223 22:01:57.487898   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 22:03:59.48785697 +0200 EET m=+110479.880467182 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 22:01:57 freenas k3s[16024]: I1223 22:01:57.684844   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 22:01:57 freenas k3s[16024]: E1223 22:01:57.688931   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 22:03:59.688905042 +0200 EET m=+110480.081515254 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 22:02:14 freenas k3s[16024]: time="2023-12-23T22:02:14+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:02:14 freenas k3s[16024]: time="2023-12-23T22:02:14+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:02:19 freenas k3s[16024]: time="2023-12-23T22:02:19+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 22:02:39 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:02:39.772+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:02:40 freenas k3s[16024]: time="2023-12-23T22:02:40+02:00" level=info msg="COMPACT compactRev=67562547 targetCompactRev=67563250 currentRev=67564250"
Dec 23 22:02:40 freenas k3s[16024]: time="2023-12-23T22:02:40+02:00" level=info msg="COMPACT deleted 703 rows from 703 revisions in 27.292142ms - compacted to 67563250/67564250"
Dec 23 22:03:14 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:03:14.569+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:03:45 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:03:45.926+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:03:59 freenas k3s[16024]: I1223 22:03:59.403867   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 22:03:59 freenas k3s[16024]: E1223 22:03:59.407739   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 22:06:01.407717893 +0200 EET m=+110601.800328065 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 22:03:59 freenas k3s[16024]: I1223 22:03:59.505331   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 22:03:59 freenas k3s[16024]: E1223 22:03:59.508417   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 22:06:01.508396011 +0200 EET m=+110601.901006193 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 22:03:59 freenas k3s[16024]: I1223 22:03:59.708442   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 22:03:59 freenas k3s[16024]: E1223 22:03:59.712812   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 22:06:01.712786032 +0200 EET m=+110602.105396244 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 22:04:18 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:04:18.772+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:04:29 freenas k3s[16024]: time="2023-12-23T22:04:29+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:04:29 freenas k3s[16024]: time="2023-12-23T22:04:29+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:04:34 freenas k3s[16024]: time="2023-12-23T22:04:34+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 22:04:50 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:04:50.602+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:05:35 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:05:35.138+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:06:01 freenas k3s[16024]: I1223 22:06:01.450245   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 22:06:01 freenas k3s[16024]: E1223 22:06:01.454715   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 22:08:03.454688991 +0200 EET m=+110723.847299273 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 22:06:01 freenas k3s[16024]: I1223 22:06:01.551377   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 22:06:01 freenas k3s[16024]: E1223 22:06:01.555437   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 22:08:03.555414049 +0200 EET m=+110723.948024241 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 22:06:01 freenas k3s[16024]: I1223 22:06:01.754254   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 22:06:01 freenas k3s[16024]: E1223 22:06:01.758452   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 22:08:03.758415823 +0200 EET m=+110724.151026035 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 22:06:06 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:06:06.504+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:06:40 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:06:40.271+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:06:44 freenas k3s[16024]: time="2023-12-23T22:06:44+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:06:44 freenas k3s[16024]: time="2023-12-23T22:06:44+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:06:49 freenas k3s[16024]: time="2023-12-23T22:06:49+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 22:07:12 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:07:12.279+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:07:40 freenas k3s[16024]: time="2023-12-23T22:07:40+02:00" level=info msg="COMPACT compactRev=67563250 targetCompactRev=67563954 currentRev=67564954"
Dec 23 22:07:40 freenas k3s[16024]: time="2023-12-23T22:07:40+02:00" level=info msg="COMPACT deleted 704 rows from 704 revisions in 24.494798ms - compacted to 67563954/67564954"
Dec 23 22:07:51 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:07:51.239+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:08:03 freenas k3s[16024]: I1223 22:08:03.481129   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 22:08:03 freenas k3s[16024]: E1223 22:08:03.485490   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 22:10:05.485459316 +0200 EET m=+110845.878069518 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 22:08:03 freenas k3s[16024]: I1223 22:08:03.582072   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 22:08:03 freenas k3s[16024]: E1223 22:08:03.586321   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 22:10:05.586300054 +0200 EET m=+110845.978910226 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 22:08:03 freenas k3s[16024]: I1223 22:08:03.784868   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 22:08:03 freenas k3s[16024]: E1223 22:08:03.789142   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 22:10:05.789107487 +0200 EET m=+110846.181717699 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 22:08:28 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:08:28.369+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:09:00 freenas k3s[16024]: time="2023-12-23T22:09:00+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:09:00 freenas k3s[16024]: time="2023-12-23T22:09:00+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:09:05 freenas k3s[16024]: time="2023-12-23T22:09:05+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 22:09:11 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:09:11.157+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:09:51 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:09:51.318+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:10:05 freenas k3s[16024]: I1223 22:10:05.521270   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 22:10:05 freenas k3s[16024]: E1223 22:10:05.525366   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 22:12:07.52534251 +0200 EET m=+110967.917952792 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 22:10:05 freenas k3s[16024]: I1223 22:10:05.621851   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 22:10:05 freenas k3s[16024]: E1223 22:10:05.626021   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 22:12:07.625986492 +0200 EET m=+110968.018596704 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 22:10:05 freenas k3s[16024]: I1223 22:10:05.824177   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 22:10:05 freenas k3s[16024]: E1223 22:10:05.829105   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 22:12:07.829079982 +0200 EET m=+110968.221690194 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 22:10:36 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:10:36.203+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:11:14 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:11:14.091+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:11:15 freenas k3s[16024]: time="2023-12-23T22:11:15+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:11:15 freenas k3s[16024]: time="2023-12-23T22:11:15+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:11:20 freenas k3s[16024]: time="2023-12-23T22:11:20+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 22:11:47 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:11:47.477+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:12:07 freenas k3s[16024]: I1223 22:12:07.552992   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 22:12:07 freenas k3s[16024]: E1223 22:12:07.557786   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 22:14:09.557753785 +0200 EET m=+111089.950364057 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 22:12:07 freenas k3s[16024]: I1223 22:12:07.654096   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 22:12:07 freenas k3s[16024]: E1223 22:12:07.658629   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 22:14:09.658605575 +0200 EET m=+111090.051215777 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 22:12:07 freenas k3s[16024]: I1223 22:12:07.855773   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 22:12:07 freenas k3s[16024]: E1223 22:12:07.860358   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 22:14:09.860329694 +0200 EET m=+111090.252939876 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 22:12:31 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:12:31.958+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:12:40 freenas k3s[16024]: time="2023-12-23T22:12:40+02:00" level=info msg="COMPACT compactRev=67563954 targetCompactRev=67564657 currentRev=67565657"
Dec 23 22:12:40 freenas k3s[16024]: time="2023-12-23T22:12:40+02:00" level=info msg="COMPACT deleted 703 rows from 703 revisions in 28.177556ms - compacted to 67564657/67565657"
Dec 23 22:13:13 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:13:13.188+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:13:30 freenas k3s[16024]: time="2023-12-23T22:13:30+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:13:30 freenas k3s[16024]: time="2023-12-23T22:13:30+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:13:35 freenas k3s[16024]: time="2023-12-23T22:13:35+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 22:13:55 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:13:55.153+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:14:09 freenas k3s[16024]: I1223 22:14:09.600049   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 22:14:09 freenas k3s[16024]: E1223 22:14:09.605052   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 22:16:11.605020059 +0200 EET m=+111211.997630261 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 22:14:09 freenas k3s[16024]: I1223 22:14:09.701353   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 22:14:09 freenas k3s[16024]: E1223 22:14:09.705811   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 22:16:11.705780463 +0200 EET m=+111212.098390675 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 22:14:09 freenas k3s[16024]: I1223 22:14:09.904218   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 22:14:09 freenas k3s[16024]: E1223 22:14:09.909045   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 22:16:11.909019211 +0200 EET m=+111212.301629423 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 22:14:36 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:14:36.359+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:15:11 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:15:11.049+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:15:45 freenas k3s[16024]: time="2023-12-23T22:15:45+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:15:45 freenas k3s[16024]: time="2023-12-23T22:15:45+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:15:49 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:15:49.437+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:15:50 freenas k3s[16024]: time="2023-12-23T22:15:50+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 22:16:11 freenas k3s[16024]: I1223 22:16:11.634194   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 22:16:11 freenas k3s[16024]: E1223 22:16:11.637781   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 22:18:13.63775435 +0200 EET m=+111334.030364563 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 22:16:11 freenas k3s[16024]: I1223 22:16:11.734804   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 22:16:11 freenas k3s[16024]: E1223 22:16:11.739714   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 22:18:13.739681048 +0200 EET m=+111334.132291250 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 22:16:11 freenas k3s[16024]: I1223 22:16:11.936763   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 22:16:11 freenas k3s[16024]: E1223 22:16:11.942042   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 22:18:13.942012209 +0200 EET m=+111334.334622421 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 22:16:28 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:16:28.780+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:17:00 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:17:00.947+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:17:38 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:17:38.650+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:17:40 freenas k3s[16024]: time="2023-12-23T22:17:40+02:00" level=info msg="COMPACT compactRev=67564657 targetCompactRev=67565359 currentRev=67566359"
Dec 23 22:17:40 freenas k3s[16024]: time="2023-12-23T22:17:40+02:00" level=info msg="COMPACT deleted 702 rows from 702 revisions in 19.190817ms - compacted to 67565359/67566359"
Dec 23 22:18:00 freenas k3s[16024]: time="2023-12-23T22:18:00+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:18:00 freenas k3s[16024]: time="2023-12-23T22:18:00+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:18:05 freenas k3s[16024]: time="2023-12-23T22:18:05+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 22:18:13 freenas k3s[16024]: I1223 22:18:13.734756   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 22:18:13 freenas k3s[16024]: E1223 22:18:13.738995   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 22:20:15.738966265 +0200 EET m=+111456.131576467 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 22:18:13 freenas k3s[16024]: I1223 22:18:13.835929   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 22:18:13 freenas k3s[16024]: E1223 22:18:13.839575   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 22:20:15.839551205 +0200 EET m=+111456.232161407 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 22:18:14 freenas k3s[16024]: I1223 22:18:14.038582   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 22:18:14 freenas k3s[16024]: E1223 22:18:14.042486   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 22:20:16.042469298 +0200 EET m=+111456.435079490 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 22:18:16 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:18:16.196+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:18:50 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:18:50.738+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:19:33 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:19:33.099+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:20:03 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:20:03.330+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:20:15 freenas k3s[16024]: I1223 22:20:15.744633   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 22:20:15 freenas k3s[16024]: E1223 22:20:15.749463   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 22:22:17.74942873 +0200 EET m=+111578.142038942 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 22:20:15 freenas k3s[16024]: I1223 22:20:15.845394   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 22:20:15 freenas k3s[16024]: E1223 22:20:15.849987   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 22:22:17.849954409 +0200 EET m=+111578.242564611 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 22:20:15 freenas k3s[16024]: time="2023-12-23T22:20:15+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:20:15 freenas k3s[16024]: time="2023-12-23T22:20:15+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:20:16 freenas k3s[16024]: I1223 22:20:16.047839   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 22:20:16 freenas k3s[16024]: E1223 22:20:16.052549   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 22:22:18.052523892 +0200 EET m=+111578.445134104 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 22:20:20 freenas k3s[16024]: time="2023-12-23T22:20:20+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 22:20:45 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:20:45.138+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:21:27 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:21:27.136+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:22:05 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:22:05.231+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:22:17 freenas k3s[16024]: I1223 22:22:17.774217   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 22:22:17 freenas k3s[16024]: E1223 22:22:17.778601   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 22:24:19.778565698 +0200 EET m=+111700.171175900 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 22:22:17 freenas k3s[16024]: I1223 22:22:17.875456   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 22:22:17 freenas k3s[16024]: E1223 22:22:17.879866   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 22:24:19.879833697 +0200 EET m=+111700.272443909 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 22:22:18 freenas k3s[16024]: I1223 22:22:18.077570   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 22:22:18 freenas k3s[16024]: E1223 22:22:18.082314   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 22:24:20.082276728 +0200 EET m=+111700.474886940 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 22:22:31 freenas k3s[16024]: time="2023-12-23T22:22:31+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:22:31 freenas k3s[16024]: time="2023-12-23T22:22:31+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:22:36 freenas k3s[16024]: time="2023-12-23T22:22:36+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 22:22:40 freenas k3s[16024]: time="2023-12-23T22:22:40+02:00" level=info msg="COMPACT compactRev=67565359 targetCompactRev=67566065 currentRev=67567065"
Dec 23 22:22:40 freenas k3s[16024]: time="2023-12-23T22:22:40+02:00" level=info msg="COMPACT deleted 706 rows from 706 revisions in 30.723012ms - compacted to 67566065/67567065"
Dec 23 22:22:44 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:22:44.490+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:23:19 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:23:19.012+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:23:50 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:23:50.020+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:24:19 freenas k3s[16024]: I1223 22:24:19.829087   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 22:24:19 freenas k3s[16024]: E1223 22:24:19.833075   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 22:26:21.833055602 +0200 EET m=+111822.225665784 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 22:24:19 freenas k3s[16024]: I1223 22:24:19.930229   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 22:24:19 freenas k3s[16024]: E1223 22:24:19.934358   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 22:26:21.93433869 +0200 EET m=+111822.326948862 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 22:24:20 freenas k3s[16024]: I1223 22:24:20.132345   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 22:24:20 freenas k3s[16024]: E1223 22:24:20.136943   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 22:26:22.136911119 +0200 EET m=+111822.529521331 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 22:24:27 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:24:27.649+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:24:46 freenas k3s[16024]: time="2023-12-23T22:24:46+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:24:46 freenas k3s[16024]: time="2023-12-23T22:24:46+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:24:51 freenas k3s[16024]: time="2023-12-23T22:24:51+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 22:25:10 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:25:10.755+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:25:50 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:25:50.317+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:26:21 freenas k3s[16024]: I1223 22:26:21.855203   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 22:26:21 freenas k3s[16024]: E1223 22:26:21.859977   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 22:28:23.859946822 +0200 EET m=+111944.252557024 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 22:26:21 freenas k3s[16024]: I1223 22:26:21.956272   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 22:26:21 freenas k3s[16024]: E1223 22:26:21.960243   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 22:28:23.960219367 +0200 EET m=+111944.352829569 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 22:26:22 freenas k3s[16024]: I1223 22:26:22.158606   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 22:26:22 freenas k3s[16024]: E1223 22:26:22.162875   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 22:28:24.162847302 +0200 EET m=+111944.555457514 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 22:26:25 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:26:25.661+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:27:01 freenas k3s[16024]: time="2023-12-23T22:27:01+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:27:01 freenas k3s[16024]: time="2023-12-23T22:27:01+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:27:05 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:27:05.321+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:27:06 freenas k3s[16024]: time="2023-12-23T22:27:06+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 22:27:37 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:27:37.498+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:27:40 freenas k3s[16024]: time="2023-12-23T22:27:40+02:00" level=info msg="COMPACT compactRev=67566065 targetCompactRev=67566768 currentRev=67567768"
Dec 23 22:27:40 freenas k3s[16024]: time="2023-12-23T22:27:40+02:00" level=info msg="COMPACT deleted 703 rows from 703 revisions in 11.698537ms - compacted to 67566768/67567768"
Dec 23 22:28:18 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:28:18.497+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:28:23 freenas k3s[16024]: I1223 22:28:23.889600   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 22:28:23 freenas k3s[16024]: E1223 22:28:23.894202   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 22:30:25.894172118 +0200 EET m=+112066.286782330 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 22:28:23 freenas k3s[16024]: I1223 22:28:23.990625   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 22:28:23 freenas k3s[16024]: E1223 22:28:23.994535   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 22:30:25.994504026 +0200 EET m=+112066.387114238 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 22:28:24 freenas k3s[16024]: I1223 22:28:24.193166   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 22:28:24 freenas k3s[16024]: E1223 22:28:24.196456   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 22:30:26.19642614 +0200 EET m=+112066.589036352 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 22:28:56 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:28:56.198+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:29:16 freenas k3s[16024]: time="2023-12-23T22:29:16+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:29:16 freenas k3s[16024]: time="2023-12-23T22:29:16+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:29:21 freenas k3s[16024]: time="2023-12-23T22:29:21+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 22:29:39 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:29:39.576+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:30:11 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:30:11.572+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:30:25 freenas k3s[16024]: I1223 22:30:25.935881   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 22:30:25 freenas k3s[16024]: E1223 22:30:25.940767   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 22:32:27.940737658 +0200 EET m=+112188.333347870 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 22:30:26 freenas k3s[16024]: I1223 22:30:26.036490   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 22:30:26 freenas k3s[16024]: E1223 22:30:26.040917   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 22:32:28.040885374 +0200 EET m=+112188.433495576 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 22:30:26 freenas k3s[16024]: I1223 22:30:26.238128   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 22:30:26 freenas k3s[16024]: E1223 22:30:26.242280   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 22:32:28.242246845 +0200 EET m=+112188.634857047 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 22:30:46 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:30:46.055+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:31:30 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:31:30.567+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:31:31 freenas k3s[16024]: time="2023-12-23T22:31:31+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:31:31 freenas k3s[16024]: time="2023-12-23T22:31:31+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:31:36 freenas k3s[16024]: time="2023-12-23T22:31:36+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 22:32:13 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:32:13.594+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:32:27 freenas k3s[16024]: I1223 22:32:27.972479   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 22:32:27 freenas k3s[16024]: E1223 22:32:27.977512   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 22:34:29.97747995 +0200 EET m=+112310.370090162 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 22:32:28 freenas k3s[16024]: I1223 22:32:28.073269   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 22:32:28 freenas k3s[16024]: E1223 22:32:28.077332   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 22:34:30.077303194 +0200 EET m=+112310.469913396 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 22:32:28 freenas k3s[16024]: I1223 22:32:28.275270   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 22:32:28 freenas k3s[16024]: E1223 22:32:28.279278   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 22:34:30.279246018 +0200 EET m=+112310.671856230 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 22:32:40 freenas k3s[16024]: time="2023-12-23T22:32:40+02:00" level=info msg="COMPACT compactRev=67566768 targetCompactRev=67567476 currentRev=67568476"
Dec 23 22:32:40 freenas k3s[16024]: time="2023-12-23T22:32:40+02:00" level=info msg="COMPACT deleted 708 rows from 708 revisions in 30.137953ms - compacted to 67567476/67568476"
Dec 23 22:32:44 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:32:44.208+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:33:24 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:33:24.405+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:33:47 freenas k3s[16024]: time="2023-12-23T22:33:47+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:33:47 freenas k3s[16024]: time="2023-12-23T22:33:47+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:33:52 freenas k3s[16024]: time="2023-12-23T22:33:52+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 22:33:56 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:33:56.089+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:34:30 freenas k3s[16024]: I1223 22:34:30.004582   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 22:34:30 freenas k3s[16024]: E1223 22:34:30.009289   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 22:36:32.009254795 +0200 EET m=+112432.401865007 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 22:34:30 freenas k3s[16024]: I1223 22:34:30.105773   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 22:34:30 freenas k3s[16024]: E1223 22:34:30.110385   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 22:36:32.110358421 +0200 EET m=+112432.502968633 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 22:34:30 freenas k3s[16024]: I1223 22:34:30.309685   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 22:34:30 freenas k3s[16024]: E1223 22:34:30.314587   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 22:36:32.314557281 +0200 EET m=+112432.707167484 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 22:34:33 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:34:33.777+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:35:07 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:35:07.732+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:35:43 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:35:43.349+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:36:02 freenas k3s[16024]: time="2023-12-23T22:36:02+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:36:02 freenas k3s[16024]: time="2023-12-23T22:36:02+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:36:07 freenas k3s[16024]: time="2023-12-23T22:36:07+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 22:36:21 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:36:21.694+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:36:32 freenas k3s[16024]: I1223 22:36:32.079720   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 22:36:32 freenas k3s[16024]: E1223 22:36:32.084628   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 22:38:34.084581864 +0200 EET m=+112554.477192076 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 22:36:32 freenas k3s[16024]: I1223 22:36:32.181462   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 22:36:32 freenas k3s[16024]: E1223 22:36:32.185524   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 22:38:34.185499273 +0200 EET m=+112554.578109535 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 22:36:32 freenas k3s[16024]: I1223 22:36:32.382751   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 22:36:32 freenas k3s[16024]: E1223 22:36:32.387368   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 22:38:34.387350461 +0200 EET m=+112554.779960643 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 22:36:54 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:36:54.328+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:37:30 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:37:30.344+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:37:40 freenas k3s[16024]: time="2023-12-23T22:37:40+02:00" level=info msg="COMPACT compactRev=67567476 targetCompactRev=67568180 currentRev=67569180"
Dec 23 22:37:40 freenas k3s[16024]: time="2023-12-23T22:37:40+02:00" level=info msg="COMPACT deleted 704 rows from 704 revisions in 18.178761ms - compacted to 67568180/67569180"
Dec 23 22:38:00 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:38:00.803+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:38:17 freenas k3s[16024]: time="2023-12-23T22:38:17+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:38:17 freenas k3s[16024]: time="2023-12-23T22:38:17+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:38:22 freenas k3s[16024]: time="2023-12-23T22:38:22+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 22:38:34 freenas k3s[16024]: I1223 22:38:34.143944   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 22:38:34 freenas k3s[16024]: E1223 22:38:34.148143   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 22:40:36.148116794 +0200 EET m=+112676.540726996 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 22:38:34 freenas k3s[16024]: I1223 22:38:34.244548   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 22:38:34 freenas k3s[16024]: E1223 22:38:34.248753   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 22:40:36.248722052 +0200 EET m=+112676.641332264 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 22:38:34 freenas k3s[16024]: I1223 22:38:34.446101   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 22:38:34 freenas k3s[16024]: E1223 22:38:34.450508   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 22:40:36.4504829 +0200 EET m=+112676.843093102 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 22:38:41 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:38:41.298+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:39:25 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:39:25.102+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:40:03 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:40:03.520+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:40:32 freenas k3s[16024]: time="2023-12-23T22:40:32+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:40:32 freenas k3s[16024]: time="2023-12-23T22:40:32+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:40:36 freenas k3s[16024]: I1223 22:40:36.152915   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 22:40:36 freenas k3s[16024]: E1223 22:40:36.157103   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 22:42:38.157079489 +0200 EET m=+112798.549689691 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 22:40:36 freenas k3s[16024]: I1223 22:40:36.254378   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 22:40:36 freenas k3s[16024]: E1223 22:40:36.258162   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 22:42:38.258139627 +0200 EET m=+112798.650749830 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 22:40:36 freenas k3s[16024]: I1223 22:40:36.456023   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 22:40:36 freenas k3s[16024]: E1223 22:40:36.459969   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 22:42:38.459945873 +0200 EET m=+112798.852556075 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 22:40:37 freenas k3s[16024]: time="2023-12-23T22:40:37+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 22:40:37 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:40:37.537+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:41:18 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:41:18.491+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:41:49 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:41:49.310+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:42:27 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:42:27.198+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:42:38 freenas k3s[16024]: I1223 22:42:38.177942   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 22:42:38 freenas k3s[16024]: E1223 22:42:38.182785   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 22:44:40.182757003 +0200 EET m=+112920.575367215 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 22:42:38 freenas k3s[16024]: I1223 22:42:38.279430   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 22:42:38 freenas k3s[16024]: E1223 22:42:38.283520   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 22:44:40.283490295 +0200 EET m=+112920.676100507 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 22:42:38 freenas k3s[16024]: I1223 22:42:38.481814   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 22:42:38 freenas k3s[16024]: E1223 22:42:38.486464   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 22:44:40.486436071 +0200 EET m=+112920.879046273 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 22:42:40 freenas k3s[16024]: time="2023-12-23T22:42:40+02:00" level=info msg="COMPACT compactRev=67568180 targetCompactRev=67568885 currentRev=67569885"
Dec 23 22:42:40 freenas k3s[16024]: time="2023-12-23T22:42:40+02:00" level=info msg="COMPACT deleted 705 rows from 705 revisions in 18.240237ms - compacted to 67568885/67569885"
Dec 23 22:42:47 freenas k3s[16024]: time="2023-12-23T22:42:47+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:42:47 freenas k3s[16024]: time="2023-12-23T22:42:47+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:42:52 freenas k3s[16024]: time="2023-12-23T22:42:52+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 22:43:10 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:43:10.621+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:43:43 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:43:43.893+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:44:28 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:44:28.701+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:44:40 freenas k3s[16024]: I1223 22:44:40.198683   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 22:44:40 freenas k3s[16024]: E1223 22:44:40.203787   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 22:46:42.203760792 +0200 EET m=+113042.596370994 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 22:44:40 freenas k3s[16024]: I1223 22:44:40.299415   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 22:44:40 freenas k3s[16024]: E1223 22:44:40.303644   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 22:46:42.303622365 +0200 EET m=+113042.696232568 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 22:44:40 freenas k3s[16024]: I1223 22:44:40.501249   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 22:44:40 freenas k3s[16024]: E1223 22:44:40.505155   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 22:46:42.505130991 +0200 EET m=+113042.897741193 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 22:45:02 freenas k3s[16024]: time="2023-12-23T22:45:02+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:45:02 freenas k3s[16024]: time="2023-12-23T22:45:02+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:45:07 freenas k3s[16024]: time="2023-12-23T22:45:07+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 22:45:08 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:45:08.189+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:45:49 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:45:49.032+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:46:21 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:46:21.810+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:46:42 freenas k3s[16024]: I1223 22:46:42.216179   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 22:46:42 freenas k3s[16024]: E1223 22:46:42.221238   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 22:48:44.221212282 +0200 EET m=+113164.613822484 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 22:46:42 freenas k3s[16024]: I1223 22:46:42.317565   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 22:46:42 freenas k3s[16024]: E1223 22:46:42.322044   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 22:48:44.322009026 +0200 EET m=+113164.714619228 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 22:46:42 freenas k3s[16024]: I1223 22:46:42.519982   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 22:46:42 freenas k3s[16024]: E1223 22:46:42.524482   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 22:48:44.524464002 +0200 EET m=+113164.917074184 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 22:46:55 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:46:55.455+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:47:18 freenas k3s[16024]: time="2023-12-23T22:47:18+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:47:18 freenas k3s[16024]: time="2023-12-23T22:47:18+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:47:23 freenas k3s[16024]: time="2023-12-23T22:47:23+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 22:47:29 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:47:29.300+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:47:40 freenas k3s[16024]: time="2023-12-23T22:47:40+02:00" level=info msg="COMPACT compactRev=67568885 targetCompactRev=67569588 currentRev=67570588"
Dec 23 22:47:40 freenas k3s[16024]: time="2023-12-23T22:47:40+02:00" level=info msg="COMPACT deleted 703 rows from 703 revisions in 28.236218ms - compacted to 67569588/67570588"
Dec 23 22:48:02 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:48:02.018+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:48:44 freenas k3s[16024]: I1223 22:48:44.269182   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 22:48:44 freenas k3s[16024]: E1223 22:48:44.274155   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 22:50:46.274127352 +0200 EET m=+113286.666737554 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 22:48:44 freenas k3s[16024]: I1223 22:48:44.370255   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 22:48:44 freenas k3s[16024]: E1223 22:48:44.374721   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 22:50:46.374693795 +0200 EET m=+113286.767303998 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 22:48:44 freenas k3s[16024]: I1223 22:48:44.572088   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 22:48:44 freenas k3s[16024]: E1223 22:48:44.576493   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 22:50:46.576463581 +0200 EET m=+113286.969073783 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 22:48:46 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:48:46.709+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:49:23 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:49:23.588+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:49:33 freenas k3s[16024]: time="2023-12-23T22:49:33+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:49:33 freenas k3s[16024]: time="2023-12-23T22:49:33+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:49:38 freenas k3s[16024]: time="2023-12-23T22:49:38+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 22:49:54 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:49:54.326+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:50:34 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:50:34.697+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:50:46 freenas k3s[16024]: I1223 22:50:46.329454   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 22:50:46 freenas k3s[16024]: E1223 22:50:46.334945   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 22:52:48.33491539 +0200 EET m=+113408.727525662 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 22:50:46 freenas k3s[16024]: I1223 22:50:46.430668   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 22:50:46 freenas k3s[16024]: E1223 22:50:46.435626   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 22:52:48.435583888 +0200 EET m=+113408.828194090 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 22:50:46 freenas k3s[16024]: I1223 22:50:46.634058   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 22:50:46 freenas k3s[16024]: E1223 22:50:46.638777   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 22:52:48.638737021 +0200 EET m=+113409.031347233 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 22:51:13 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:51:13.528+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:51:48 freenas k3s[16024]: time="2023-12-23T22:51:48+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:51:48 freenas k3s[16024]: time="2023-12-23T22:51:48+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:51:53 freenas k3s[16024]: time="2023-12-23T22:51:53+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 22:51:53 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:51:53.429+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:52:31 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:52:31.850+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:52:40 freenas k3s[16024]: time="2023-12-23T22:52:40+02:00" level=info msg="COMPACT compactRev=67569588 targetCompactRev=67570288 currentRev=67571288"
Dec 23 22:52:40 freenas k3s[16024]: time="2023-12-23T22:52:40+02:00" level=info msg="COMPACT deleted 700 rows from 700 revisions in 13.57023ms - compacted to 67570288/67571288"
Dec 23 22:52:48 freenas k3s[16024]: I1223 22:52:48.400974   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 22:52:48 freenas k3s[16024]: E1223 22:52:48.406177   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 22:54:50.406142105 +0200 EET m=+113530.798752317 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 22:52:48 freenas k3s[16024]: I1223 22:52:48.501620   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 22:52:48 freenas k3s[16024]: E1223 22:52:48.506202   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 22:54:50.506162774 +0200 EET m=+113530.898772976 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 22:52:48 freenas k3s[16024]: I1223 22:52:48.703939   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 22:52:48 freenas k3s[16024]: E1223 22:52:48.708671   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 22:54:50.708639993 +0200 EET m=+113531.101250195 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 22:53:08 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:53:08.796+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:53:52 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:53:52.840+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:54:03 freenas k3s[16024]: time="2023-12-23T22:54:03+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:54:03 freenas k3s[16024]: time="2023-12-23T22:54:03+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:54:08 freenas k3s[16024]: time="2023-12-23T22:54:08+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 22:54:24 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:54:24.655+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:54:50 freenas k3s[16024]: I1223 22:54:50.456727   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 22:54:50 freenas k3s[16024]: E1223 22:54:50.461001   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 22:56:52.460975479 +0200 EET m=+113652.853585691 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 22:54:50 freenas k3s[16024]: I1223 22:54:50.557504   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 22:54:50 freenas k3s[16024]: E1223 22:54:50.561594   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 22:56:52.561569322 +0200 EET m=+113652.954179534 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 22:54:50 freenas k3s[16024]: I1223 22:54:50.759037   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 22:54:50 freenas k3s[16024]: E1223 22:54:50.762893   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 22:56:52.762868381 +0200 EET m=+113653.155478594 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 22:54:57 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:54:57.868+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:55:36 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:55:36.814+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:56:18 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:56:18.481+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:56:18 freenas k3s[16024]: time="2023-12-23T22:56:18+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:56:18 freenas k3s[16024]: time="2023-12-23T22:56:18+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:56:23 freenas k3s[16024]: time="2023-12-23T22:56:23+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 22:56:52 freenas k3s[16024]: I1223 22:56:52.521667   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 22:56:52 freenas k3s[16024]: E1223 22:56:52.526312   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 22:58:54.526288184 +0200 EET m=+113774.918898396 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 22:56:52 freenas k3s[16024]: I1223 22:56:52.622665   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 22:56:52 freenas k3s[16024]: E1223 22:56:52.627269   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 22:58:54.627240884 +0200 EET m=+113775.019851076 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 22:56:52 freenas k3s[16024]: I1223 22:56:52.826175   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 22:56:52 freenas k3s[16024]: E1223 22:56:52.830584   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 22:58:54.830555509 +0200 EET m=+113775.223165721 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 22:56:54 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:56:54.433+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:57:37 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:57:37.247+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:57:40 freenas k3s[16024]: time="2023-12-23T22:57:40+02:00" level=info msg="COMPACT compactRev=67570288 targetCompactRev=67570991 currentRev=67571991"
Dec 23 22:57:40 freenas k3s[16024]: time="2023-12-23T22:57:40+02:00" level=info msg="COMPACT deleted 703 rows from 703 revisions in 18.297105ms - compacted to 67570991/67571991"
Dec 23 22:58:08 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:58:08.631+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:58:33 freenas k3s[16024]: time="2023-12-23T22:58:33+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:58:33 freenas k3s[16024]: time="2023-12-23T22:58:33+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 22:58:38 freenas k3s[16024]: time="2023-12-23T22:58:38+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 22:58:41 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:58:41.818+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:58:54 freenas k3s[16024]: I1223 22:58:54.556796   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 22:58:54 freenas k3s[16024]: E1223 22:58:54.561929   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 23:00:56.56190266 +0200 EET m=+113896.954512862 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 22:58:54 freenas k3s[16024]: I1223 22:58:54.657682   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 22:58:54 freenas k3s[16024]: E1223 22:58:54.662160   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 23:00:56.662138318 +0200 EET m=+113897.054748530 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 22:58:54 freenas k3s[16024]: I1223 22:58:54.859347   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 22:58:54 freenas k3s[16024]: E1223 22:58:54.863412   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 23:00:56.863392371 +0200 EET m=+113897.256002553 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 22:59:13 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:59:13.183+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 22:59:51 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T22:59:51.964+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:00:22 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:00:22.049+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:00:49 freenas k3s[16024]: time="2023-12-23T23:00:49+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:00:49 freenas k3s[16024]: time="2023-12-23T23:00:49+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:00:54 freenas k3s[16024]: time="2023-12-23T23:00:54+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 23:00:56 freenas k3s[16024]: I1223 23:00:56.588365   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 23:00:56 freenas k3s[16024]: E1223 23:00:56.593047   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 23:02:58.593020163 +0200 EET m=+114018.985630375 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 23:00:56 freenas k3s[16024]: I1223 23:00:56.689081   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 23:00:56 freenas k3s[16024]: E1223 23:00:56.693451   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 23:02:58.693413462 +0200 EET m=+114019.086023664 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 23:00:56 freenas k3s[16024]: I1223 23:00:56.891177   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 23:00:56 freenas k3s[16024]: E1223 23:00:56.895291   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 23:02:58.895264958 +0200 EET m=+114019.287875170 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 23:00:58 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:00:58.109+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:01:32 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:01:32.331+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:02:03 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:02:03.398+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:02:40 freenas k3s[16024]: time="2023-12-23T23:02:40+02:00" level=info msg="COMPACT compactRev=67570991 targetCompactRev=67571695 currentRev=67572695"
Dec 23 23:02:40 freenas k3s[16024]: time="2023-12-23T23:02:40+02:00" level=info msg="COMPACT deleted 704 rows from 704 revisions in 17.747724ms - compacted to 67571695/67572695"
Dec 23 23:02:41 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:02:41.439+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:02:58 freenas k3s[16024]: I1223 23:02:58.642906   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 23:02:58 freenas k3s[16024]: E1223 23:02:58.647993   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 23:05:00.64796445 +0200 EET m=+114141.040574662 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 23:02:58 freenas k3s[16024]: I1223 23:02:58.743770   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 23:02:58 freenas k3s[16024]: E1223 23:02:58.748029   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 23:05:00.748003281 +0200 EET m=+114141.140613483 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 23:02:58 freenas k3s[16024]: I1223 23:02:58.946196   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 23:02:58 freenas k3s[16024]: E1223 23:02:58.950066   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 23:05:00.950026917 +0200 EET m=+114141.342637119 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 23:03:04 freenas k3s[16024]: time="2023-12-23T23:03:04+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:03:04 freenas k3s[16024]: time="2023-12-23T23:03:04+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:03:09 freenas k3s[16024]: time="2023-12-23T23:03:09+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 23:03:18 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:03:18.725+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:04:02 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:04:02.141+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:04:41 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:04:41.476+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:05:00 freenas k3s[16024]: I1223 23:05:00.682435   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 23:05:00 freenas k3s[16024]: E1223 23:05:00.687480   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 23:07:02.687454231 +0200 EET m=+114263.080064433 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 23:05:00 freenas k3s[16024]: I1223 23:05:00.784052   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 23:05:00 freenas k3s[16024]: E1223 23:05:00.788753   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 23:07:02.788726642 +0200 EET m=+114263.181336855 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 23:05:00 freenas k3s[16024]: I1223 23:05:00.986789   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 23:05:00 freenas k3s[16024]: E1223 23:05:00.991336   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 23:07:02.991314096 +0200 EET m=+114263.383924279 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 23:05:19 freenas k3s[16024]: time="2023-12-23T23:05:19+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:05:19 freenas k3s[16024]: time="2023-12-23T23:05:19+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:05:22 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:05:22.122+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:05:24 freenas k3s[16024]: time="2023-12-23T23:05:24+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 23:05:57 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:05:57.991+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:06:29 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:06:29.471+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:07:02 freenas k3s[16024]: I1223 23:07:02.699707   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 23:07:02 freenas k3s[16024]: E1223 23:07:02.704321   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 23:09:04.704290491 +0200 EET m=+114385.096900703 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 23:07:02 freenas k3s[16024]: I1223 23:07:02.800945   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 23:07:02 freenas k3s[16024]: E1223 23:07:02.804935   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 23:09:04.804910755 +0200 EET m=+114385.197520957 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 23:07:03 freenas k3s[16024]: I1223 23:07:03.003308   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 23:07:03 freenas k3s[16024]: E1223 23:07:03.007328   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 23:09:05.007297404 +0200 EET m=+114385.399907606 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 23:07:12 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:07:12.585+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:07:34 freenas k3s[16024]: time="2023-12-23T23:07:34+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:07:34 freenas k3s[16024]: time="2023-12-23T23:07:34+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:07:39 freenas k3s[16024]: time="2023-12-23T23:07:39+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 23:07:40 freenas k3s[16024]: time="2023-12-23T23:07:40+02:00" level=info msg="COMPACT compactRev=67571695 targetCompactRev=67572400 currentRev=67573400"
Dec 23 23:07:40 freenas k3s[16024]: time="2023-12-23T23:07:40+02:00" level=info msg="COMPACT deleted 705 rows from 705 revisions in 17.739198ms - compacted to 67572400/67573400"
Dec 23 23:07:52 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:07:52.200+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:08:35 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:08:35.927+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:09:04 freenas k3s[16024]: I1223 23:09:04.728494   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 23:09:04 freenas k3s[16024]: E1223 23:09:04.733234   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 23:11:06.733207414 +0200 EET m=+114507.125817626 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 23:09:04 freenas k3s[16024]: I1223 23:09:04.830117   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 23:09:04 freenas k3s[16024]: E1223 23:09:04.834311   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 23:11:06.834285984 +0200 EET m=+114507.226896196 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 23:09:05 freenas k3s[16024]: I1223 23:09:05.033290   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 23:09:05 freenas k3s[16024]: E1223 23:09:05.037405   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 23:11:07.037384836 +0200 EET m=+114507.429995008 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 23:09:19 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:09:19.859+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:09:49 freenas k3s[16024]: time="2023-12-23T23:09:49+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:09:49 freenas k3s[16024]: time="2023-12-23T23:09:49+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:09:53 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:09:53.579+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:09:54 freenas k3s[16024]: time="2023-12-23T23:09:54+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 23:10:24 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:10:24.065+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:10:59 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:10:59.183+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:11:06 freenas k3s[16024]: I1223 23:11:06.763862   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 23:11:06 freenas k3s[16024]: E1223 23:11:06.767830   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 23:13:08.767805897 +0200 EET m=+114629.160416099 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 23:11:06 freenas k3s[16024]: I1223 23:11:06.865549   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 23:11:06 freenas k3s[16024]: E1223 23:11:06.868697   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 23:13:08.868674158 +0200 EET m=+114629.261284370 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 23:11:07 freenas k3s[16024]: I1223 23:11:07.067105   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 23:11:07 freenas k3s[16024]: E1223 23:11:07.071253   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 23:13:09.071227763 +0200 EET m=+114629.463837966 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 23:11:34 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:11:34.506+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:12:04 freenas k3s[16024]: time="2023-12-23T23:12:04+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:12:04 freenas k3s[16024]: time="2023-12-23T23:12:04+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:12:09 freenas k3s[16024]: time="2023-12-23T23:12:09+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 23:12:17 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:12:17.113+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:12:40 freenas k3s[16024]: time="2023-12-23T23:12:40+02:00" level=info msg="COMPACT compactRev=67572400 targetCompactRev=67573103 currentRev=67574103"
Dec 23 23:12:40 freenas k3s[16024]: time="2023-12-23T23:12:40+02:00" level=info msg="COMPACT deleted 703 rows from 703 revisions in 19.600902ms - compacted to 67573103/67574103"
Dec 23 23:12:48 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:12:48.381+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:13:08 freenas k3s[16024]: I1223 23:13:08.811003   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 23:13:08 freenas k3s[16024]: E1223 23:13:08.815645   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 23:15:10.815614763 +0200 EET m=+114751.208224975 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 23:13:08 freenas k3s[16024]: I1223 23:13:08.911972   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 23:13:08 freenas k3s[16024]: E1223 23:13:08.915882   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 23:15:10.915860097 +0200 EET m=+114751.308470279 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 23:13:09 freenas k3s[16024]: I1223 23:13:09.114133   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 23:13:09 freenas k3s[16024]: E1223 23:13:09.118845   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 23:15:11.118819491 +0200 EET m=+114751.511429703 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 23:13:25 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:13:25.314+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:14:01 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:14:01.709+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:14:20 freenas k3s[16024]: time="2023-12-23T23:14:20+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:14:20 freenas k3s[16024]: time="2023-12-23T23:14:20+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:14:25 freenas k3s[16024]: time="2023-12-23T23:14:25+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 23:14:36 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:14:36.359+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:15:10 freenas k3s[16024]: I1223 23:15:10.841537   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 23:15:10 freenas k3s[16024]: E1223 23:15:10.846183   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 23:17:12.846145272 +0200 EET m=+114873.238755484 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 23:15:10 freenas k3s[16024]: I1223 23:15:10.942818   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 23:15:10 freenas k3s[16024]: E1223 23:15:10.946856   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 23:17:12.946829947 +0200 EET m=+114873.339440159 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 23:15:11 freenas k3s[16024]: I1223 23:15:11.145161   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 23:15:11 freenas k3s[16024]: E1223 23:15:11.149429   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 23:17:13.149402981 +0200 EET m=+114873.542013183 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 23:15:16 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:15:16.058+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:15:54 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:15:54.074+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:16:34 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:16:34.223+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:16:35 freenas k3s[16024]: time="2023-12-23T23:16:35+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:16:35 freenas k3s[16024]: time="2023-12-23T23:16:35+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:16:40 freenas k3s[16024]: time="2023-12-23T23:16:40+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 23:17:10 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:17:10.533+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:17:12 freenas k3s[16024]: I1223 23:17:12.878496   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 23:17:12 freenas k3s[16024]: E1223 23:17:12.883563   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 23:19:14.883538945 +0200 EET m=+114995.276149147 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 23:17:12 freenas k3s[16024]: I1223 23:17:12.979371   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 23:17:12 freenas k3s[16024]: E1223 23:17:12.983898   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 23:19:14.983874282 +0200 EET m=+114995.376484484 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 23:17:13 freenas k3s[16024]: I1223 23:17:13.181530   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 23:17:13 freenas k3s[16024]: E1223 23:17:13.186043   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 23:19:15.186008276 +0200 EET m=+114995.578618478 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 23:17:40 freenas k3s[16024]: time="2023-12-23T23:17:40+02:00" level=info msg="COMPACT compactRev=67573103 targetCompactRev=67573808 currentRev=67574808"
Dec 23 23:17:40 freenas k3s[16024]: time="2023-12-23T23:17:40+02:00" level=info msg="COMPACT deleted 705 rows from 705 revisions in 18.877267ms - compacted to 67573808/67574808"
Dec 23 23:17:43 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:17:43.885+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:18:15 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:18:15.687+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:18:45 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:18:45.937+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:18:50 freenas k3s[16024]: time="2023-12-23T23:18:50+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:18:50 freenas k3s[16024]: time="2023-12-23T23:18:50+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:18:55 freenas k3s[16024]: time="2023-12-23T23:18:55+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 23:19:14 freenas k3s[16024]: I1223 23:19:14.908511   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 23:19:14 freenas k3s[16024]: E1223 23:19:14.912789   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 23:21:16.912759116 +0200 EET m=+115117.305369318 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 23:19:15 freenas k3s[16024]: I1223 23:19:15.009891   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 23:19:15 freenas k3s[16024]: E1223 23:19:15.012927   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 23:21:17.01291017 +0200 EET m=+115117.405520342 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 23:19:15 freenas k3s[16024]: I1223 23:19:15.212598   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 23:19:15 freenas k3s[16024]: E1223 23:19:15.216424   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 23:21:17.21639542 +0200 EET m=+115117.609005632 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 23:19:24 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:19:24.934+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:20:04 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:20:04.225+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:20:43 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:20:43.449+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:21:05 freenas k3s[16024]: time="2023-12-23T23:21:05+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:21:05 freenas k3s[16024]: time="2023-12-23T23:21:05+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:21:10 freenas k3s[16024]: time="2023-12-23T23:21:10+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 23:21:16 freenas k3s[16024]: I1223 23:21:16.960012   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 23:21:16 freenas k3s[16024]: E1223 23:21:16.964883   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 23:23:18.964842352 +0200 EET m=+115239.357452564 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 23:21:17 freenas k3s[16024]: I1223 23:21:17.060565   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 23:21:17 freenas k3s[16024]: E1223 23:21:17.064978   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 23:23:19.06494353 +0200 EET m=+115239.457553742 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 23:21:17 freenas k3s[16024]: I1223 23:21:17.263425   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 23:21:17 freenas k3s[16024]: E1223 23:21:17.267777   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 23:23:19.267751904 +0200 EET m=+115239.660362116 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 23:21:27 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:21:27.603+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:21:59 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:21:59.752+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:22:37 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:22:37.218+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:22:40 freenas k3s[16024]: time="2023-12-23T23:22:40+02:00" level=info msg="COMPACT compactRev=67573808 targetCompactRev=67574513 currentRev=67575513"
Dec 23 23:22:40 freenas k3s[16024]: time="2023-12-23T23:22:40+02:00" level=info msg="COMPACT deleted 705 rows from 705 revisions in 20.219986ms - compacted to 67574513/67575513"
Dec 23 23:23:15 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:23:15.636+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:23:18 freenas k3s[16024]: I1223 23:23:18.989851   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 23:23:18 freenas k3s[16024]: E1223 23:23:18.993935   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 23:25:20.993918757 +0200 EET m=+115361.386528929 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 23:23:19 freenas k3s[16024]: I1223 23:23:19.091017   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 23:23:19 freenas k3s[16024]: E1223 23:23:19.094998   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 23:25:21.09496811 +0200 EET m=+115361.487578312 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 23:23:19 freenas k3s[16024]: I1223 23:23:19.292989   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 23:23:19 freenas k3s[16024]: E1223 23:23:19.297037   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 23:25:21.29701142 +0200 EET m=+115361.689621632 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 23:23:20 freenas k3s[16024]: time="2023-12-23T23:23:20+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:23:20 freenas k3s[16024]: time="2023-12-23T23:23:20+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:23:25 freenas k3s[16024]: time="2023-12-23T23:23:25+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 23:23:50 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:23:50.514+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:24:34 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:24:34.708+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:25:17 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:25:17.598+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:25:21 freenas k3s[16024]: I1223 23:25:21.035327   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 23:25:21 freenas k3s[16024]: E1223 23:25:21.040544   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 23:27:23.040516623 +0200 EET m=+115483.433126825 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 23:25:21 freenas k3s[16024]: I1223 23:25:21.136575   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 23:25:21 freenas k3s[16024]: E1223 23:25:21.141006   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 23:27:23.140973172 +0200 EET m=+115483.533583424 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 23:25:21 freenas k3s[16024]: I1223 23:25:21.339212   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 23:25:21 freenas k3s[16024]: E1223 23:25:21.343715   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 23:27:23.343686665 +0200 EET m=+115483.736296867 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 23:25:35 freenas k3s[16024]: time="2023-12-23T23:25:35+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:25:35 freenas k3s[16024]: time="2023-12-23T23:25:35+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:25:40 freenas k3s[16024]: time="2023-12-23T23:25:40+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 23:25:55 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:25:55.273+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:26:36 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:26:36.301+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:27:18 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:27:18.125+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:27:23 freenas k3s[16024]: I1223 23:27:23.048364   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 23:27:23 freenas k3s[16024]: E1223 23:27:23.053692   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 23:29:25.053650911 +0200 EET m=+115605.446261113 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 23:27:23 freenas k3s[16024]: I1223 23:27:23.149453   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 23:27:23 freenas k3s[16024]: E1223 23:27:23.154296   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 23:29:25.154265903 +0200 EET m=+115605.546876115 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 23:27:23 freenas k3s[16024]: I1223 23:27:23.351905   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 23:27:23 freenas k3s[16024]: E1223 23:27:23.355711   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 23:29:25.355688435 +0200 EET m=+115605.748298617 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 23:27:40 freenas k3s[16024]: time="2023-12-23T23:27:40+02:00" level=info msg="COMPACT compactRev=67574513 targetCompactRev=67575217 currentRev=67576217"
Dec 23 23:27:40 freenas k3s[16024]: time="2023-12-23T23:27:40+02:00" level=info msg="COMPACT deleted 704 rows from 704 revisions in 18.41379ms - compacted to 67575217/67576217"
Dec 23 23:27:51 freenas k3s[16024]: time="2023-12-23T23:27:51+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:27:51 freenas k3s[16024]: time="2023-12-23T23:27:51+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:27:55 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:27:55.223+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:27:56 freenas k3s[16024]: time="2023-12-23T23:27:56+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 23:28:34 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:28:34.625+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:29:05 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:29:05.438+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:29:25 freenas k3s[16024]: I1223 23:29:25.072398   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 23:29:25 freenas k3s[16024]: E1223 23:29:25.076411   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 23:31:27.076390311 +0200 EET m=+115727.469000483 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 23:29:25 freenas k3s[16024]: I1223 23:29:25.173749   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 23:29:25 freenas k3s[16024]: E1223 23:29:25.177969   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 23:31:27.177943937 +0200 EET m=+115727.570554139 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 23:29:25 freenas k3s[16024]: I1223 23:29:25.376538   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 23:29:25 freenas k3s[16024]: E1223 23:29:25.380268   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 23:31:27.380247029 +0200 EET m=+115727.772857211 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 23:29:47 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:29:47.113+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:30:06 freenas k3s[16024]: time="2023-12-23T23:30:06+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:30:06 freenas k3s[16024]: time="2023-12-23T23:30:06+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:30:11 freenas k3s[16024]: time="2023-12-23T23:30:11+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 23:30:19 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:30:19.588+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:30:57 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:30:57.411+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:31:27 freenas k3s[16024]: I1223 23:31:27.112440   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 23:31:27 freenas k3s[16024]: E1223 23:31:27.115928   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 23:33:29.115910548 +0200 EET m=+115849.508520720 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 23:31:27 freenas k3s[16024]: I1223 23:31:27.213346   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 23:31:27 freenas k3s[16024]: E1223 23:31:27.216912   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 23:33:29.216879816 +0200 EET m=+115849.609490028 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 23:31:27 freenas k3s[16024]: I1223 23:31:27.416105   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 23:31:27 freenas k3s[16024]: E1223 23:31:27.420399   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 23:33:29.420379358 +0200 EET m=+115849.812989550 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 23:31:32 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:31:32.780+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:32:07 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:32:07.405+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:32:21 freenas k3s[16024]: time="2023-12-23T23:32:21+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:32:21 freenas k3s[16024]: time="2023-12-23T23:32:21+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:32:26 freenas k3s[16024]: time="2023-12-23T23:32:26+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 23:32:40 freenas k3s[16024]: time="2023-12-23T23:32:40+02:00" level=info msg="COMPACT compactRev=67575217 targetCompactRev=67575922 currentRev=67576922"
Dec 23 23:32:40 freenas k3s[16024]: time="2023-12-23T23:32:40+02:00" level=info msg="COMPACT deleted 705 rows from 705 revisions in 26.492722ms - compacted to 67575922/67576922"
Dec 23 23:32:41 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:32:41.370+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:33:20 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:33:20.588+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:33:29 freenas k3s[16024]: I1223 23:33:29.152258   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 23:33:29 freenas k3s[16024]: E1223 23:33:29.156941   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 23:35:31.156912661 +0200 EET m=+115971.549522873 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 23:33:29 freenas k3s[16024]: I1223 23:33:29.253392   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 23:33:29 freenas k3s[16024]: E1223 23:33:29.257151   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 23:35:31.257113177 +0200 EET m=+115971.649723430 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 23:33:29 freenas k3s[16024]: I1223 23:33:29.456063   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 23:33:29 freenas k3s[16024]: E1223 23:33:29.461180   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 23:35:31.461145228 +0200 EET m=+115971.853755441 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 23:34:00 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:34:00.459+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:34:31 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:34:31.348+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:34:36 freenas k3s[16024]: time="2023-12-23T23:34:36+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:34:36 freenas k3s[16024]: time="2023-12-23T23:34:36+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:34:41 freenas k3s[16024]: time="2023-12-23T23:34:41+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 23:35:11 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:35:11.517+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:35:31 freenas k3s[16024]: I1223 23:35:31.209074   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 23:35:31 freenas k3s[16024]: E1223 23:35:31.214097   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 23:37:33.214064405 +0200 EET m=+116093.606674617 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 23:35:31 freenas k3s[16024]: I1223 23:35:31.309974   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 23:35:31 freenas k3s[16024]: E1223 23:35:31.314311   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 23:37:33.314274861 +0200 EET m=+116093.706885073 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 23:35:31 freenas k3s[16024]: I1223 23:35:31.511790   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 23:35:31 freenas k3s[16024]: E1223 23:35:31.516210   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 23:37:33.516181244 +0200 EET m=+116093.908791456 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 23:35:56 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:35:56.447+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:36:31 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:36:31.355+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:36:51 freenas k3s[16024]: time="2023-12-23T23:36:51+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:36:51 freenas k3s[16024]: time="2023-12-23T23:36:51+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:36:56 freenas k3s[16024]: time="2023-12-23T23:36:56+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 23:37:02 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:37:02.920+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:37:33 freenas k3s[16024]: I1223 23:37:33.261001   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 23:37:33 freenas k3s[16024]: E1223 23:37:33.266074   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 23:39:35.266045355 +0200 EET m=+116215.658655557 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 23:37:33 freenas k3s[16024]: I1223 23:37:33.362056   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 23:37:33 freenas k3s[16024]: E1223 23:37:33.365811   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 23:39:35.365775952 +0200 EET m=+116215.758386164 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 23:37:33 freenas k3s[16024]: I1223 23:37:33.564223   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 23:37:33 freenas k3s[16024]: E1223 23:37:33.568237   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 23:39:35.568211889 +0200 EET m=+116215.960822091 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 23:37:36 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:37:36.831+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:37:40 freenas k3s[16024]: time="2023-12-23T23:37:40+02:00" level=info msg="COMPACT compactRev=67575922 targetCompactRev=67576624 currentRev=67577624"
Dec 23 23:37:40 freenas k3s[16024]: time="2023-12-23T23:37:40+02:00" level=info msg="COMPACT deleted 702 rows from 702 revisions in 18.53203ms - compacted to 67576624/67577624"
Dec 23 23:38:18 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:38:18.150+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:38:57 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:38:57.084+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:39:06 freenas k3s[16024]: time="2023-12-23T23:39:06+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:39:06 freenas k3s[16024]: time="2023-12-23T23:39:06+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:39:11 freenas k3s[16024]: time="2023-12-23T23:39:11+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 23:39:33 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:39:33.499+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:39:35 freenas k3s[16024]: I1223 23:39:35.268675   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 23:39:35 freenas k3s[16024]: E1223 23:39:35.273204   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 23:41:37.273175997 +0200 EET m=+116337.665786179 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 23:39:35 freenas k3s[16024]: I1223 23:39:35.369659   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 23:39:35 freenas k3s[16024]: E1223 23:39:35.373858   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 23:41:37.373830442 +0200 EET m=+116337.766440644 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 23:39:35 freenas k3s[16024]: I1223 23:39:35.571549   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 23:39:35 freenas k3s[16024]: E1223 23:39:35.575764   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 23:41:37.575734982 +0200 EET m=+116337.968345165 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 23:40:05 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:40:05.741+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:40:39 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:40:39.192+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:41:10 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:41:10.537+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:41:22 freenas k3s[16024]: time="2023-12-23T23:41:22+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:41:22 freenas k3s[16024]: time="2023-12-23T23:41:22+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:41:27 freenas k3s[16024]: time="2023-12-23T23:41:27+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 23:41:37 freenas k3s[16024]: I1223 23:41:37.312882   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 23:41:37 freenas k3s[16024]: E1223 23:41:37.317660   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 23:43:39.317632329 +0200 EET m=+116459.710242541 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 23:41:37 freenas k3s[16024]: I1223 23:41:37.414557   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 23:41:37 freenas k3s[16024]: E1223 23:41:37.418807   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 23:43:39.418774738 +0200 EET m=+116459.811384950 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 23:41:37 freenas k3s[16024]: I1223 23:41:37.616825   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 23:41:37 freenas k3s[16024]: E1223 23:41:37.621296   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 23:43:39.621271701 +0200 EET m=+116460.013881903 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 23:41:47 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:41:47.691+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:42:23 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:42:23.310+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:42:40 freenas k3s[16024]: time="2023-12-23T23:42:40+02:00" level=info msg="COMPACT compactRev=67576624 targetCompactRev=67577327 currentRev=67578327"
Dec 23 23:42:40 freenas k3s[16024]: time="2023-12-23T23:42:40+02:00" level=info msg="COMPACT deleted 703 rows from 703 revisions in 20.395505ms - compacted to 67577327/67578327"
Dec 23 23:42:58 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:42:58.823+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:43:29 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:43:29.659+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:43:37 freenas k3s[16024]: time="2023-12-23T23:43:37+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:43:37 freenas k3s[16024]: time="2023-12-23T23:43:37+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:43:39 freenas k3s[16024]: I1223 23:43:39.348697   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 23:43:39 freenas k3s[16024]: E1223 23:43:39.352683   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 23:45:41.352654078 +0200 EET m=+116581.745264280 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 23:43:39 freenas k3s[16024]: I1223 23:43:39.449464   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 23:43:39 freenas k3s[16024]: E1223 23:43:39.453768   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 23:45:41.453738526 +0200 EET m=+116581.846348738 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 23:43:39 freenas k3s[16024]: I1223 23:43:39.653006   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 23:43:39 freenas k3s[16024]: E1223 23:43:39.656762   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 23:45:41.656739625 +0200 EET m=+116582.049349807 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 23:43:42 freenas k3s[16024]: time="2023-12-23T23:43:42+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 23:44:14 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:44:14.056+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:44:47 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:44:47.919+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:45:28 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:45:28.238+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:45:41 freenas k3s[16024]: I1223 23:45:41.409250   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 23:45:41 freenas k3s[16024]: E1223 23:45:41.413900   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 23:47:43.413873076 +0200 EET m=+116703.806483288 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 23:45:41 freenas k3s[16024]: I1223 23:45:41.510271   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 23:45:41 freenas k3s[16024]: E1223 23:45:41.514444   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 23:47:43.514421347 +0200 EET m=+116703.907031549 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 23:45:41 freenas k3s[16024]: I1223 23:45:41.712723   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 23:45:41 freenas k3s[16024]: E1223 23:45:41.717154   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 23:47:43.717124024 +0200 EET m=+116704.109734236 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 23:45:52 freenas k3s[16024]: time="2023-12-23T23:45:52+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:45:52 freenas k3s[16024]: time="2023-12-23T23:45:52+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:45:57 freenas k3s[16024]: time="2023-12-23T23:45:57+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 23:46:02 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:46:02.492+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:46:46 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:46:46.705+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:47:23 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:47:23.956+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:47:40 freenas k3s[16024]: time="2023-12-23T23:47:40+02:00" level=info msg="COMPACT compactRev=67577327 targetCompactRev=67578027 currentRev=67579027"
Dec 23 23:47:40 freenas k3s[16024]: time="2023-12-23T23:47:40+02:00" level=info msg="COMPACT deleted 700 rows from 700 revisions in 17.779174ms - compacted to 67578027/67579027"
Dec 23 23:47:43 freenas k3s[16024]: I1223 23:47:43.503083   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 23:47:43 freenas k3s[16024]: E1223 23:47:43.506883   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 23:49:45.506855815 +0200 EET m=+116825.899465997 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 23:47:43 freenas k3s[16024]: I1223 23:47:43.604778   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 23:47:43 freenas k3s[16024]: E1223 23:47:43.608577   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 23:49:45.60855007 +0200 EET m=+116826.001160272 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 23:47:43 freenas k3s[16024]: I1223 23:47:43.806924   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 23:47:43 freenas k3s[16024]: E1223 23:47:43.810787   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 23:49:45.810754333 +0200 EET m=+116826.203364525 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 23:48:04 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:48:04.083+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:48:07 freenas k3s[16024]: time="2023-12-23T23:48:07+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:48:07 freenas k3s[16024]: time="2023-12-23T23:48:07+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:48:12 freenas k3s[16024]: time="2023-12-23T23:48:12+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 23:48:37 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:48:37.692+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:49:14 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:49:14.036+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:49:45 freenas k3s[16024]: I1223 23:49:45.560742   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 23:49:45 freenas k3s[16024]: E1223 23:49:45.565433   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 23:51:47.565406712 +0200 EET m=+116947.958016914 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 23:49:45 freenas k3s[16024]: I1223 23:49:45.662295   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 23:49:45 freenas k3s[16024]: E1223 23:49:45.666226   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 23:51:47.666200623 +0200 EET m=+116948.058810835 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 23:49:45 freenas k3s[16024]: I1223 23:49:45.863910   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 23:49:45 freenas k3s[16024]: E1223 23:49:45.868147   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 23:51:47.868129118 +0200 EET m=+116948.260739290 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 23:49:47 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:49:47.583+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:50:22 freenas k3s[16024]: time="2023-12-23T23:50:22+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:50:22 freenas k3s[16024]: time="2023-12-23T23:50:22+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:50:27 freenas k3s[16024]: time="2023-12-23T23:50:27+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 23:50:32 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:50:32.558+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:51:13 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:51:13.595+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:51:47 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:51:47.537+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:51:47 freenas k3s[16024]: I1223 23:51:47.609177   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 23:51:47 freenas k3s[16024]: E1223 23:51:47.613363   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 23:53:49.613334076 +0200 EET m=+117070.005944278 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 23:51:47 freenas k3s[16024]: I1223 23:51:47.709628   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 23:51:47 freenas k3s[16024]: E1223 23:51:47.713960   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 23:53:49.713924278 +0200 EET m=+117070.106534490 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 23:51:47 freenas k3s[16024]: I1223 23:51:47.912688   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 23:51:47 freenas k3s[16024]: E1223 23:51:47.917305   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 23:53:49.917279473 +0200 EET m=+117070.309889686 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 23:52:29 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:52:29.717+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:52:37 freenas k3s[16024]: time="2023-12-23T23:52:37+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:52:37 freenas k3s[16024]: time="2023-12-23T23:52:37+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:52:40 freenas k3s[16024]: time="2023-12-23T23:52:40+02:00" level=info msg="COMPACT compactRev=67578027 targetCompactRev=67578731 currentRev=67579731"
Dec 23 23:52:40 freenas k3s[16024]: time="2023-12-23T23:52:40+02:00" level=info msg="COMPACT deleted 704 rows from 704 revisions in 18.619974ms - compacted to 67578731/67579731"
Dec 23 23:52:42 freenas k3s[16024]: time="2023-12-23T23:52:42+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 23:53:06 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:53:06.789+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:53:41 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:53:41.067+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:53:49 freenas k3s[16024]: I1223 23:53:49.665999   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 23:53:49 freenas k3s[16024]: E1223 23:53:49.670370   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 23:55:51.670344023 +0200 EET m=+117192.062954235 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 23:53:49 freenas k3s[16024]: I1223 23:53:49.766769   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 23:53:49 freenas k3s[16024]: E1223 23:53:49.769992   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 23:55:51.769979397 +0200 EET m=+117192.162589569 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 23:53:49 freenas k3s[16024]: I1223 23:53:49.969024   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 23:53:49 freenas k3s[16024]: E1223 23:53:49.972464   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 23:55:51.972447145 +0200 EET m=+117192.365057317 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 23:54:23 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:54:23.480+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:54:53 freenas k3s[16024]: time="2023-12-23T23:54:53+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:54:53 freenas k3s[16024]: time="2023-12-23T23:54:53+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:54:55 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:54:55.523+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:54:58 freenas k3s[16024]: time="2023-12-23T23:54:58+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 23:55:29 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:55:29.263+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:55:51 freenas k3s[16024]: I1223 23:55:51.714500   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 23:55:51 freenas k3s[16024]: E1223 23:55:51.718732   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 23:57:53.718717001 +0200 EET m=+117314.111327193 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 23:55:51 freenas k3s[16024]: I1223 23:55:51.815186   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 23:55:51 freenas k3s[16024]: E1223 23:55:51.819866   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 23:57:53.819840123 +0200 EET m=+117314.212450325 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 23:55:52 freenas k3s[16024]: I1223 23:55:52.017962   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 23:55:52 freenas k3s[16024]: E1223 23:55:52.022594   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 23:57:54.022568639 +0200 EET m=+117314.415178841 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 23:56:12 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:56:12.171+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:56:51 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:56:51.223+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:57:08 freenas k3s[16024]: time="2023-12-23T23:57:08+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:57:08 freenas k3s[16024]: time="2023-12-23T23:57:08+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:57:13 freenas k3s[16024]: time="2023-12-23T23:57:13+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 23:57:32 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:57:32.954+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:57:40 freenas k3s[16024]: time="2023-12-23T23:57:40+02:00" level=info msg="COMPACT compactRev=67578731 targetCompactRev=67579437 currentRev=67580437"
Dec 23 23:57:40 freenas k3s[16024]: time="2023-12-23T23:57:40+02:00" level=info msg="COMPACT deleted 706 rows from 706 revisions in 10.955736ms - compacted to 67579437/67580437"
Dec 23 23:57:53 freenas k3s[16024]: I1223 23:57:53.741650   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 23:57:53 freenas k3s[16024]: E1223 23:57:53.746729   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-23 23:59:55.746702655 +0200 EET m=+117436.139312857 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 23:57:53 freenas k3s[16024]: I1223 23:57:53.843496   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 23:57:53 freenas k3s[16024]: E1223 23:57:53.848433   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-23 23:59:55.848406738 +0200 EET m=+117436.241016950 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 23:57:54 freenas k3s[16024]: I1223 23:57:54.046668   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 23:57:54 freenas k3s[16024]: E1223 23:57:54.051349   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-23 23:59:56.051313054 +0200 EET m=+117436.443923266 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 23 23:58:09 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:58:09.950+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:58:40 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:58:40.712+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:59:17 freenas k3s[16024]: {"level":"warn","ts":"2023-12-23T23:59:17.878+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 23 23:59:23 freenas k3s[16024]: time="2023-12-23T23:59:23+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:59:23 freenas k3s[16024]: time="2023-12-23T23:59:23+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 23 23:59:28 freenas k3s[16024]: time="2023-12-23T23:59:28+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 23 23:59:55 freenas k3s[16024]: I1223 23:59:55.752338   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 23 23:59:55 freenas k3s[16024]: E1223 23:59:55.756934   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-24 00:01:57.756906648 +0200 EET m=+117558.149516850 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 23 23:59:55 freenas k3s[16024]: I1223 23:59:55.853258   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 23 23:59:55 freenas k3s[16024]: E1223 23:59:55.857627   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-24 00:01:57.857600248 +0200 EET m=+117558.250210450 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 23 23:59:56 freenas k3s[16024]: I1223 23:59:56.054754   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 23 23:59:56 freenas k3s[16024]: E1223 23:59:56.059019   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-24 00:01:58.058999039 +0200 EET m=+117558.451609211 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 24 00:00:00 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:00:00.299+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:00:34 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:00:34.576+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:01:11 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:01:11.444+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:01:38 freenas k3s[16024]: time="2023-12-24T00:01:38+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 24 00:01:38 freenas k3s[16024]: time="2023-12-24T00:01:38+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 24 00:01:43 freenas k3s[16024]: time="2023-12-24T00:01:43+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 24 00:01:50 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:01:50.353+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:01:57 freenas k3s[16024]: I1224 00:01:57.770428   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 24 00:01:57 freenas k3s[16024]: E1224 00:01:57.775133   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-24 00:03:59.775111443 +0200 EET m=+117680.167721715 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 24 00:01:57 freenas k3s[16024]: I1224 00:01:57.871533   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 24 00:01:57 freenas k3s[16024]: E1224 00:01:57.875924   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-24 00:03:59.87589283 +0200 EET m=+117680.268503012 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 24 00:01:58 freenas k3s[16024]: I1224 00:01:58.073438   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 24 00:01:58 freenas k3s[16024]: E1224 00:01:58.077369   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-24 00:04:00.077339714 +0200 EET m=+117680.469949926 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 24 00:02:23 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:02:23.785+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:02:40 freenas k3s[16024]: time="2023-12-24T00:02:40+02:00" level=info msg="COMPACT compactRev=67579437 targetCompactRev=67580142 currentRev=67581142"
Dec 24 00:02:40 freenas k3s[16024]: time="2023-12-24T00:02:40+02:00" level=info msg="COMPACT deleted 705 rows from 705 revisions in 14.468719ms - compacted to 67580142/67581142"
Dec 24 00:03:05 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:03:05.989+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:03:45 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:03:45.349+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:03:53 freenas k3s[16024]: time="2023-12-24T00:03:53+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 24 00:03:53 freenas k3s[16024]: time="2023-12-24T00:03:53+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 24 00:03:58 freenas k3s[16024]: time="2023-12-24T00:03:58+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 24 00:03:59 freenas k3s[16024]: I1224 00:03:59.787309   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 24 00:03:59 freenas k3s[16024]: E1224 00:03:59.792445   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-24 00:06:01.792403698 +0200 EET m=+117802.185013951 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 24 00:03:59 freenas k3s[16024]: I1224 00:03:59.888247   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 24 00:03:59 freenas k3s[16024]: E1224 00:03:59.892336   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-24 00:06:01.892316055 +0200 EET m=+117802.284926237 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 24 00:04:00 freenas k3s[16024]: I1224 00:04:00.090581   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 24 00:04:00 freenas k3s[16024]: E1224 00:04:00.094131   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-24 00:06:02.094107792 +0200 EET m=+117802.486717985 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 24 00:04:17 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:04:17.215+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:04:48 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:04:48.820+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:05:25 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:05:25.506+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:05:55 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:05:55.792+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:06:01 freenas k3s[16024]: I1224 00:06:01.850569   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 24 00:06:01 freenas k3s[16024]: E1224 00:06:01.855490   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-24 00:08:03.85546365 +0200 EET m=+117924.248073903 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 24 00:06:01 freenas k3s[16024]: I1224 00:06:01.952331   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 24 00:06:01 freenas k3s[16024]: E1224 00:06:01.957521   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-24 00:08:03.957475865 +0200 EET m=+117924.350086078 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 24 00:06:02 freenas k3s[16024]: I1224 00:06:02.155878   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 24 00:06:02 freenas k3s[16024]: E1224 00:06:02.159585   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-24 00:08:04.159556245 +0200 EET m=+117924.552166447 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 24 00:06:08 freenas k3s[16024]: time="2023-12-24T00:06:08+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 24 00:06:08 freenas k3s[16024]: time="2023-12-24T00:06:08+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 24 00:06:13 freenas k3s[16024]: time="2023-12-24T00:06:13+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 24 00:06:30 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:06:30.378+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:07:13 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:07:13.252+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:07:40 freenas k3s[16024]: time="2023-12-24T00:07:40+02:00" level=info msg="COMPACT compactRev=67580142 targetCompactRev=67580849 currentRev=67581849"
Dec 24 00:07:40 freenas k3s[16024]: time="2023-12-24T00:07:40+02:00" level=info msg="COMPACT deleted 707 rows from 707 revisions in 19.624516ms - compacted to 67580849/67581849"
Dec 24 00:07:49 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:07:49.234+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:08:03 freenas k3s[16024]: I1224 00:08:03.895942   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 24 00:08:03 freenas k3s[16024]: E1224 00:08:03.901198   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-24 00:10:05.901166134 +0200 EET m=+118046.293776316 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 24 00:08:03 freenas k3s[16024]: I1224 00:08:03.997152   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 24 00:08:04 freenas k3s[16024]: E1224 00:08:04.001488   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-24 00:10:06.001462756 +0200 EET m=+118046.394073009 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 24 00:08:04 freenas k3s[16024]: I1224 00:08:04.199753   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 24 00:08:04 freenas k3s[16024]: E1224 00:08:04.203976   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-24 00:10:06.203959603 +0200 EET m=+118046.596569785 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 24 00:08:24 freenas k3s[16024]: time="2023-12-24T00:08:24+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 24 00:08:24 freenas k3s[16024]: time="2023-12-24T00:08:24+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 24 00:08:29 freenas k3s[16024]: time="2023-12-24T00:08:29+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 24 00:08:29 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:08:29.384+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:09:04 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:09:04.538+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:09:38 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:09:38.288+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:10:05 freenas k3s[16024]: I1224 00:10:05.986624   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 24 00:10:05 freenas k3s[16024]: E1224 00:10:05.990840   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-24 00:12:07.990817224 +0200 EET m=+118168.383427406 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 24 00:10:06 freenas k3s[16024]: I1224 00:10:06.087648   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 24 00:10:06 freenas k3s[16024]: E1224 00:10:06.092136   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-24 00:12:08.09210881 +0200 EET m=+118168.484719012 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 24 00:10:06 freenas k3s[16024]: I1224 00:10:06.289371   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 24 00:10:06 freenas k3s[16024]: E1224 00:10:06.294123   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-24 00:12:08.294099689 +0200 EET m=+118168.686709901 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 24 00:10:12 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:10:12.842+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:10:39 freenas k3s[16024]: time="2023-12-24T00:10:39+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 24 00:10:39 freenas k3s[16024]: time="2023-12-24T00:10:39+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 24 00:10:44 freenas k3s[16024]: time="2023-12-24T00:10:44+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 24 00:10:55 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:10:55.400+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:11:25 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:11:25.932+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:12:06 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:12:06.343+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:12:08 freenas k3s[16024]: I1224 00:12:08.000144   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 24 00:12:08 freenas k3s[16024]: E1224 00:12:08.004049   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-24 00:14:10.004026984 +0200 EET m=+118290.396637166 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 24 00:12:08 freenas k3s[16024]: I1224 00:12:08.101230   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 24 00:12:08 freenas k3s[16024]: E1224 00:12:08.105396   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-24 00:14:10.105371252 +0200 EET m=+118290.497981454 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 24 00:12:08 freenas k3s[16024]: I1224 00:12:08.304056   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 24 00:12:08 freenas k3s[16024]: E1224 00:12:08.307942   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-24 00:14:10.307907093 +0200 EET m=+118290.700517305 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 24 00:12:38 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:12:38.858+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:12:40 freenas k3s[16024]: time="2023-12-24T00:12:40+02:00" level=info msg="COMPACT compactRev=67580849 targetCompactRev=67581554 currentRev=67582554"
Dec 24 00:12:40 freenas k3s[16024]: time="2023-12-24T00:12:40+02:00" level=info msg="COMPACT deleted 705 rows from 705 revisions in 19.801955ms - compacted to 67581554/67582554"
Dec 24 00:12:54 freenas k3s[16024]: time="2023-12-24T00:12:54+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 24 00:12:54 freenas k3s[16024]: time="2023-12-24T00:12:54+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 24 00:12:59 freenas k3s[16024]: time="2023-12-24T00:12:59+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 24 00:13:17 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:13:17.038+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:13:52 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:13:52.256+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:14:10 freenas k3s[16024]: I1224 00:14:10.032456   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 24 00:14:10 freenas k3s[16024]: E1224 00:14:10.036484   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-24 00:16:12.036454924 +0200 EET m=+118412.429065126 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 24 00:14:10 freenas k3s[16024]: I1224 00:14:10.133909   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 24 00:14:10 freenas k3s[16024]: E1224 00:14:10.137972   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-24 00:16:12.137947785 +0200 EET m=+118412.530557997 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 24 00:14:10 freenas k3s[16024]: I1224 00:14:10.335712   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 24 00:14:10 freenas k3s[16024]: E1224 00:14:10.340395   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-24 00:16:12.34036553 +0200 EET m=+118412.732975732 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 24 00:14:27 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:14:27.927+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:15:05 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:15:05.925+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:15:09 freenas k3s[16024]: time="2023-12-24T00:15:09+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 24 00:15:09 freenas k3s[16024]: time="2023-12-24T00:15:09+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 24 00:15:14 freenas k3s[16024]: time="2023-12-24T00:15:14+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 24 00:15:42 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:15:42.941+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:16:12 freenas k3s[16024]: I1224 00:16:12.065252   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 24 00:16:12 freenas k3s[16024]: E1224 00:16:12.069883   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-24 00:18:14.069852058 +0200 EET m=+118534.462462260 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 24 00:16:12 freenas k3s[16024]: I1224 00:16:12.166636   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 24 00:16:12 freenas k3s[16024]: E1224 00:16:12.171135   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-24 00:18:14.171105601 +0200 EET m=+118534.563715813 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 24 00:16:12 freenas k3s[16024]: I1224 00:16:12.369325   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 24 00:16:12 freenas k3s[16024]: E1224 00:16:12.373740   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-24 00:18:14.373714532 +0200 EET m=+118534.766324734 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 24 00:16:25 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:16:25.313+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:17:09 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:17:09.952+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:17:24 freenas k3s[16024]: time="2023-12-24T00:17:24+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 24 00:17:24 freenas k3s[16024]: time="2023-12-24T00:17:24+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 24 00:17:29 freenas k3s[16024]: time="2023-12-24T00:17:29+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 24 00:17:40 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:17:40.255+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:17:40 freenas k3s[16024]: time="2023-12-24T00:17:40+02:00" level=info msg="COMPACT compactRev=67581554 targetCompactRev=67582258 currentRev=67583258"
Dec 24 00:17:40 freenas k3s[16024]: time="2023-12-24T00:17:40+02:00" level=info msg="COMPACT deleted 704 rows from 704 revisions in 19.405728ms - compacted to 67582258/67583258"
Dec 24 00:18:14 freenas k3s[16024]: I1224 00:18:14.090317   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 24 00:18:14 freenas k3s[16024]: E1224 00:18:14.094213   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-24 00:20:16.094187632 +0200 EET m=+118656.486797834 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 24 00:18:14 freenas k3s[16024]: I1224 00:18:14.191618   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 24 00:18:14 freenas k3s[16024]: E1224 00:18:14.195633   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-24 00:20:16.195609938 +0200 EET m=+118656.588220110 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 24 00:18:14 freenas k3s[16024]: I1224 00:18:14.393944   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 24 00:18:14 freenas k3s[16024]: E1224 00:18:14.397447   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-24 00:20:16.397411023 +0200 EET m=+118656.790021235 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 24 00:18:20 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:18:20.947+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:18:56 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:18:56.529+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:19:29 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:19:29.723+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:19:39 freenas k3s[16024]: time="2023-12-24T00:19:39+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 24 00:19:39 freenas k3s[16024]: time="2023-12-24T00:19:39+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 24 00:19:44 freenas k3s[16024]: time="2023-12-24T00:19:44+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 24 00:20:09 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:20:09.029+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:20:16 freenas k3s[16024]: I1224 00:20:16.097366   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 24 00:20:16 freenas k3s[16024]: E1224 00:20:16.102744   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-24 00:22:18.10271021 +0200 EET m=+118778.495320412 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 24 00:20:16 freenas k3s[16024]: I1224 00:20:16.198073   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 24 00:20:16 freenas k3s[16024]: E1224 00:20:16.203064   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-24 00:22:18.203038901 +0200 EET m=+118778.595649114 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 24 00:20:16 freenas k3s[16024]: I1224 00:20:16.400580   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 24 00:20:16 freenas k3s[16024]: E1224 00:20:16.404980   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-24 00:22:18.404952889 +0200 EET m=+118778.797563101 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 24 00:20:49 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:20:49.988+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:21:30 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:21:30.904+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:21:55 freenas k3s[16024]: time="2023-12-24T00:21:55+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 24 00:21:55 freenas k3s[16024]: time="2023-12-24T00:21:55+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 24 00:22:00 freenas k3s[16024]: time="2023-12-24T00:22:00+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 24 00:22:08 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:22:08.179+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:22:18 freenas k3s[16024]: I1224 00:22:18.114782   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 24 00:22:18 freenas k3s[16024]: E1224 00:22:18.119054   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-24 00:24:20.11902551 +0200 EET m=+118900.511635722 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 24 00:22:18 freenas k3s[16024]: I1224 00:22:18.216404   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 24 00:22:18 freenas k3s[16024]: E1224 00:22:18.219645   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-24 00:24:20.219631873 +0200 EET m=+118900.612242045 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 24 00:22:18 freenas k3s[16024]: I1224 00:22:18.419174   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 24 00:22:18 freenas k3s[16024]: E1224 00:22:18.423316   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-24 00:24:20.423287865 +0200 EET m=+118900.815898077 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 24 00:22:40 freenas k3s[16024]: time="2023-12-24T00:22:40+02:00" level=info msg="COMPACT compactRev=67582258 targetCompactRev=67582964 currentRev=67583964"
Dec 24 00:22:40 freenas k3s[16024]: time="2023-12-24T00:22:40+02:00" level=info msg="COMPACT deleted 706 rows from 706 revisions in 20.023961ms - compacted to 67582964/67583964"
Dec 24 00:22:44 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:22:44.654+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:23:17 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:23:17.699+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:24:01 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:24:01.799+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:24:10 freenas k3s[16024]: time="2023-12-24T00:24:10+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 24 00:24:10 freenas k3s[16024]: time="2023-12-24T00:24:10+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 24 00:24:15 freenas k3s[16024]: time="2023-12-24T00:24:15+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 24 00:24:20 freenas k3s[16024]: I1224 00:24:20.210186   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 24 00:24:20 freenas k3s[16024]: E1224 00:24:20.214621   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-24 00:26:22.214591533 +0200 EET m=+119022.607201745 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 24 00:24:20 freenas k3s[16024]: I1224 00:24:20.310860   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 24 00:24:20 freenas k3s[16024]: E1224 00:24:20.313883   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-24 00:26:22.313868041 +0200 EET m=+119022.706478223 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 24 00:24:20 freenas k3s[16024]: I1224 00:24:20.512639   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 24 00:24:20 freenas k3s[16024]: E1224 00:24:20.516738   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-24 00:26:22.516719883 +0200 EET m=+119022.909330065 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 24 00:24:32 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:24:32.359+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:25:06 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:25:06.601+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:25:42 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:25:42.145+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:26:22 freenas k3s[16024]: I1224 00:26:22.216718   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 24 00:26:22 freenas k3s[16024]: E1224 00:26:22.221677   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-24 00:28:24.22165139 +0200 EET m=+119144.614261593 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 24 00:26:22 freenas k3s[16024]: I1224 00:26:22.318133   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 24 00:26:22 freenas k3s[16024]: E1224 00:26:22.322818   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-24 00:28:24.322792978 +0200 EET m=+119144.715403190 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 24 00:26:22 freenas k3s[16024]: I1224 00:26:22.521021   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 24 00:26:22 freenas k3s[16024]: E1224 00:26:22.525219   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-24 00:28:24.525187265 +0200 EET m=+119144.917797467 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 24 00:26:24 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:26:24.483+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:26:25 freenas k3s[16024]: time="2023-12-24T00:26:25+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 24 00:26:25 freenas k3s[16024]: time="2023-12-24T00:26:25+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 24 00:26:30 freenas k3s[16024]: time="2023-12-24T00:26:30+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 24 00:26:57 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:26:57.758+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:27:40 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:27:40.095+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:27:40 freenas k3s[16024]: time="2023-12-24T00:27:40+02:00" level=info msg="COMPACT compactRev=67582964 targetCompactRev=67583667 currentRev=67584667"
Dec 24 00:27:40 freenas k3s[16024]: time="2023-12-24T00:27:40+02:00" level=info msg="COMPACT deleted 703 rows from 703 revisions in 22.041441ms - compacted to 67583667/67584667"
Dec 24 00:28:17 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:28:17.003+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:28:24 freenas k3s[16024]: I1224 00:28:24.241593   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 24 00:28:24 freenas k3s[16024]: E1224 00:28:24.247899   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-24 00:30:26.247836417 +0200 EET m=+119266.640446629 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 24 00:28:24 freenas k3s[16024]: I1224 00:28:24.343756   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 24 00:28:24 freenas k3s[16024]: E1224 00:28:24.348081   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-24 00:30:26.348051101 +0200 EET m=+119266.740661313 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 24 00:28:24 freenas k3s[16024]: I1224 00:28:24.546663   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 24 00:28:24 freenas k3s[16024]: E1224 00:28:24.551152   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-24 00:30:26.551121091 +0200 EET m=+119266.943731283 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 24 00:28:40 freenas k3s[16024]: time="2023-12-24T00:28:40+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 24 00:28:40 freenas k3s[16024]: time="2023-12-24T00:28:40+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 24 00:28:45 freenas k3s[16024]: time="2023-12-24T00:28:45+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 24 00:28:52 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:28:52.130+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:29:26 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:29:26.301+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:29:57 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:29:57.385+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:30:26 freenas k3s[16024]: I1224 00:30:26.308820   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 24 00:30:26 freenas k3s[16024]: E1224 00:30:26.312795   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-24 00:32:28.312756804 +0200 EET m=+119388.705367016 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 24 00:30:26 freenas k3s[16024]: I1224 00:30:26.409361   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 24 00:30:26 freenas k3s[16024]: E1224 00:30:26.413988   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-24 00:32:28.41395557 +0200 EET m=+119388.806565853 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 24 00:30:26 freenas k3s[16024]: I1224 00:30:26.612044   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 24 00:30:26 freenas k3s[16024]: E1224 00:30:26.616510   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-24 00:32:28.616484565 +0200 EET m=+119389.009094767 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 24 00:30:34 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:30:34.043+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:30:55 freenas k3s[16024]: time="2023-12-24T00:30:55+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 24 00:30:55 freenas k3s[16024]: time="2023-12-24T00:30:55+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 24 00:31:00 freenas k3s[16024]: time="2023-12-24T00:31:00+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 24 00:31:14 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:31:14.656+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:31:49 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:31:49.829+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:32:19 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:32:19.844+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:32:28 freenas k3s[16024]: I1224 00:32:28.346563   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 24 00:32:28 freenas k3s[16024]: E1224 00:32:28.351672   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-24 00:34:30.351643608 +0200 EET m=+119510.744253810 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 24 00:32:28 freenas k3s[16024]: I1224 00:32:28.448415   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 24 00:32:28 freenas k3s[16024]: E1224 00:32:28.452924   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-24 00:34:30.452898111 +0200 EET m=+119510.845508313 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 24 00:32:28 freenas k3s[16024]: I1224 00:32:28.651026   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 24 00:32:28 freenas k3s[16024]: E1224 00:32:28.655720   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-24 00:34:30.655686873 +0200 EET m=+119511.048297085 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 24 00:32:40 freenas k3s[16024]: time="2023-12-24T00:32:40+02:00" level=info msg="COMPACT compactRev=67583667 targetCompactRev=67584372 currentRev=67585372"
Dec 24 00:32:40 freenas k3s[16024]: time="2023-12-24T00:32:40+02:00" level=info msg="COMPACT deleted 705 rows from 705 revisions in 19.275188ms - compacted to 67584372/67585372"
Dec 24 00:33:04 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:33:04.841+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:33:10 freenas k3s[16024]: time="2023-12-24T00:33:10+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 24 00:33:10 freenas k3s[16024]: time="2023-12-24T00:33:10+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 24 00:33:15 freenas k3s[16024]: time="2023-12-24T00:33:15+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 24 00:33:47 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:33:47.312+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:34:24 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:34:24.566+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:34:30 freenas k3s[16024]: I1224 00:34:30.390264   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 24 00:34:30 freenas k3s[16024]: E1224 00:34:30.394994   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-24 00:36:32.394968998 +0200 EET m=+119632.787579210 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 24 00:34:30 freenas k3s[16024]: I1224 00:34:30.491832   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 24 00:34:30 freenas k3s[16024]: E1224 00:34:30.496243   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-24 00:36:32.496216628 +0200 EET m=+119632.888826840 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 24 00:34:30 freenas k3s[16024]: I1224 00:34:30.694270   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 24 00:34:30 freenas k3s[16024]: E1224 00:34:30.697949   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-24 00:36:32.697924461 +0200 EET m=+119633.090534643 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 24 00:35:04 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:35:04.041+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:35:26 freenas k3s[16024]: time="2023-12-24T00:35:26+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 24 00:35:26 freenas k3s[16024]: time="2023-12-24T00:35:26+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 24 00:35:31 freenas k3s[16024]: time="2023-12-24T00:35:31+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 24 00:35:37 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:35:37.802+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:36:11 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:36:11.551+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:36:32 freenas k3s[16024]: I1224 00:36:32.417999   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 24 00:36:32 freenas k3s[16024]: E1224 00:36:32.422612   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-24 00:38:34.422578771 +0200 EET m=+119754.815188973 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 24 00:36:32 freenas k3s[16024]: I1224 00:36:32.519178   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 24 00:36:32 freenas k3s[16024]: E1224 00:36:32.523647   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-24 00:38:34.523615898 +0200 EET m=+119754.916226121 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 24 00:36:32 freenas k3s[16024]: I1224 00:36:32.721039   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 24 00:36:32 freenas k3s[16024]: E1224 00:36:32.725584   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-24 00:38:34.725549774 +0200 EET m=+119755.118159976 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 24 00:36:48 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:36:48.942+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:37:25 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:37:25.545+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:37:40 freenas k3s[16024]: time="2023-12-24T00:37:40+02:00" level=info msg="COMPACT compactRev=67584372 targetCompactRev=67585074 currentRev=67586074"
Dec 24 00:37:40 freenas k3s[16024]: time="2023-12-24T00:37:40+02:00" level=info msg="COMPACT deleted 702 rows from 702 revisions in 11.897378ms - compacted to 67585074/67586074"
Dec 24 00:37:41 freenas k3s[16024]: time="2023-12-24T00:37:41+02:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 24 00:37:41 freenas k3s[16024]: time="2023-12-24T00:37:41+02:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.4.100:6443: connect: connection timed out"
Dec 24 00:37:46 freenas k3s[16024]: time="2023-12-24T00:37:46+02:00" level=info msg="Connecting to proxy" url="wss://192.168.4.100:6443/v1-k3s/connect"
Dec 24 00:38:03 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:38:03.139+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 24 00:38:34 freenas k3s[16024]: I1224 00:38:34.460183   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-8ab9801d-59d6-4639-ad8f-fff418009721\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721\") pod \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\" (UID: \"ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89\") "
Dec 24 00:38:34 freenas k3s[16024]: E1224 00:38:34.465124   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721 podName:ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89 nodeName:}" failed. No retries permitted until 2023-12-24 00:40:36.465093288 +0200 EET m=+119876.857703500 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-8ab9801d-59d6-4639-ad8f-fff418009721") pod "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89" (UID: "ec5a0fb5-f8c6-44f5-86ed-0294ed9c3d89") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-8ab9801d-59d6-4639-ad8f-fff418009721 err : zfsvolumes.zfs.openebs.io "pvc-8ab9801d-59d6-4639-ad8f-fff418009721" not found
Dec 24 00:38:34 freenas k3s[16024]: I1224 00:38:34.561128   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b\") pod \"0396a4b0-8122-4039-89a5-cee2aae7de11\" (UID: \"0396a4b0-8122-4039-89a5-cee2aae7de11\") "
Dec 24 00:38:34 freenas k3s[16024]: E1224 00:38:34.565571   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b podName:0396a4b0-8122-4039-89a5-cee2aae7de11 nodeName:}" failed. No retries permitted until 2023-12-24 00:40:36.565541715 +0200 EET m=+119876.958151897 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b") pod "0396a4b0-8122-4039-89a5-cee2aae7de11" (UID: "0396a4b0-8122-4039-89a5-cee2aae7de11") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b err : zfsvolumes.zfs.openebs.io "pvc-9992fe2c-0646-454f-a55c-e1bbb9744a0b" not found
Dec 24 00:38:34 freenas k3s[16024]: I1224 00:38:34.762879   16024 reconciler_common.go:169] "operationExecutor.UnmountVolume started for volume \"pvc-c0e89009-5812-4b80-a74b-7b426b833d11\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11\") pod \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\" (UID: \"73d57b59-32ea-4bc8-a2b3-fcb7427aae52\") "
Dec 24 00:38:34 freenas k3s[16024]: E1224 00:38:34.767162   16024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11 podName:73d57b59-32ea-4bc8-a2b3-fcb7427aae52 nodeName:}" failed. No retries permitted until 2023-12-24 00:40:36.767131114 +0200 EET m=+119877.159741326 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" (UniqueName: "kubernetes.io/csi/zfs.csi.openebs.io^pvc-c0e89009-5812-4b80-a74b-7b426b833d11") pod "73d57b59-32ea-4bc8-a2b3-fcb7427aae52" (UID: "73d57b59-32ea-4bc8-a2b3-fcb7427aae52") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = Internal desc = not able to get the ZFSVolume pvc-c0e89009-5812-4b80-a74b-7b426b833d11 err : zfsvolumes.zfs.openebs.io "pvc-c0e89009-5812-4b80-a74b-7b426b833d11" not found
Dec 24 00:38:38 freenas k3s[16024]: {"level":"warn","ts":"2023-12-24T00:38:38.640+0200","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c2c000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
