Pods cannot deploy nor deleted

departy

Dabbler
Joined
Oct 24, 2021
Messages
17
Today I noticed that I have container, which cannot be deployed or deleted. I noticed this few days ago after Rebooting the system (from menu not power outage)
Today I rebooted the system once again today and its the same. I tried to delete the pod and its stuck there.
I have not observed the issue with any other container/pod

TrueNAS Scale version: TrueNAS-SCALE-22.02-RC.1-2

Can someone help me fix the issue!

1638866916664.png


2021-12-07 10:28:50 Stopping container photoprism
2021-12-07 10:28:50 Deleted pod: photoprism-6cff978d76-zssjw
2021-12-07 10:28:50 Scaled down replica set photoprism-6cff978d76 to 0
0/1 nodes are available: 1 node(s) had taint {ix-svc-start: }, that the pod didn't tolerate.
2021-12-07 10:01:30 Startup probe failed: dial tcp 172.16.1.34:2342: connect: connection refused
2021-12-07 10:01:20 Started container photoprism
2021-12-07 10:01:19 Created container photoprism
2021-12-07 10:01:12 Container image "docker.io/photoprism/photoprism:20211018@sha256:f61fb623187faa33d1e3f4d86e384cdf401442d4ba6bec96897f42fa4fbb84b4" already present on machine
2021-12-07 10:01:07 Started container autopermissions
2021-12-07 10:01:07 Created container autopermissions
2021-12-07 10:01:00 Container image "ghcr.io/truecharts/alpine:v3.14.2@sha256:a537d87e3d22c5b3f695218ca1fb5a031fb0ccafa0e3e256ef45188ab0575be6" already present on machine
2021-12-07 10:01:00
Add eth0 [172.16.1.34/16] from ix-net
2021-12-07 10:00:43 Unable to attach or mount volumes: unmounted volumes=[varlogs list-0 kube-api-access-k5wbk shared storage temp], unattached volumes=[varlogs list-0 kube-api-access-k5wbk shared storage temp]: timed out waiting for the condition
2021-12-07 9:59:48 Started container lb-port-2342
2021-12-07 9:58:44 MountVolume.MountDevice failed for volume "pvc-633edd59-87fb-4ad1-940c-cd22b77976c5" : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers
2021-12-07 9:59:48 Created container lb-port-2342
2021-12-07 9:59:37 Container image "rancher/klipper-lb:v0.1.2" already present on machine
2021-12-07 9:59:37 Add eth0 [172.16.1.24/16] from ix-net
Successfully assigned ix-photoprism/photoprism-6cff978d76-zssjw to ix-truenas
Successfully assigned ix-photoprism/svclb-photoprism-6rt6c to ix-truenas
2021-12-07 9:59:22 Created pod: svclb-photoprism-6rt6c
2021-12-07 9:59:04 Error: failed to start container "lb-port-2342": Error response from daemon: cannot join network of a non running container: 9c48fb2fbfdc838558675da4dddf6e67856e147ca8e1e56ec2c6a3ec2759b491
2021-12-07 9:59:02 Created container lb-port-2342
2021-12-07 9:58:47 Container image "rancher/klipper-lb:v0.1.2" already present on machine
2021-12-07 9:58:39 Pod sandbox changed, it will be killed and re-created.
2021-12-07 9:58:29 network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
2021-12-07 9:58:29 network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
2021-12-07 9:58:45 Marking for deletion Pod ix-photoprism/svclb-photoprism-24f6s
2021-12-07 9:58:45 Marking for deletion Pod ix-photoprism/photoprism-mariadb-0
2021-12-07 9:58:45 Cancelling deletion of Pod ix-photoprism/svclb-photoprism-24f6s
2021-12-07 9:58:45 Cancelling deletion of Pod ix-photoprism/photoprism-mariadb-0
2021-12-07 9:58:45 Marking for deletion Pod ix-photoprism/photoprism-6cff978d76-qd76r
2021-12-07 9:58:45 Cancelling deletion of Pod ix-photoprism/photoprism-6cff978d76-qd76r
2021-12-07 9:58:46 Created pod: photoprism-6cff978d76-zssjw
2021-12-07 9:58:45 Deleted pod: svclb-photoprism-24f6s
2021-12-07 9:58:45 AttachVolume.FindAttachablePluginBySpec failed for volume "pvc-633edd59-87fb-4ad1-940c-cd22b77976c5"
2021-12-07 9:58:46 Add eth0 [172.16.1.5/16] from ix-net
0/1 nodes are available: 1 node(s) had taint {ix-svc-start: }, that the pod didn't tolerate.
 

truecharts

Guru
Joined
Aug 19, 2021
Messages
787
Normally we would say: "Get support at our Discord".
But in this case, the issue seems to be with iX's part in things.

"0/1 nodes are available: 1 node(s) had taint {ix-svc-start: }, that the pod didn't tolerate."
Is clearly a bug, so please make a bugreport at the iX Systems Jira and be sure to attach a debug log, that way iX can get this all fixed for everyone
 

waqarahmed

iXsystems
iXsystems
Joined
Aug 28, 2019
Messages
136
@departy can you please DM me a debug of your system at waqar at the rate of ixsystems.com ? I can have a peek as to what might be going on. Thanks!
 
Top