Upgrade to Scale 22.02.2 - Kubernetes won't start

[somewhatdamaged]

Upgrade my baremetal install of Scale to 22.02.2 and noticed that none of my apps will start, all stuck deploying. Output of "systemctl status k3s" below. Roll back to previous version and it's working. Something definitely not right with the latest version. Upgrading my second virtual instance of Scale did work however, but it's only running two apps

 

[stavros-k]

I'd suggest opening a ticket in https://jira.ixsystems.com/login.jsp
Devs usually don't look the forums

 

[netspear.io]

@somewhatdamaged

I'm having the same issue.

I opened a ticket on Jira,

https://jira.ixsystems.com/browse/NAS-116875

If you opened a ticket also, can you post a link to it so i can follow also?

 

[somewhatdamaged]

Sorry i've not done it yet as need to create an account etc. I'll do it tonight though. Just to add, did you not just roll back to the previous version? I did this and everything immediately came back up

 

[yottabit]

I'm having the same issue. After upgrading from 22.02.1 to 22.02.2, all of my apps are stuck in the "deploying" state.

I have unending log entries like this:

Code:

Jun 24 16:53:48 nas1 k3s[24030]: E0624 16:53:48.428110   24030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"0522f128-d7c4-4631-ab8d-cdc9d57cb711\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=9
Jun 24 16:53:50 nas1 k3s[24030]: E0624 16:53:50.434548   24030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"0522f128-d7c4-4631-ab8d-cdc9d57cb711\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=9
Jun 24 16:53:52 nas1 k3s[24030]: E0624 16:53:52.435371   24030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"0522f128-d7c4-4631-ab8d-cdc9d57cb711\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=9
Jun 24 16:53:54 nas1 k3s[24030]: E0624 16:53:54.432368   24030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"0522f128-d7c4-4631-ab8d-cdc9d57cb711\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=9
Jun 24 16:53:56 nas1 k3s[24030]: E0624 16:53:56.434500   24030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"0522f128-d7c4-4631-ab8d-cdc9d57cb711\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=9
Jun 24 16:53:58 nas1 k3s[24030]: E0624 16:53:58.423165   24030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"0522f128-d7c4-4631-ab8d-cdc9d57cb711\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=9
Jun 24 16:54:00 nas1 k3s[24030]: E0624 16:54:00.424679   24030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"0522f128-d7c4-4631-ab8d-cdc9d57cb711\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=9
Jun 24 16:54:02 nas1 k3s[24030]: E0624 16:54:02.434925   24030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"0522f128-d7c4-4631-ab8d-cdc9d57cb711\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=9
Jun 24 16:54:04 nas1 k3s[24030]: E0624 16:54:04.423083   24030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"0522f128-d7c4-4631-ab8d-cdc9d57cb711\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=9
Jun 24 16:54:06 nas1 k3s[24030]: E0624 16:54:06.435420   24030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"0522f128-d7c4-4631-ab8d-cdc9d57cb711\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=9
Jun 24 16:54:08 nas1 k3s[24030]: E0624 16:54:08.435254   24030 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"0522f128-d7c4-4631-ab8d-cdc9d57cb711\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=9

 

[somewhatdamaged]

I've also rolled back now, so can't upload the debug.... Loathe to upgrade and fail again, potentially risking data

 

[yottabit]

After one additional reboot after the upgrade, my apps/containers started. /shruggie

 

[somewhatdamaged]

OK so upgraded again just so i could raise a ticket to help out the guys above..and it's now working. FML

 

[yottabit]

I had something similar happen on the upgrade to 22.02. iirc the base system services didn't start, such as SSH and NFS, even though they were enabled. One extra reboot and they started fine.