Apps not starting, k3s "error not a directory occurred when trying to remove the volumes dir"

[flammen]

Apps don't start and "Installed Applications" screen loads indefinitely.
Truenas Scale 22.02.4
Apps Nextcloud, Pihole, Plex, Tailscale were running fine before.
Pools are all online and no errors from scrub or smart.

Please help.

That is the systemctl status k3s output:

Code:

root@truenas[~]# systemctl status k3s
● k3s.service - Lightweight Kubernetes
     Loaded: loaded (/lib/systemd/system/k3s.service; disabled; vendor preset: disabled)
     Active: active (running) since Sat 2022-10-22 00:58:37 CEST; 8min ago
       Docs: https://k3s.io
    Process: 13714 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
    Process: 13716 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
   Main PID: 13718 (k3s-server)
      Tasks: 45
     Memory: 548.0M
     CGroup: /system.slice/k3s.service
             └─13718 /usr/local/bin/k3s server

Oct 22 01:06:50 truenas k3s[13718]: E1022 01:06:50.133653   13718 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"8e5c666c-4015-4e17-803d-0159c09da806\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2
Oct 22 01:06:52 truenas k3s[13718]: E1022 01:06:52.130996   13718 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"8e5c666c-4015-4e17-803d-0159c09da806\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2
Oct 22 01:06:54 truenas k3s[13718]: E1022 01:06:54.133384   13718 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"8e5c666c-4015-4e17-803d-0159c09da806\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2
Oct 22 01:06:56 truenas k3s[13718]: E1022 01:06:56.131735   13718 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"8e5c666c-4015-4e17-803d-0159c09da806\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2
Oct 22 01:06:58 truenas k3s[13718]: E1022 01:06:58.132199   13718 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"8e5c666c-4015-4e17-803d-0159c09da806\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2
Oct 22 01:07:00 truenas k3s[13718]: E1022 01:07:00.133130   13718 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"8e5c666c-4015-4e17-803d-0159c09da806\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2
Oct 22 01:07:02 truenas k3s[13718]: E1022 01:07:02.131328   13718 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"8e5c666c-4015-4e17-803d-0159c09da806\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2
Oct 22 01:07:04 truenas k3s[13718]: E1022 01:07:04.132872   13718 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"8e5c666c-4015-4e17-803d-0159c09da806\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2
Oct 22 01:07:06 truenas k3s[13718]: E1022 01:07:06.132907   13718 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"8e5c666c-4015-4e17-803d-0159c09da806\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2
Oct 22 01:07:08 truenas k3s[13718]: E1022 01:07:08.131391   13718 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"8e5c666c-4015-4e17-803d-0159c09da806\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=2

 

[radomirpolach]

I have been complaining about k3s errors for a while: https://ixsystems.atlassian.net/browse/NAS-118536?focusedCommentId=163193

I have never seen so many issues with k3s like on TrueNAS.

 

[radomirpolach]

Please register to IX Jira above and report your issues as well.

 

[flammen]

Thank you. I have reported the issue: https://ixsystems.atlassian.net/browse/NAS-118686
Do you have any clue how to solve it? I have my apps on a dedicated pool, Is there a way to roll back all apps (or the whole pool?) to a time before the issue occured? I am confused by the different snapshots created from ix-applications dataset and substructures.

 

[jasonmicron]

I updated the ticket with these steps. It helped me resolve the same issue. I don't know how it happened but I also linked what appears to be a similar issue in the comments on the ticket as well.

Log in with Atlassian account

Log in to Jira, Confluence, and all other Atlassian Cloud products here. Not an Atlassian user? Sign up for free.

ixsystems.atlassian.net