I can't access my Apps in the TNS interface. I'm running the few commands that I can. I also can't generate a debug file because it seems to never finish and about an hour later I get some long python error.
Here are a few console errors that I get (70% of the time it returns the error), also not sure why all of the kube-system are terminating:
root@truenas[~]# k3s kubectl get pods -A
The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?
root@truenas[~]# k3s kubectl get pods -A
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get pods)
root@truenas[~]# k3s kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
ix-truenas-scale-chia truenas-scale-chia-5c59b65f79-j6wqw 1/1 Running 0 35h
ix-truenas-scale-plex truenas-scale-plex-648b9b44f9-wbdfg 1/1 Running 0 4h3m
kube-system openebs-zfs-controller-0 4/5 Terminating 22 4h9m
kube-system coredns-7448499f4d-m2542 1/1 Terminating 0 4h9m
kube-system openebs-zfs-node-mr76b 2/2 Terminating 0 4h9m
Part of the k3s.service status that I could copy off.
Dec 01 01:51:44 truenas.local k3s[100008]: time="2021-12-01T01:51:44.734007070+07:00" level=info msg="Stopped tunnel to 127.0.0.1:6443"
Dec 01 01:51:44 truenas.local k3s[100008]: time="2021-12-01T01:51:44.734035030+07:00" level=info msg="Proxy done" err="context canceled" url="wss://127.0.0.1:6443/v1-k3s/connect"
Dec 01 01:51:44 truenas.local k3s[100008]: time="2021-12-01T01:51:44.734039690+07:00" level=info msg="Connecting to proxy" url="wss://10.100.10.4:6443/v1-k3s/connect"
Dec 01 01:51:44 truenas.local k3s[100008]: time="2021-12-01T01:51:44.734127160+07:00" level=info msg="error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF"
Dec 01 01:51:44 truenas.local k3s[100008]: time="2021-12-01T01:51:44.739834459+07:00" level=info msg="Handling backend connection request [ix-truenas]"
Dec 01 01:51:44 truenas.local k3s[100008]: I1201 01:51:44.780662 100008 shared_informer.go:240] Waiting for caches to sync for tokens
Dec 01 01:51:44 truenas.local k3s[100008]: I1201 01:51:44.785476 100008 controllermanager.go:574] Started "podgc"
Dec 01 01:51:44 truenas.local k3s[100008]: I1201 01:51:44.785594 100008 gc_controller.go:89] Starting GC controller
Dec 01 01:51:44 truenas.local k3s[100008]: I1201 01:51:44.785614 100008 shared_informer.go:240] Waiting for caches to sync for GC
Dec 01 01:51:44 truenas.local k3s[100008]: I1201 01:51:44.811740 100008 controllermanager.go:574] Started "namespace"
Dec 01 01:51:44 truenas.local k3s[100008]: I1201 01:51:44.811815 100008 namespace_controller.go:200] Starting namespace controller
Dec 01 01:51:44 truenas.local k3s[100008]: I1201 01:51:44.811840 100008 shared_informer.go:240] Waiting for caches to sync for namespace
Dec 01 01:51:44 truenas.local k3s[100008]: I1201 01:51:44.816138 100008 controllermanager.go:574] Started "deployment"
Dec 01 01:51:44 truenas.local k3s[100008]: I1201 01:51:44.816229 100008 deployment_controller.go:153] "Starting controller" controller="deployment"
Dec 01 01:51:44 truenas.local k3s[100008]: I1201 01:51:44.816250 100008 shared_informer.go:240] Waiting for caches to sync for deployment
Dec 01 01:51:44 truenas.local k3s[100008]: I1201 01:51:44.820216 100008 node_ipam_controller.go:91] Sending events to api server.
Dec 01 01:51:44 truenas.local k3s[100008]: I1201 01:51:44.871625 100008 server.go:660] "--cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to /"
Dec 01 01:51:44 truenas.local k3s[100008]: I1201 01:51:44.871865 100008 container_manager_linux.go:291] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Dec 01 01:51:44 truenas.local k3s[100008]: I1201 01:51:44.871956 100008 container_manager_linux.go:296] "Creating Container Manager object based on Node Config" nodeConfig={RuntimeCgroupsName: SystemCgroupsName: KubeletCgrou>
Dec 01 01:51:44 truenas.local k3s[100008]: I1201 01:51:44.871985 100008 topology_manager.go:120] "Creating topology manager with policy per scope" topologyPolicyName="none" topologyScopeName="container"
Dec 01 01:51:44 truenas.local k3s[100008]: I1201 01:51:44.872001 100008 container_manager_linux.go:327] "Initializing Topology Manager" policy="none" scope="container"
Dec 01 01:51:44 truenas.local k3s[100008]: I1201 01:51:44.872013 100008 container_manager_linux.go:332] "Creating device plugin manager" devicePluginEnabled=true
Dec 01 01:51:44 truenas.local k3s[100008]: I1201 01:51:44.872091 100008 kubelet.go:310] "Using dockershim is deprecated, please consider using a full-fledged CRI implementation"
Dec 01 01:51:44 truenas.local k3s[100008]: I1201 01:51:44.872115 100008 client.go:78] "Connecting to docker on the dockerEndpoint" endpoint="unix:///var/run/docker.sock"
Dec 01 01:51:44 truenas.local k3s[100008]: I1201 01:51:44.872132 100008 client.go:97] "Start docker client with request timeout" timeout="2m0s"
Dec 01 01:51:44 truenas.local k3s[100008]: E1201 01:51:44.872363 100008 server.go:288] "Failed to run kubelet" err="failed to run Kubelet: failed to get docker version: Cannot connect to the Docker daemon at unix:///var/run/
Here are a few console errors that I get (70% of the time it returns the error), also not sure why all of the kube-system are terminating:
root@truenas[~]# k3s kubectl get pods -A
The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?
root@truenas[~]# k3s kubectl get pods -A
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get pods)
root@truenas[~]# k3s kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
ix-truenas-scale-chia truenas-scale-chia-5c59b65f79-j6wqw 1/1 Running 0 35h
ix-truenas-scale-plex truenas-scale-plex-648b9b44f9-wbdfg 1/1 Running 0 4h3m
kube-system openebs-zfs-controller-0 4/5 Terminating 22 4h9m
kube-system coredns-7448499f4d-m2542 1/1 Terminating 0 4h9m
kube-system openebs-zfs-node-mr76b 2/2 Terminating 0 4h9m
Part of the k3s.service status that I could copy off.
Dec 01 01:51:44 truenas.local k3s[100008]: time="2021-12-01T01:51:44.734007070+07:00" level=info msg="Stopped tunnel to 127.0.0.1:6443"
Dec 01 01:51:44 truenas.local k3s[100008]: time="2021-12-01T01:51:44.734035030+07:00" level=info msg="Proxy done" err="context canceled" url="wss://127.0.0.1:6443/v1-k3s/connect"
Dec 01 01:51:44 truenas.local k3s[100008]: time="2021-12-01T01:51:44.734039690+07:00" level=info msg="Connecting to proxy" url="wss://10.100.10.4:6443/v1-k3s/connect"
Dec 01 01:51:44 truenas.local k3s[100008]: time="2021-12-01T01:51:44.734127160+07:00" level=info msg="error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF"
Dec 01 01:51:44 truenas.local k3s[100008]: time="2021-12-01T01:51:44.739834459+07:00" level=info msg="Handling backend connection request [ix-truenas]"
Dec 01 01:51:44 truenas.local k3s[100008]: I1201 01:51:44.780662 100008 shared_informer.go:240] Waiting for caches to sync for tokens
Dec 01 01:51:44 truenas.local k3s[100008]: I1201 01:51:44.785476 100008 controllermanager.go:574] Started "podgc"
Dec 01 01:51:44 truenas.local k3s[100008]: I1201 01:51:44.785594 100008 gc_controller.go:89] Starting GC controller
Dec 01 01:51:44 truenas.local k3s[100008]: I1201 01:51:44.785614 100008 shared_informer.go:240] Waiting for caches to sync for GC
Dec 01 01:51:44 truenas.local k3s[100008]: I1201 01:51:44.811740 100008 controllermanager.go:574] Started "namespace"
Dec 01 01:51:44 truenas.local k3s[100008]: I1201 01:51:44.811815 100008 namespace_controller.go:200] Starting namespace controller
Dec 01 01:51:44 truenas.local k3s[100008]: I1201 01:51:44.811840 100008 shared_informer.go:240] Waiting for caches to sync for namespace
Dec 01 01:51:44 truenas.local k3s[100008]: I1201 01:51:44.816138 100008 controllermanager.go:574] Started "deployment"
Dec 01 01:51:44 truenas.local k3s[100008]: I1201 01:51:44.816229 100008 deployment_controller.go:153] "Starting controller" controller="deployment"
Dec 01 01:51:44 truenas.local k3s[100008]: I1201 01:51:44.816250 100008 shared_informer.go:240] Waiting for caches to sync for deployment
Dec 01 01:51:44 truenas.local k3s[100008]: I1201 01:51:44.820216 100008 node_ipam_controller.go:91] Sending events to api server.
Dec 01 01:51:44 truenas.local k3s[100008]: I1201 01:51:44.871625 100008 server.go:660] "--cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to /"
Dec 01 01:51:44 truenas.local k3s[100008]: I1201 01:51:44.871865 100008 container_manager_linux.go:291] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Dec 01 01:51:44 truenas.local k3s[100008]: I1201 01:51:44.871956 100008 container_manager_linux.go:296] "Creating Container Manager object based on Node Config" nodeConfig={RuntimeCgroupsName: SystemCgroupsName: KubeletCgrou>
Dec 01 01:51:44 truenas.local k3s[100008]: I1201 01:51:44.871985 100008 topology_manager.go:120] "Creating topology manager with policy per scope" topologyPolicyName="none" topologyScopeName="container"
Dec 01 01:51:44 truenas.local k3s[100008]: I1201 01:51:44.872001 100008 container_manager_linux.go:327] "Initializing Topology Manager" policy="none" scope="container"
Dec 01 01:51:44 truenas.local k3s[100008]: I1201 01:51:44.872013 100008 container_manager_linux.go:332] "Creating device plugin manager" devicePluginEnabled=true
Dec 01 01:51:44 truenas.local k3s[100008]: I1201 01:51:44.872091 100008 kubelet.go:310] "Using dockershim is deprecated, please consider using a full-fledged CRI implementation"
Dec 01 01:51:44 truenas.local k3s[100008]: I1201 01:51:44.872115 100008 client.go:78] "Connecting to docker on the dockerEndpoint" endpoint="unix:///var/run/docker.sock"
Dec 01 01:51:44 truenas.local k3s[100008]: I1201 01:51:44.872132 100008 client.go:97] "Start docker client with request timeout" timeout="2m0s"
Dec 01 01:51:44 truenas.local k3s[100008]: E1201 01:51:44.872363 100008 server.go:288] "Failed to run kubelet" err="failed to run Kubelet: failed to get docker version: Cannot connect to the Docker daemon at unix:///var/run/