Scale 22.02.2 - Cannot Launch Docker Image

[Jorn]

Just upgraded to the new version 22.02.2

• Now I cannot "Launch Docker Image" anymore. The button is not active.
• Settings sub menus are not active either.

Anybody seeing the same ?

 

[sretalla]

For a few minutes on startup... seems to be while Kubernetes is starting up. After that, the text goes from Grey to white and the button can be clicked.

 

[Jorn]

I have been waiting since my first post a 8:12 AM but still not able to Launch Docker Image in Applications.

Also, the sub menus to the Settings menu are inactive.

I have now tried to roll back to 22.02.1 and everything is working fine. Then updated to 22.02.2 again and the error is back.

 

[Kris Moore]

Can you open a shell and run "systemctl status k3s" ? Sounds like kubernetes didn't start. I'd recommend going to Settings -> Unset Pool, and then Settings -> Set Pool again to see if a fresh init of k3s fixes it.

 

[Kris Moore]

Actually scratch that, looks like I have a repro case here. Will file a ticket and investigate.

 

[Kris Moore]

Ticket filed. We may end up needing to do a hotfix for this, stay tuned and thanks for the report:

https://jira.ixsystems.com/browse/NAS-116837

 

[Jorn]

Thanks for looking it to it

 

[Jorn]

Hi @Kris Moore

I don't know if it is related, but I'm also getting this error every time I boot to the 22.02.2:
If I roll back to 22.02.1 everything is fine.

 

[Kris Moore]

Not sure what that error is, can you file a ticket on https://jira.ixsystems.com so we can investigate? Please include a debug (System -> Advanced -> Save Debug)

Also, found out the "Launch Docker Image" button works fine if you go to the "Available Applications Tab" first, but we'll get it fixed either way.

 

[Jorn]

Not sure what that error is, can you file a ticket on https://jira.ixsystems.com so we can investigate? Please include a debug (System -> Advanced -> Save Debug)

Also, found out the "Launch Docker Image" button works fine if you go to the "Available Applications Tab" first, but we'll get it fixed either way.

Remember that it also the sub menus to the Settings menu are inactive.

 

[Kris Moore]

Yep, those sub-menus also work on that tab, so we think its same bug hitting them all. We'll get a fix soon.

 

[sesipod]

Hello,

I am having the same issue above - The Launch Docker Image is grayed out and I cannot click it.
The menu items under settings are not working - Choose Pool / Advanced Settings / Unset Pool.
I have tried chrome / edge / safari all with the same issue. I have also checked the service for k3s and its running. Restarted the service waited 5 min with no change.

When I check the service status this is in the log.
Jun 26 00:52:32 truenas k3s[43695]: {"level":"warn","ts":"2022-06-26T00:52:32.692-0700","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x>

TrueNAS Scale - TrueNAS-SCALE-22.02.2

Code:

k3s.service - Lightweight Kubernetes
     Loaded: loaded (/lib/systemd/system/k3s.service; disabled; vendor preset: disabled)
     Active: active (running) since Sun 2022-06-26 00:52:07 PDT; 32s ago
       Docs: https://k3s.io
    Process: 43693 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
    Process: 43694 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
   Main PID: 43695 (k3s-server)
      Tasks: 28
     Memory: 384.9M
     CGroup: /system.slice/k3s.service
             └─43695 /usr/local/bin/k3s server

Jun 26 00:52:16 truenas k3s[43695]: I0626 00:52:16.718687   43695 shared_informer.go:247] Caches are synced for ReplicationController
Jun 26 00:52:16 truenas k3s[43695]: I0626 00:52:16.779982   43695 shared_informer.go:247] Caches are synced for disruption
Jun 26 00:52:16 truenas k3s[43695]: I0626 00:52:16.780035   43695 disruption.go:371] Sending events to api server.
Jun 26 00:52:16 truenas k3s[43695]: I0626 00:52:16.782166   43695 shared_informer.go:247] Caches are synced for service account
Jun 26 00:52:16 truenas k3s[43695]: I0626 00:52:16.792528   43695 shared_informer.go:247] Caches are synced for resource quota
Jun 26 00:52:16 truenas k3s[43695]: I0626 00:52:16.835965   43695 shared_informer.go:247] Caches are synced for namespace
Jun 26 00:52:17 truenas k3s[43695]: I0626 00:52:17.209177   43695 shared_informer.go:247] Caches are synced for garbage collector
Jun 26 00:52:17 truenas k3s[43695]: I0626 00:52:17.290380   43695 shared_informer.go:247] Caches are synced for garbage collector
Jun 26 00:52:17 truenas k3s[43695]: I0626 00:52:17.290419   43695 garbagecollector.go:155] Garbage collector: all resource monitors have synced. Proceeding to collect garbage
Jun 26 00:52:32 truenas k3s[43695]: {"level":"warn","ts":"2022-06-26T00:52:32.692-0700","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x>
~

 

[sesipod]

https://jira.ixsystems.com/browse/NAS-116922

 

[lipelipe]

It seems like /etc/docker/ is gone also. No idea if that's related.

 

[lipelipe]

It seems like /etc/docker/ is gone also. No idea if that's related.

Please disregard this message.

 

[mtindall1]

Once the pool has been set, I try to launch the "Launch Docker Image" it just spins and the output from teh kubernetes command shows teh below. I assume this is part of the bug as well.

systemctl status k3s
● k3s.service - Lightweight Kubernetes
Loaded: loaded (/lib/systemd/system/k3s.service; disabled; vendor preset: disabled)
Active: active (running) since Mon 2022-07-18 10:58:12 CDT; 1h 5min ago
Docs: https://k3s.io
Process: 8978 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
Process: 8979 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
Main PID: 8980 (k3s-server)
Tasks: 43
Memory: 510.7M
CGroup: /system.slice/k3s.service
└─8980 /usr/local/bin/k3s server

Jul 18 12:02:58 ADCR2C11truenasC k3s[8980]: E0718 12:02:58.340250 8980 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" fo>
Jul 18 12:03:02 ADCR2C11truenasC k3s[8980]: {"level":"warn","ts":"2022-07-18T12:03:02.720-0500","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor>
Jul 18 12:03:06 ADCR2C11truenasC k3s[8980]: E0718 12:03:06.343444 8980 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" fo>
Jul 18 12:03:10 ADCR2C11truenasC k3s[8980]: E0718 12:03:10.339566 8980 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" fo>
Jul 18 12:03:23 ADCR2C11truenasC k3s[8980]: E0718 12:03:23.339989 8980 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" fo>
Jul 18 12:03:34 ADCR2C11truenasC k3s[8980]: E0718 12:03:34.338608 8980 pod_workers.go:949] "Error syncing pod, skipping" err="[failed to \"StartContainer\" fo>
Jul 18 12:03:36 ADCR2C11truenasC k3s[8980]: E0718 12:03:36.345406 8980 remote_image.go:216] "PullImage from image service failed" err="rpc error: code = Unkno>
Jul 18 12:03:36 ADCR2C11truenasC k3s[8980]: E0718 12:03:36.345458 8980 kuberuntime_image.go:51] "Failed to pull image" err="rpc error: code = Unknown desc = E>
Jul 18 12:03:36 ADCR2C11truenasC k3s[8980]: E0718 12:03:36.345580 8980 kuberuntime_manager.go:919] container &Container{Name:csi-resizer,Image:k8s.gcr.io/sig->
Jul 18 12:03:40 ADCR2C11truenasC k3s[8980]: {"level":"warn","ts":"2022-07-18T12:03:40.336-0

 

[watson1423]

i just faced that problem and found a "fix"

systemctl start k3s

in theory you can place this as one of the startup scripts and the problem should be solved.

 

[gregodfather]

I got the same problem with k3s not coming up after updating to TrueNAS-SCALE-22.02.2.1.
The systemd logs show me, that
a) connection to the k8s API is not allowed (also not working when running `k3s kubectl get nodes`
level=error msg="failed to get CA certs: Get \"https://127.0.0.1:6443/cacerts\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
b) and that might be the cause for a): ETCD seems to be unreachable
{"level":"warn","ts":"2022-08-06T08:50:40.736Z","logger":"etcd-client","caller":"v3@v3.5.1-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000d74540/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}

So I'd make the guess that either
a) etcd is not booting correctly
or
b) etcd data was somehow overwritten during the update... which would not be very cool

Any idea how to fix that?

 

[crkinard]

This is still an issue in TrueNAS-SCALE-22.02.3? Ugh...

 

[morganL]

This is still an issue in TrueNAS-SCALE-22.02.3? Ugh...

SCALE 22.02.4 just dropped today....another 100+ bug fixes. The quality is improving, but its not yet at same quality as TrueNAS CORE 13.0-U2.

Please be aware that some of these annoying issues are hardware dependent or related to a specific configs/sequences of events. Neither of these are easily caught in our automated testing. So we do depend on community feedback and helping to test the fixes.

Please try out SCALE 22.02.4.... if you have any issues, please report them here or start a new thread.