All my apps are gone and I cannot install app after update to TrueNAS-SCALE-22.02.1 Unable to connect to kubernetes cluster

[HoshinoArika]

Before update to version 22.02.1 I got a alert

CRITICAL
Failed to start kubernetes cluster for Applications: [EFAULT] Unable to configure node: Cannot connect to host 127.0.0.1:6443 ssl:default [Connection reset by peer]
2022-05-05 12:26:45 (Asia/Shanghai)
Dismiss

I did not see it and I update the system and after the update, all apps were gone

when I try to install a new app (for example ddns-go from truechats )

I got this Error

Error: [EFAULT] Unable to connect to kubernetes cluster

Error: Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/job.py", line 412, in run
await self.future
File "/usr/lib/python3/dist-packages/middlewared/job.py", line 448, in __run_body
rv = await self.method(*([self] + args))
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1129, in nf
res = await f(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1261, in nf
return await func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/plugins/chart_releases_linux/chart_release.py", line 378, in do_create
await self.middleware.call('kubernetes.validate_k8s_setup')
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1308, in call
return await self._call(
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1257, in _call
return await methodobj(*prepared_call.args)
File "/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/update.py", line 335, in validate_k8s_setup
raise CallError(error)
middlewared.service_exception.CallError: [EFAULT] Unable to connect to kubernetes cluster

I tried to check status of k3s by running command "systemctl status k3s" and I got

systemctl status k3s
● k3s.service - Lightweight Kubernetes
Loaded: loaded (/lib/systemd/system/k3s.service; disabled; vendor preset: >
Active: active (running) since Thu 2022-05-05 12:56:57 CST; 4h 21min ago
Docs: https://k3s.io
Process: 6836 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status>
Process: 6842 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SU>
Main PID: 6849 (k3s-server)
Tasks: 31
Memory: 500.0M
CGroup: /system.slice/k3s.service
└─6849 /usr/local/bin/k3s server

May 05 17:18:40 BDStudioTrueNAS k3s[6849]: E0505 17:18:40.001808 6849 pod_wo>
May 05 17:18:40 BDStudioTrueNAS k3s[6849]: {"level":"warn","ts":"2022-05-05T17:>
May 05 17:18:40 BDStudioTrueNAS k3s[6849]: E0505 17:18:40.698873 6849 kubele>
May 05 17:18:41 BDStudioTrueNAS k3s[6849]: E0505 17:18:41.001811 6849 pod_wo>
May 05 17:18:41 BDStudioTrueNAS k3s[6849]: E0505 17:18:41.002940 6849 remote>
May 05 17:18:41 BDStudioTrueNAS k3s[6849]: E0505 17:18:41.002970 6849 kuberu>
May 05 17:18:41 BDStudioTrueNAS k3s[6849]: E0505 17:18:41.003933 6849 remote>
May 05 17:18:41 BDStudioTrueNAS k3s[6849]: E0505 17:18:41.003956 6849 kuberu>
May 05 17:18:41 BDStudioTrueNAS k3s[6849]: E0505 17:18:41.004004 6849 kubele>
May 05 17:18:41 BDStudioTrueNAS k3s[6849]: E0505 17:18:41.004028 6849 pod_wo>

My hardware setup

• Motherboard Asrock b550m pro4
• CPU AMD R3 3100
• RAM 32GB Hynix ECC DDR4 2666Hz
• Hard drives, quantity, model numbers, and RAID configuration, including boot drives
  • 2 * 120GB SSD (mirror system)
  • 2 * WD42EJRX in same pool with mirror mode
• Network cards : onboard

 

[HoshinoArika]

Update: I downgrade my TrueNAS Scale from 22.02.1 to 22.02.0.1 and the Apps are back

 

[HoshinoArika]

But I still don't know what caused the issue

 

[morganL]

But I still don't know what caused the issue

Not sure.... suggest you wait for anyone else seeing same issue.
If not and error is cleared, try again... and report-a-bug if it reoccurs.

 

[truecharts]

First off: your apps are not "gone" they are just not showing.

Try changing the kubernetes IP under advanced-settings in the Apps screen, wait a while, then change it back.

 

[smckee]

I have had the same issue show up when I updated to 22.02.1. I didn't immediately notice the issue after rebooting my two TrueNAS nodes but when I try to get to "Refresh All" under Available Applications I get:

Refreshing​

Error: 403, message='Forbidden', url=URL('http://cache3.local:3128')

This is the response from our Squid proxy (cache3.local) because it was asked to connect to: 127.0.0.1:6443

Our two servers have ONLY private addresses assigned and require a proxy to get to TrueNAS for updates or apps. This was working fine prior to the upgrade to 22.02.1 I have not tried downgrading, since I wanted other fixes in 22.02.1 I can provide additional details if they would be helpful. - Shawn

 

[truecharts]

I have had the same issue show up when I updated to 22.02.1. I didn't immediately notice the issue after rebooting my two TrueNAS nodes but when I try to get to "Refresh All" under Available Applications I get:

Refreshing​

Error: 403, message='Forbidden', url=URL('http://cache3.local:3128')

This is the response from our Squid proxy (cache3.local) because it was asked to connect to: 127.0.0.1:6443

Our two servers have ONLY private addresses assigned and require a proxy to get to TrueNAS for updates or apps. This was working fine prior to the upgrade to 22.02.1 I have not tried downgrading, since I wanted other fixes in 22.02.1 I can provide additional details if they would be helpful. - Shawn

It might be best to file an actual bugreport for this with iX Systems.

 

[sususu]

I have had the same issue show up when I updated to 22.02.1. I deleted the HTTP proxy settings. It is fixed.

 

[smckee]

I can confirm that the Applications show up if I remove the HTTP proxy settings, BUT I then have no access to the catalog updates or to new versions of TrueNAS since my host has only private IP addresses. So while removing the http proxy settings allows the applications to show up, it is not a fix, at least for me. This information should help the TrueNAS team fix the issue I hope.

 

[joace]

This is the same issue I also had after upgrading to 22.02.1, the system sits behind a corp proxy, once removed the proxy setting from UI, the installed applications show up, if not, an endless spinning cycle on "Installed Applicaitons" page. Eventually a failure message logged.

Failed to sync OFFICIAL catalog: [EFAULT] 503, message='Service Unavailable', url=URL('http://xxxxxx:8080')​

2022-05-14 16:15:37 (America/Chicago)

Without proxy, system or catelog update are not working, seems a ship stopper issue.

 

[Sveken]

Same issue here, we are also behind a HTTP proxy and the application page just keeps spinning

 

[HoshinoArika]

Update: I upgrade from 22.02.1 to 22.02.2.1 and everything is working properly
P.S. I made no change to the apps, system and settings (and other things) compare to my last try on May 5 2022.