Problem: Kubernetes service is not running - TrueNAS Scale

[lonemuffin]

I recently updated my TrueNAS Scale system to version 22.02.1 and now my apps don't seem to be running and are not installable. Every time I try to install and start apps like netdata or plex, it says:

Error
[EFAULT] Kubernetes service not running.

Then lists the following info:

Code:

Error: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 412, in run
    await self.future
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 448, in __run_body
    rv = await self.method(*([self] + args))
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1129, in nf
    res = await f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1261, in nf
    return await func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/chart_releases_linux/chart_release.py", line 378, in do_create
    await self.middleware.call('kubernetes.validate_k8s_setup')
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1308, in call
    return await self._call(
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1257, in _call
    return await methodobj(*prepared_call.args)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/update.py", line 335, in validate_k8s_setup
    raise CallError(error)
middlewared.service_exception.CallError: [EFAULT] Kubernetes service is not running.

I'm not sure if it matters or not but a couple months ago I unmounted then mounted my pool to rename it, which was successful, however I don't think I noticed any problems with the apps. Not sure if that could be the cause as well or not. Any help would be greatly appreciated.

 

[morganL]

I recently updated my TrueNAS Scale system to version 22.02.1 and now my apps don't seem to be running and are not installable. Every time I try to install and start apps like netdata or plex, it says:

Error
[EFAULT] Kubernetes service not running.

Then lists the following info:

Code:

Error: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 412, in run
    await self.future
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 448, in __run_body
    rv = await self.method(*([self] + args))
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1129, in nf
    res = await f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1261, in nf
    return await func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/chart_releases_linux/chart_release.py", line 378, in do_create
    await self.middleware.call('kubernetes.validate_k8s_setup')
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1308, in call
    return await self._call(
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1257, in _call
    return await methodobj(*prepared_call.args)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/update.py", line 335, in validate_k8s_setup
    raise CallError(error)
middlewared.service_exception.CallError: [EFAULT] Kubernetes service is not running.

I'm not sure if it matters or not but a couple months ago I unmounted then mounted my pool to rename it, which was successful, however I don't think I noticed any problems with the apps. Not sure if that could be the cause as well or not. Any help would be greatly appreciated.

Does the issue survive a reboot?

If so, I'd suggest collect the debugs and "report-an-issue".

Rollin back to the previous release would determine that its an issue with the new version.... or a general system problem.

 

[jbarranco]

I am having this same issue - was there any resolution on this?

 

[Bonnie Follweiler]

I am having this same issue - was there any resolution on this?

Are you on TrueNAS SCALE version 22.02.3 or the latest Nightly?
Please report this issue by creating a support ticket. There is a link in the UI.
Log into your TrueNAS, navigate to the System Settings menu, and then select General. Click File Ticket. Click Login to JIRA. Login to Jira or create a Jira account if you don’t have one yet.
Fill out the File Ticket information and click the Attach Debug checkbox. Attach any screenshots you have. If possible, in the ticket's Description area, provide the previous steps that lead to the error message (to assist in diagnosing/reproduction). Click Submit.

 

[asw2012]

Hello, it seems I am having the same problem on 22.03.3 - What should I do here? Rebooting the system, does not help.

 

[jbarranco]

I submitted ticket available here:

Jira

ixsystems.atlassian.net

My systemctl status k3s command also returns the same as asw2012's

 

[asw2012]

I just submitted ticket here:

Jira

ixsystems.atlassian.net

 

[JarekLB]

I had this issue and was able to fix it by unsetting the pool then reassigning it.

 

[jbarranco]

Just as follow up here, I also had to un-set the pool and then reassign it. That seems to be the workaround here.

 

[asw2012]

It seems after the latest update/patch TrueNAS-SCALE-22.02.4, the problem, at least on my installation, has been fixed. All Apps are OK.

 

[sophware]

It seems after the latest update/patch TrueNAS-SCALE-22.02.4, the problem, at least on my installation, has been fixed. All Apps are OK.

22.02.4 didn't prevent the issue for me. Had to unset and re-set.

 

[morganL]

If anyone has tried Bluefin, we'd like to know if the issue is resolved. If not, please report a bug and help us diagnose.

 

[skittlebrau]

If anyone has tried Bluefin, we'd like to know if the issue is resolved. If not, please report a bug and help us diagnose.

I had the same problem as OP, but unsetting the pool, rebooting and then setting/choosing the pool in Apps resolved the problem.

 

[picklefish]

Apps > Settings > Unset Pool
Restart System
Select pool

Fixed it for me as well

 

[Ryan Haver]

@morganL I'm running Bluefin and can't seem to get past this issue with the workaround that has been working for others.

Additional steps that were completed on my end in an effort to work around the issue:

1. Delete docker containers
2. Restart System
3. Delete the ix-applications dataset
4. Go back to Applications and choose the original Pool
5. Attempt to deploy an App but receive the same error as the original poster in this thread

Update 1: I've added a v4 interface and gateway to the advanced settings for Kubernetes. I've deleted the original pool, recreated it, and attempted to install an App again. I'm getting the following error now:

Failed to start kubernetes cluster for Applications: [EFAULT] Failed to configure PV/PVCs support: Cannot connect to host 127.0.0.1:6443 ssl:default [Connect call failed ('127.0.0.1', 6443)]​

Currently looking into the new error and it looks like this may be a game of having more patience per this thread: https://www.truenas.com/community/threads/k3s-not-starting-after-upgrade.106130/

Update 2: No idea what resolved this, but after a reboot, I deleted the ix-applications dataset again...got busy and had to walk away for some time. When I got back to things, I added/set the pool I wanted to use, and Kubernetes came up fine without issues. Now I can get to the deploying stage for apps. I'll update again if any dramatic happens :)