Applications are not running

newbacai · May 29, 2022

Hi,

After an unexpected power failure yesterday, all containers failed and the Applicaiton pages showed: Applications are not running, and the reboot of TrueNAS didn't work.

I found logs in /var/log/k3s_daemon.log:

May 30 10:18:30 tn k3s[25407]: time="2022-05-30T10:18:30+08:00" level=info msg="Starting k3s v1.23.5+k3s-937d546a-dirty (937d546a)"
May 30 10:18:30 tn k3s[25407]: time="2022-05-30T10:18:30+08:00" level=info msg="Configuring sqlite3 database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
May 30 10:18:30 tn k3s[25407]: time="2022-05-30T10:18:30+08:00" level=info msg="Configuring database table schema and indexes, this may take a moment..."
May 30 10:18:30 tn k3s[25407]: time="2022-05-30T10:18:30+08:00" level=info msg="Database tables and indexes are up to date"
May 30 10:18:30 tn k3s[25407]: time="2022-05-30T10:18:30+08:00" level=info msg="Kine available at unix://kine.sock"
May 30 10:18:30 tn k3s[25407]: time="2022-05-30T10:18:30+08:00" level=fatal msg="starting kubernetes: preparing server: failed to normalize token; must be in format K10<CA-HASH>::<USERNAME>:<PASSWORD> or <PASSWORD>"

I guess this is the point but I have no idea about how to fix it:

May 30 10:18:30 tn k3s[25407]: time="2022-05-30T10:18:30+08:00" level=fatal msg="starting kubernetes: preparing server: failed to normalize token; must be in format K10<CA-HASH>::<USERNAME>:<PASSWORD> or <PASSWORD>"

Thanks!

DaSnipe · May 30, 2022

Sometimes after an unexpected reset the Kubernetes network gets reset, make sure in Settings -> Advanced Settings has your info filled in

If that's setup, I'd file a bug report with iX on their JIRA

newbacai · May 31, 2022

@DaSnipe Thanks, the network settings was indeed blank before. But it still failed after I filled in and reboot the system.

DaSnipe · May 31, 2022

newbacai said:
@DaSnipe Thanks, the network settings was indeed blank before. But it still failed after I filled in and reboot the system.

Yep, try to reboot again but I'd file a bug report

newbacai · Jun 1, 2022

DaSnipe said:
Yep, try to reboot again but I'd file a bug report

I've rebooted many times. Not work.
Do you need me to provide some information for your report?

stavros-k · Jun 1, 2022

newbacai said:
I've rebooted many times. Not work.
Do you need me to provide some information for your report?

I think he meant, If I were you. I'd go file a bur report, now that he will report it for you :)

You can go to System > General > File Ticket.

or go to https://jira.ixsystems.com/

Just make sure you attach your debug file.
System > Advanced > Save Debug

newbacai · Jun 1, 2022

stavros-k said:
I think he meant, If I were you. I'd go file a bur report, now that he will report it for you :)

OK, I'll report it later. Thanks.

manu_hna · Jun 16, 2022

Hi! Any luck with this?

I have had a similar problem: I had a power outage and when restarting my truenas scale, no Apps are shown.

I have tried resetting kubernetes settings and rebooting and it did nothing.
Also, I have checked and my machine has network connectivity and DNS resolution on point.

Some logs that i managed to get:

Code:

root@truenas[~]# journalctl | grep k3s | tail -n 8
Jun 16 13:58:12 truenas k3s[41303]: time="2022-06-16T13:58:12+02:00" level=info msg="Starting k3s v1.23.5+k3s-937d546a-dirty (937d546a)"
Jun 16 13:58:12 truenas k3s[41303]: time="2022-06-16T13:58:12+02:00" level=info msg="Configuring sqlite3 database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
Jun 16 13:58:12 truenas k3s[41303]: time="2022-06-16T13:58:12+02:00" level=info msg="Configuring database table schema and indexes, this may take a moment..."
Jun 16 13:58:12 truenas k3s[41303]: time="2022-06-16T13:58:12+02:00" level=info msg="Database tables and indexes are up to date"
Jun 16 13:58:12 truenas k3s[41303]: time="2022-06-16T13:58:12+02:00" level=info msg="Kine available at unix://kine.sock"
Jun 16 13:58:12 truenas k3s[41303]: time="2022-06-16T13:58:12+02:00" level=fatal msg="starting kubernetes: preparing server: failed to normalize token; must be in format K10<CA-HASH>::<USERNAME>:<PASSWORD> or <PASSWORD>"
Jun 16 13:58:12 truenas systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE
Jun 16 13:58:12 truenas systemd[1]: k3s.service: Failed with result 'exit-code'.

This logs repeat themselves a lot, as k3s is not capable of starting.

Service status

Code:

root@truenas[~]# sudo systemctl status k3s
● k3s.service - Lightweight Kubernetes
     Loaded: loaded (/lib/systemd/system/k3s.service; disabled; vendor preset: disabled)
     Active: activating (auto-restart) (Result: exit-code) since Thu 2022-06-16 13:57:08 CEST; 4s ago
       Docs: https://k3s.io
    Process: 40291 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
    Process: 40292 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
    Process: 40293 ExecStart=/usr/local/bin/k3s server --flannel-backend=none --disable=traefik,metrics-server,local-storage --disable-kube-proxy --disable-network-policy --disable-cloud-controller --node-name=ix-truenas --docker (code=exited, status=1/FAILURE)
   Main PID: 40293 (code=exited, status=1/FAILURE)

Maybe this helps but this is what my k3s check-config yields:

Code:

root@truenas[~]# k3s check-config

Verifying binaries in /mnt/pool-1/ix-applications/k3s/data/11c442af7d27af92e276ebd499b6ebab41fa574b9c6fafd965fe38cffa10e51d/bin:
- sha256sum: good
- links: good

System:
- /usr/sbin iptables v1.8.7 (nf_tables): ok
- swap: should be disabled
- routes: ok

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

modprobe: FATAL: Module configs not found in directory /lib/modules/5.10.109+truenas
info: reading kernel config from /boot/config-5.10.109+truenas ...

Generally Necessary:
- cgroup hierarchy: cgroups Hybrid mounted, cpuset|memory controllers status: good
- /usr/sbin/apparmor_parser
apparmor: enabled and tools installed
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled (as module)
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
- CONFIG_IP_NF_NAT: enabled (as module)
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_POSIX_MQUEUE: enabled

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: enabled
- CONFIG_NET_CLS_CGROUP: enabled (as module)
- CONFIG_CGROUP_NET_PRIO: enabled
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: missing
- CONFIG_IP_NF_TARGET_REDIRECT: enabled (as module)
- CONFIG_IP_SET: enabled (as module)
- CONFIG_IP_VS: enabled (as module)
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_PROTO_TCP: enabled
- CONFIG_IP_VS_PROTO_UDP: enabled
- CONFIG_IP_VS_RR: enabled (as module)
- CONFIG_EXT4_FS: enabled (as module)
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: enabled (as module)
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled (as module)
      - CONFIG_CRYPTO_GCM: enabled (as module)
      - CONFIG_CRYPTO_SEQIV: enabled (as module)
      - CONFIG_CRYPTO_GHASH: enabled (as module)
      - CONFIG_XFRM: enabled
      - CONFIG_XFRM_USER: enabled (as module)
      - CONFIG_XFRM_ALGO: enabled (as module)
      - CONFIG_INET_ESP: enabled (as module)
      - CONFIG_INET_XFRM_MODE_TRANSPORT: missing
- Storage Drivers:
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled (as module)

STATUS: pass

Is there any way I can fix this without having to reinstall?

Thanks!

rubenverhoef · Jun 23, 2022

i had the same issue, after upgrading to TrueNAS Scale 20.02.2

I have fixed the issue by deleting/renaming the following file:

Code:

$app_pool/ix-applications/k3s/server/token

For me this was at the following location:

Code:

/mnt/ssd/ix-applications/k3s/server/token

Hope this will help you with the issue.

netspear.io · Jun 23, 2022

@rubenverhoef

I'm having the same issue.
I tried that, but it didn't work for me. Would you have any other suggestions as to what might cause this issue?

rubenverhoef · Jun 23, 2022

netspear.io said:
@rubenverhoef

I'm having the same issue.
I tried that, but it didn't work for me. Would you have any other suggestions as to what might cause this issue?

The problem i saw in the log was:

Code:

msg="starting kubernetes: preparing server: failed to normalize token; must be in format K10<CA-HASH>::<USERNAME>:<PASSWORD> or <PASSWORD>"

After some searching online i found this post: https://github.com/k3s-io/k3s/issues/5025#issuecomment-1071465155

Removing that file fixed my issue.
I also set the kubernetes settings correctly, they were blank. See post above.

Xenouniv3rse · Aug 10, 2022

I am having the same issue and cant seem to solve it. Has anyone found a solution for this?

Xenouniv3rse · Aug 10, 2022

rubenverhoef said:
i had the same issue, after upgrading to TrueNAS Scale 20.02.2

I have fixed the issue by deleting/renaming the following file:

Code:
$app_pool/ix-applications/k3s/server/token

For me this was at the following location:

Code:
/mnt/ssd/ix-applications/k3s/server/token

Hope this will help you with the issue.

You are a lifesaver!!! THANK YOU

mjflower · Aug 15, 2022

I had the exact same issue this last couple days and deleting the token file solved the problem. Thanks heaps.

dazzywazzy · Sep 20, 2022

rubenverhoef said:
i had the same issue, after upgrading to TrueNAS Scale 20.02.2 I have fixed the issue by deleting/renaming the following file:
Code:
$app_pool/ix-applications/k3s/server/token
For me this was at the following location:
Code:
/mnt/ssd/ix-applications/k3s/server/token
Hope this will help you with the issue.

Created an account just to say thank you for this, been tearing my hair out trying to resolve, and this little fix did it.

Amazing

stavros-k · Sep 21, 2022

dazzywazzy said:
Created an account just to say thank you for this, been tearing my hair out trying to resolve, and this little fix did it.

Amazing

Creating an account to iX Systems Jira, and reporting it, would have been a lot better time spent :)
Without a proper fix is just a matter of time to happen to someone.

Just my 2 cents

ummario · Nov 21, 2022

rubenverhoef said:
i had the same issue, after upgrading to TrueNAS Scale 20.02.2

I have fixed the issue by deleting/renaming the following file:

Code:
$app_pool/ix-applications/k3s/server/token

For me this was at the following location:

Code:
/mnt/ssd/ix-applications/k3s/server/token

Hope this will help you with the issue.

I have the same problem. I upgraded to Blufin RC (big mistake I would say now) but had to revert to Angelfish since i could not mount my SMB shares.

Now that i'm back to Angelfish my apps stopped working. Updated the Kubernetes network settings as mentioned by @DaSnipe but still can't make it work.

Sorry @rubenverhoef I'm totally newbie to Truenas. How do I find and delete that file you mentioned. Can you point me to any guide?

Thanks a lot in advance

kbax · Jan 7, 2023

rubenverhoef said:
i had the same issue, after upgrading to TrueNAS Scale 20.02.2

I have fixed the issue by deleting/renaming the following file:

Code:
$app_pool/ix-applications/k3s/server/token

For me this was at the following location:

Code:
/mnt/ssd/ix-applications/k3s/server/token

Hope this will help you with the issue.

Thank you. Worked for me.

snorian · Jan 14, 2023

Hi, I seem to be experiencing this same issue and I have fixed the network settings for Kubernetes, but can't delete the token file. How am I supposed to do that. I don't seem to be allowed to create a share for the ix-applications dataset and I also can't seem to find it using the shell in System Settings.
When I use ls to see the contents in shell, I only get the following: dead.letter samba syslog tdb
Searching through these options doesn't seem to provide a path to my pool.

Nubus · Feb 8, 2023

Check your date and times in the bios

Important Announcement for the TrueNAS Community.

Applications are not running

Dabbler

Explorer

Dabbler

Explorer

Dabbler

Patron

Dabbler

Cadet

Cadet

Dabbler

Cadet

Dabbler

Dabbler

Dabbler

Cadet

Patron

Cadet

Dabbler

Cadet

Cadet

Similar threads