Cobia -23.10.0.1: App server is not initialized yet, Failed to start kubernetes cluster for Applications

[toury]

TrueNAS-SCALE-23.10.0.1
Intel(R) Core(TM) i3-9100T
32 GB ECC RAM

Hi I'm new to Truenas, when I upgrade to Cobia-23.10.0.1, the kubernetes cluster keeps failed, it doesn't happened on Bluefin and Cobia-23.10.0-RC.

---issue----
Error In Apps Service
Failed to start kubernetes cluster for Applications: server is not initialized yet

when the error occurs, all apps will stuck in deploying.

---my solution---
Reset current pool in the app setting , that is unset the pool then choose the pool could temporary fix the problem, however the error comes back after few hours.
Since reset the pool could temporary fix the issue, I assume my current system setting and configuration should be ok?


Does anyone encounter the same problem?

 

[MrssM]

TrueNAS-SCALE-23.10.0.1
Intel(R) Core(TM) i3-9100T
32 GB ECC RAM

Hi I'm new to Truenas, when I upgrade to Cobia-23.10.0.1, the kubernetes cluster keeps failed, it doesn't happened on Bluefin and Cobia-23.10.0-RC.

---issue----
Error In Apps Service
Failed to start kubernetes cluster for Applications: server is not initialized yet

when the error occurs, all apps will stuck in deploying.

---my solution---
Reset current pool in the app setting , that is unset the pool then choose the pool could temporary fix the problem, however the error comes back after few hours.
Since reset the pool could temporary fix the issue, I assume my current system setting and configuration should be ok?


Does anyone encounter the same problem?

I also encountered it, I encountered it after DEGRADED restarted

 

[wili4m]

same boat
i already set the bios time to UTC.
some time this issue show up, do restart and shutdown it gone, but after couple day it show again after turn on the sever.
try upgrade to new version and restart network to default, already set bios time to UTC, still show same issue

i must every day to check the server it very annoying

waiting good solution for this issue, many thanks.

 

[toury]

same boat
i already set the bios time to UTC.
some time this issue show up, do restart and shutdown it gone, but after couple day it show again after turn on the sever.
try upgrade to new version and restart network to default, already set bios time to UTC, still show same issue

i must every day to check the server it very annoying

waiting good solution for this issue, many thanks.





View attachment 73238

I may have found out why this error occurred, it may caused by "Kubernetes cluster run out of memory":

I noticed that my RAM usage usually get high to 99% , which only around 0.1GB memory are available. so I tried to stop a few apps which cost a lot of memory space in usual, to test if it can solve the problem. the result is it does solve the problem, currently my available RAM space is around 1GB and the error disappeared!

Summary:
How does the error happen?
- not enough available RAM could cause this error to happen, seems TrueNAS SCALE Cobia cannot handle such situation during high RAM usage.
How to solve it?
- in my case, stop high RAM usage apps & containers to free up used memory, keeping 1GB available memory space should be safe I assume.

 

[ABain]

Please could you report a bug link at the top of the forum pages), you will be provided a link on the bug to a private upload location to supply a debug, we'll need this to investigate.

 

[toury]

Please could you report a bug link at the top of the forum pages), you will be provided a link on the bug to a private upload location to supply a debug, we'll need this to investigate.

As you required, I have submitted a report in Jira, here is the link: [NAS-125640] Kubernetes cluster failed to start for Applications during High RAM usage, i.e. 99% used - iXsystems Jira (atlassian.net)

 

[wili4m]

I may have found out why this error occurred, it may caused by "Kubernetes cluster run out of memory":

I noticed that my RAM usage usually get high to 99% , which only around 0.1GB memory are available. so I tried to stop a few apps which cost a lot of memory space in usual, to test if it can solve the problem. the result is it does solve the problem, currently my available RAM space is around 1GB and the error disappeared!

Summary:
How does the error happen?
- not enough available RAM could cause this error to happen, seems TrueNAS SCALE Cobia cannot handle such situation during high RAM usage.
How to solve it?
- in my case, stop high RAM usage apps & containers to free up used memory, keeping 1GB available memory space should be safe I assume.

I don't think memory is issue

I still hv plenty free memory. Sometimes just do couple time restart /shutdown or unset or re set the pool and back to normal, but not for long. After shut down at night and turn on at tomorrow than the issue show again.

 

[ABain]

@wili4m as you believe your issue is different, please do raise a bug ticket and attach a debug (without this there is a little we can investigate).Link is at the top of the forums.

 

[wili4m]

@wili4m as you believe your issue is different, please do raise a bug ticket and attach a debug (without this there is a little we can investigate).Link is at the top of the forums.

I just re setup the pool from 0, let see if this issue still show I will send debug files.

Thanks.

 

[Flippy]

I have the same issue to. It happens after a restart of truenas scale. And the only fix i can find is to press the stop button on all apps, then do a reboot of truenas and then start the apps maually. Been having this issue for a while now and cant remember if it came with cobia or not

 

[toury]

I reboot the system today, unfortunately the issue comes back. fix it by reset the apps pool in GUI, I am trying to migrate my ix-applications to SSD pool to see if that can help.

 

[wili4m]

I reboot the system today, unfortunately the issue comes back. fix it by reset the apps pool in GUI, I am trying to migrate my ix-applications to SSD pool to see if that can help.

I also have same issue, already try many methods still no luck. Finally i do the hardway, backup all my data, delete old pool and re set the new pool from 0, until today i don't see the issue anymore.
Just 1 issue now i can't setup VM.
https://www.truenas.com/community/t...m-efault-failed-to-connect-to-libvirt.114879/

if possible can u raise ticket bug, hope someone can see the bugs.

good luck

 

[wili4m]

@wili4m as you believe your issue is different, please do raise a bug ticket and attach a debug (without this there is a little we can investigate).Link is at the top of the forums.

Hi I facing same issue again, sending the link report include debug

https://ixsystems.atlassian.net/browse/NAS-127198

 

[wili4m]

well today found same issue, only have 2days no issue from last post.
btw i already upgrade to TrueNAS-SCALE-23.10.1.3, still show same issue

Log in with Atlassian account

Log in to Jira, Confluence, and all other Atlassian Cloud products here. Not an Atlassian user? Sign up for free.

ixsystems.atlassian.net

 

[toury]

well today found same issue, only have 2days no issue from last post.
btw i already upgrade to TrueNAS-SCALE-23.10.1.3, still show same issue

View attachment 75462

Log in with Atlassian account

Log in to Jira, Confluence, and all other Atlassian Cloud products here. Not an Atlassian user? Sign up for free.

ixsystems.atlassian.net

Take a look at https://ixsystems.atlassian.net/browse/NAS-125640,
seems they have fixed it and the PR has been merged
24.10 PR: https://github.com/truenas/middleware/pull/13093
also https://github.com/truenas/middleware/pull/13093
let us see if it works in next Cobia release

 

[wili4m]

let us see if it works in next Cobia release

btw i already upgrade to TrueNAS-SCALE-23.10.2, but still show same issue do restart / unset the pool is done, but today it getting worst, i open new threads for more detail

https://www.truenas.com/community/threads/truenas-scale-apps-stuck-at-development-and-show.117008/

anybody can help are welcome thanks

 

[toury]

btw i already upgrade to TrueNAS-SCALE-23.10.2, but still show same issue do restart / unset the pool is done, but today it getting worst, i open new threads for more detail

https://www.truenas.com/community/threads/truenas-scale-apps-stuck-at-development-and-show.117008/

anybody can help are welcome thanks

Not 23.10.2, it was merged to 24.04-RC1

 

[wili4m]

Not 23.10.2, it was merged to 24.04-RC1

so i must wait until Mar 19?


almost 1 day the apps stuck can't access my cloud, all method i try still stuck.

 

[toury]

View attachment 76373

so i must wait until Mar 19?


almost 1 day the apps stuck can't access my cloud, all method i try still stuck.

You can try it. however, your issues might differ from mine, I can always fix the ix-pool initial problem by reset and unset.
You can report a bug, export and upload your debug log to the ix team, and it might help.

 

[wili4m]

You can try it. however, your issues might differ from mine, I can always fix the ix-pool initial problem by reset and unset.
You can report a bug, export and upload your debug log to the ix team, and it might help.

i already report the bugs, but they can't give solution coz i'm using truecharts not TrueNAS apps.