k3s not starting after upgrade

[zyrex]

I totally agree, should be 1000. However iX decided to change that, so we wonder why, because is not explained anywhere like you said very well. What if iX decides to use UID 1000 for something else internally. I found the UID issue yesterday, by pure accident, while helping an user fix his apps. This was a clean Bluefin install, I was like "Why the non-root user UID is 3000??" But they sure did fix the Debian .profile, which was broken in Angelfish. I don't have answers to Why's but at least people are aware of these non-documented changes. My thread has a fix for the OP, see Kubernetes Service section. Let me know if that fixes the issue. I've been revising it every other day with new findings, like you said. Just trying to help.

Thanks, I did try that work around now, but it made no difference unfortunately.
And the log ends with the same as danb35 have;

Code:

Dec 20 08:15:25 megabyte k3s[50575]: E1220 08:15:25.583180   50575 kubelet.go:1397] "Failed to start ContainerManager" err="failed to initialize top level QOS containers: root container [kubepods] doesn't exist"

Guess I'll try and submit a bug, however, yesterday when I registered for the Jira-page, I ended up at a weird portal page, and I couldn't see all the other bugs I could before I signed in.

 

[danb35]

Are you sure that unsetting the pool in UI and rebooting does not fix the issue? If

Just tried it again--Unset pool, reboot. Once the system comes back up, log back into the web GUI. When I browse back to Apps, it asks me to choose a pool. It then spins for a few moments, and leaves me here:

Edit: Unset pool, reboot, choose different pool gives the same result, with the same error in the log.

 

[zyrex]

Same behaviour over here as @danb35 .

 

[Daisuke]

So we are at this moment where you open a ticket for a procedure how to manually create the missing container, or delete the ix-applications dataset. I don’t even know if the dataset will be automatically recreated on reboot, it will probably not properly delete because Docker leftovers, but should be enough to let you move it to another pool after reboot.

 

[João Fernandes]

I'm having similar issues, not sure if it's related to this specific or not. Running k3s kubectl get all -A I got this:

 

[João Fernandes]

after unset zpool and reassigning after reboot, I had to manually start k3s. This is the current status of the k3s cluster

 

[danb35]

Edit: Unset pool, reboot, choose different pool gives the same result, with the same error in the log.

So, after my leaving the system alone for several hours with the different pool chosen, I checked again. k3s was then running, and the Apps page said none were installed. I've set it back to the original pool and will check it again in a bit.

 

[Daisuke]

k3s was then running

I remember seeing something similar mentioned by @truecharts into one of their announcements, after upgrade leave everything running for 30 minutes or so. Leave everything settle into new pool for 30min, glad you had it fixed. Bluefin 22.12.1 is expected to be released on February 2023, with fixes for backups and app migration.

 

[danb35]

Glad you had it fixed.

I won't consider it "fixed" until my existing apps are recognized and running--until then my conclusion is that Bluefin has one hell of a showstopping bug. Getting k3s to run at all is a necessary, but not sufficient, condition.

At a minimum, there's a giant UI problem--"Please wait for k3s to finish starting" should not be denoted by a big red stop sign.

 

[Daisuke]

I won't consider it "fixed" until my existing apps are recognized and running

I was referring to the issue related to this thread, Kubernetes not running.

 

[truecharts]

Some short notes:
- We advice to ALWAYS make an extra TrueTool backup prior to upgrading, that way you can always restore from backup when rolling back.
- The 1000 user thing, is **not** a linux standard in any way, shape or form. For TrueNAS "Normal" users actually start, starting from user 1000.
1000 is a "made up" pseudo-standard by random docker creators, most of which don't even fully understand how to make a compliant and secure docker container. User 568 is picked by iX (Ornias in a PR) and k8s-at-home, because it's unused and UNDER 1000, to avoid any conflicts with custom-made users.
- Migrations on BlueFin are currently *verified* to be broken. A hotfix is included in TrueTool, after TrueTool is ran once, the same hitfix should also apply to migrations, as verified by our community.
- A lot of users seem to be trying to DIY or reverse engineer the migration process. While it isn't rocket science, a kubernetes/apps migrations is a lot more involved than "just" a replication.

 

[Daisuke]

The 1000 user thing, is **not** a linux standard in any way, shape or form.

Of course there are standards, UIDs are stored inside uid_t system data type, which is a 32bit unsigned int.

• UID -1 (which is actually 4,294,967,294), represents an invalid UID
• UID 0 is the superuser
• UIDs 1–99 are statically allocated by the system
• UIDs 100–999 are reserved for dynamically allocated users by the system and administrators
• UIDs 1000-59999 are reserved for dynamically allocated user accounts

Feel free to look into Scale's Bluefin /etc/adduser.conf, here it is an excerpt:

Code:

# FIRST_SYSTEM_[GU]ID to LAST_SYSTEM_[GU]ID inclusive is the range for UIDs
# for dynamically allocated administrative and system accounts/groups.
# Please note that system software, such as the users allocated by the base-passwd
# package, may assume that UIDs less than 100 are unallocated.
FIRST_SYSTEM_UID=100
LAST_SYSTEM_UID=999

FIRST_SYSTEM_GID=100
LAST_SYSTEM_GID=999

# FIRST_[GU]ID to LAST_[GU]ID inclusive is the range of UIDs of dynamically
# allocated user accounts/groups.
FIRST_UID=1000
LAST_UID=59999

FIRST_GID=1000
LAST_GID=59999

I have no idea why UID 3000 is enforced by iX Systems or where, I'm just a Linux guy. I expect at least to get an explanation from devs, there is a reason behind all this.

 

[truecharts]

...

Sorry, There might've been some misunderstanding.
Missed the thing about user 3000, as writen we where explaining user 568 for Apps. Which has nothing in the standards against it or about it.

About user 3000:
We've seen MANY containers decide to start with uid 1000 as their "default" user. By starting user creation at 1000, this created a scenario where Apps, sometimes randomly, got started using the first created user.

Our assumption is that starting manual user creation from 3000 onwards, prevents those conflicts in the future. We've had quite some talks with iX about how absolutely bat-shit-bad some containers are created.

 

[danb35]

We advice to ALWAYS make an extra TrueTool backup prior to upgrading, that way you can always restore from backup when rolling back.

I have daily TrueTool backups. How do I restore from them when TrueNAS doesn't seem to be able to start k3s while set to use my data pool?

 

[truecharts]

I have daily TrueTool backups. How do I restore from them when TrueNAS doesn't seem to be able to start k3s while set to use my data pool?

If you need any help with our Tools, please be sure to reach out to our support staff directly on Discord :)

In short two options:
- Clean slate and replicate back your off-site backups
- BlueFin should allow you to trigger restore, even though k3s is not running or broken. (Bluefin specific change)

 

[Daisuke]

We've seen MANY containers decide to start with uid 1000 as their "default" user.

The thing is, this has nothing to do with UID 3000 enforced by iX Systems.

Our assumption is that starting manual user creation from 3000 onwards, prevents those conflicts in the future.

Not at all, the pod securityContext is directly governed by Linux operating system guidelines you run your Kubernetes cluster into. Devs assign UID 1000 to securityContext into templates, just because they see the word "discretionary" but don't take the time to actually read or understand the Kubernetes documentation, which clearly states:

Discretionary Access Control: Permission to access an object, like a file, is based on user ID (UID) and group ID (GID).

TrueCharts does it right, by assigning UID 568 (apps) to securityContext, a dynamically allocated user by the system:

Code:

# Include{podSecurityContextRoot}
        - variable: runAsUser
          label: "runAsUser"
          description: "The UserID of the user running the application"
          schema:
            type: int
            default: 568
        - variable: runAsGroup
          label: "runAsGroup"
          description: "The groupID this App of the user running the application"
          schema:
            type: int
            default: 568
        - variable: fsGroup
          label: "fsGroup"
          description: "The group that should own ALL storage."
          schema:
            type: int
            default: 568

By starting user creation at 1000, this created a scenario where Apps, sometimes randomly, got started using the first created user.

Exactly, you should never assign UID 1000+ to securityContext. I understand we are dealing with deployment templates, but technically you give full control to a non-system user by setting the runAsUser default to UID 1000, or even worst, UID 0. Should be set as default to UID 999, complying with Linux permissions standards.

 

[truecharts]

It has everything to do with it, because securityContext is not actually the system used by a LARGE amount of containers.
Sadly enough containers often use user 1000 internally, by default for of things like PUID, UID, PGID and GID. We've even seen it be hardcoded.


You seem to misunderstand that the best-practices on how to design containers, does not mean containers comply.
To give an example About 50% of our Apps cannot use securityContext at all.

Starting manual user creation at 3000 solves a lot potential for conflicts in our expert opinion.

 

[Daisuke]

Starting manual user creation at 3000 solves a lot potential for conflicts in our expert opinion.

I understand and I agree with your logic, even if all Linux distributions have the UID set to 1000, like I demonstrated earlier. The goal is to fix this problem in Scale and your explanation has a good outcome, as now we know why in Bluefin, admins should re-create their non-root local users to align the UID with new guidelines. Is part of my Bluefin Upgrade Checklist recommendations.

 

[danb35]

It's sounding increasingly like the recommended way to upgrade to Bluefin is just to not upgrade to Bluefin--you should instead do a clean install and reconfigure from scratch.

 

[Daisuke]

you should instead do a clean install and reconfigure from scratch.

Not really, there very few changes that are required to be done, post upgrade. I'm actually revamping as we speak the SMB Shares portion of the guide to comply with current Bluefin guidelines. Other that that, everything is covered. I personally never had any issues with the upgrade, everything went smooth.