Kubernetes clusters and API stability

[Perry The Cynic]

How should I think about the kubernetes cluster(-of-one) embedded in TrueNAS SCALE? Is it a feature that I can use, or is it an implementation detail I should stay away from? I'm not asking what I can do - I know my way around k8s. I'm asking what is supported.

Concrete questions:
Where is the kubernetes state stored? Is it part of the system configuration, or part of the system pool, or both? If I restore a configuration backup, what will happen to the cluster content? If I upgrade my TrueNAS SCALE system, will that wipe out the cluster and make a new one, or will it upgrade the cluster with my things still inside (and yes, I realize that might break them)? Which of these answers are API (stable)? :)

I'm sorely tempted to run some basic services inside the k8s cluster - Prometheus, Grafana, a bind server, a local package repository; that kind of thing. But will I shoot myself in the foot, stability wise, if I do? And yes, I realize there's a Prometheus "app", but come on... 1.0.47?

Thanks
-- perry

 

[Perry The Cynic]

Uh... ignore the Prometheus snark at the end. I misread the version stamp. [Carefully wiping egg off my face.]

 

[sretalla]

Where is the kubernetes state stored? Is it part of the system configuration, or part of the system pool, or both? If I restore a configuration backup, what will happen to the cluster content? If I upgrade my TrueNAS SCALE system, will that wipe out the cluster and make a new one, or will it upgrade the cluster with my things still inside (and yes, I realize that might break them)? Which of these answers are API (stable)?

Kubernetes config is stored (some networking setup, the designated pool, multus and kube-router CAs and tokens) in the config DB (in the boot pool, but also in config backups).

The rest of your kubernetes/apps "stuff" is in the designated pool.

All of that brings back the same cluster with a boot pool rebuild and a restored config (as long as your designated pool remains present and healthy).

 

[Perry The Cynic]

So a snapshot of the designated pool should suffice, if I restore the config from a backup there? That sounds... easy. (I think I'll go try that out while I still have my test scaffolding in the box...)

So how will the upgrade to Cobia work? What version of k8s does it run?

Thanks
-- perry

 

[sretalla]

What version of k8s does it run?

It runs k3s:
k3s version v1.26.6+k3s-63d67797-dirty (63d67797)

So how will the upgrade to Cobia work?

You change trains to Beta for Cobia... elect to take the upgrade... that's it.

So a snapshot of the designated pool should suffice, if I restore the config from a backup there?

Not clear on what you're worried about here... an upgrade won't touch your pool, but you can take a snapshot if that helps you somehow.

 

[Perry The Cynic]

I'm curious how my k8s cluster will be handled during the Cobia upgrade. The obvious options are to either recreate the cluster and load the configuration into it, or to perform a live upgrade (which 1.25->1.26 can do). Live kubernetes upgrades are always so much... fun.

The backup worry is that if the configuration is split between the boot-pool (or external config backup) and the designated pool, then atomic snapshots aren't an option and I need to make sure to coordinate the two. If I can restore everything from the designated pool, that's easy.

Cheers
-- perry

 

[sretalla]

I'm curious how my k8s cluster will be handled during the Cobia upgrade. The obvious options are to either recreate the cluster and load the configuration into it,

It goes this way... OS/appliance code is updated (including the new K3s)... actually a new boot environment is created with the updated version, config is transferred, system is rebooted.

 

[Perry The Cynic]

Great. Thank you so much for clearing that up for me. It sounds pretty clean and neat.

Cheers
-- perry