TrueNAS Core -> TrueNAS Scale

[Arwen]

The last few posts is one reason why I stayed away from the whole Docker thing. About 2 years ago the place where I worked started more serious deployment of Docker stuff. But, all I have to do is keep the OS & hardware running. Someone else keeps the app running, (Docker in this case). So I got to ignore Docker.

I guess what I learned is;
Docker front end - which could be replaced by another program
Docker engine - this runs Docker containers, also could be replaced
Docker container - not the end all, and could be replaced by a different container

Guess what was so confusing, is that their is no simple standard for Linux containers.

 

[ornias]

Guess what was so confusing, is that their is no simple standard for Linux containers.

Uhmm...:

Open Container Initiative - Open Container Initiative

 

[georgelza]

Life would have been so much simpler if k8s decided to stay with docker as their engine, then you would have had access to all those docker how to's experience out there, and to a k8s based cluster environment... but then Kubernetes had to go and have a hand bag fight with Docker...

hmm, interesting that you decided on helm as deployment tool, I know it is very popular, but so is docker-compose and think as a non prod environment, and docker-compose = simpler vs helm...

? are you doing a single node Kubernetes or a minikube... (single node) or we even maybe getting a KIND cluster deployment capability?

G

When we first started SCALE we were considering Docker engine or Kubernetes. However, after starting the work, we realized that picking one would be more efficient. Kubernetes provided a lot more tools and was much more future-proof so we picked Kubernetes and then made sure there were the tools to run a Docker container easily.

 

[morganL]

Helm was selected as the tool because we decided that Kubernetes had more future and was the requirement from our business customers.

Current (21.04) software is single node.....but the plan is multi-node Kubernetes with the same user interface and the option of clustered datasets.

There have been lots of handbag fights over the years.. Docker, Mesos, Kubernetes. It's the nature of the business. In general, it's good that at least the container format (OCI) is agreed and runs across platforms... including SCALE. For obvious reasons, we had to chose a primary container platform and make sure we integrated and tested it well. Making sure all updates go smoothly is critical and takes a lot of testing infrastructure. Fewer choices allows us to improve quality.

There's a good blog on the subject of Docker and Kubernetes and their inter-relationship here. https://semaphoreci.com/blog/kubernetes-vs-docker

 

[georgelza]

are you aiming to along for a multi node K8s environment on a single physical TrueNAS Scale node.

or would the K8s cluster node count be aligned with the TrueNAS node count ?

G

 

[Nate W]

I migrated from CORE to SCALE and it was not bad at all on my end. My process was:

* Disable run at boot for all services in CORE and turn them all off. (All VMs, Jails, etc)
* Export the pool
* Swapped out boot disks (I went from single SSD to mirrored SSDs, all cheap/small)
* Install SCALE
* Import Pool
* Add TrueCharts repo (https://truecharts.org/)
* Spin up services and point at existing datasets

If it hadn't gone well, I could swap back the old SSD, import the pool and start the services again. Thankfully this was not needed :)

That said, I would HIGHLY suggest you wait until the next release and the new TrueCharts changes to land (See https://github.com/truecharts/apps/pull/216 ). I am waiting for 21.04 + TrueCharts changes for a few services because I don't want to redo the work that @ornias and company have already done. They've done some really great work on TrueCharts and I can't suggest them enough.

 

[ornias]

are you aiming to along for a multi node K8s environment on a single physical TrueNAS Scale node.

or would the K8s cluster node count be aligned with the TrueNAS node count ?

I've had some dev-to-dev discussions about this.
Currently the it's still in the design phase when it comes to how multiple physical truenas boxes will become a k8s cluster and how this will link into the API, Middleware and Storage. Though there are some initial idea's.

When it comes to running virtual k8s clusters on a single k8s node, thats already possible with external tools (Kind clusters for example). So thats not something that is likely to be implemented.

I migrated from CORE to SCALE and it was not bad at all on my end. My process was:

* Disable run at boot for all services in CORE and turn them all off. (All VMs, Jails, etc)
* Export the pool
* Swapped out boot disks (I went from single SSD to mirrored SSDs, all cheap/small)
* Install SCALE
* Import Pool
* Add TrueCharts repo (https://truecharts.org/)
* Spin up services and point at existing datasets

If it hadn't gone well, I could swap back the old SSD, import the pool and start the services again. Thankfully this was not needed :)

Thats a solid proces, very close to my own (yes, I run SCALE in home production for months now)

That said, I would HIGHLY suggest you wait until the next release and the new TrueCharts changes to land (See https://github.com/truecharts/apps/pull/216 ).

Yeah, I REALLY want to highlight this advice. The release of 21.04 is very close by and the new Apps will not include automatic migration.

However: Migration should be relatively straight forward (copy data-away, reinstall, copy data back) and we made sure that our 21.04 update will NOT automatically break your existing Apps. So you should be able to migrate at your own pace.

I am waiting for 21.04 + TrueCharts changes for a few services because I don't want to redo the work that @ornias and company have already done. They've done some really great work on TrueCharts and I can't suggest them enough.

If there are any more changes you would like to request, feel free to open issues about them :)

 

[amiskell]

The last few posts is one reason why I stayed away from the whole Docker thing. About 2 years ago the place where I worked started more serious deployment of Docker stuff. But, all I have to do is keep the OS & hardware running. Someone else keeps the app running, (Docker in this case). So I got to ignore Docker.

I guess what I learned is;
Docker front end - which could be replaced by another program
Docker engine - this runs Docker containers, also could be replaced
Docker container - not the end all, and could be replaced by a different container

Guess what was so confusing, is that their is no simple standard for Linux containers.

This would probably explain it a lot better and remove the confusion.

The differences between Docker, containerd, CRI-O and runc

The container ecosystem is full of terms you’re expected to know. We’ve decoded them for you.

www.tutorialworks.com

 

[georgelza]

When it comes to running virtual k8s clusters on a single k8s node, thats already possible with external tools (Kind clusters for example). So thats not something that is likely to be implemented.

I"m trying to see how I can use this platform as a larger part of my lab... as such I'd like to run a multi node K8S cluster on the single node.
So easiest... best option would be something like KIND.

As far as I know KIND only run's in docker as a engine, I'm not aware that it run inside K8S, if it run's inside K8s then that would be great. Another option could be a minikube setup, again, I know you can deploy minikube inside docker, will have to see if/how it is done in another container engined platform.

G

 

[ornias]

I"m trying to see how I can use this platform as a larger part of my lab... as such I'd like to run a multi node K8S cluster on the single node.
So easiest... best option would be something like KIND.

As far as I know KIND only run's in docker as a engine, I'm not aware that it run inside K8S, if it run's inside K8s then that would be great. Another option could be a minikube setup, again, I know you can deploy minikube inside docker, will have to see if/how it is done in another container engined platform.

G

Maybe a VM is a better solution for such a lab though, considering you can revert the complete VM as a whole too?

 

[georgelza]

Thinking bout that...

Got Ubuntu installed at the moment... just finding it dead slow. hoping a more native install will be faster.

G

 

[ornias]

Thinking bout that...

Got Ubuntu installed at the moment... just finding it dead slow. hoping a more native install will be faster.

G

be sure to install guest drivers, it make A LOT of difference at times :)

 

[georgelza]

... at the moment the speed I"m looking at/comparing is simply based on the speed with which the Ubuntu image comes up and simple remote/VNC access. nothing major yet.

Currently only have 16Gb RAM... looks like I will have to up that to 32Gb rather soon potentially.

G

 

[Jip-Hop]

docker-compose is not supported though and might even be removed in the future (if iX moves to another container engine, as docker is just the k8s container engine and not something that is supported in itself).

If you really want to use docker-compose for the long run (not just temporarily or for testing), I suggest not using SCALE and looking into another Appliance or OS all together.

From the devnotes:

"SCALE allows Kubernetes to be disabled. The user will then have access to the native container services within Debian. This will include Docker, LXC (Q1 2021) or any other Kubernetes distribution. There will be a Container Storage Interface (CSI) that can couple the container services with the SCALE storage capabilities. Users can script these capabilities and then use 3rd-party tools like Portainer to manage them. This approach can be used in SCALE 20.10 and later."

Sound like simply using Docker (without Kubernetes) will be supported, right? And Portainer supports docker-compose syntax so that could be an option for people who want to use docker-compose files.

 

[shadofall]

From the devnotes:

"SCALE allows Kubernetes to be disabled. The user will then have access to the native container services within Debian. This will include Docker, LXC (Q1 2021) or any other Kubernetes distribution. There will be a Container Storage Interface (CSI) that can couple the container services with the SCALE storage capabilities. Users can script these capabilities and then use 3rd-party tools like Portainer to manage them. This approach can be used in SCALE 20.10 and later."

Sound like simply using Docker (without Kubernetes) will be supported, right? And Portainer supports docker-compose syntax so that could be an option for people who want to use docker-compose files.

Supported no

Allowed Yes, (subject to change) but it would be up to you to get it to work using various boot scripts and such

atleast thats how it was explained previously to me roughly.

 

[ornias]

Yes, "Allows" clearly means "not disabled".
But also means: "if we break your setup with an update, it's your problem" :P

 

[LouisB]

I tried the same but I couldn't sucessfully import my pool.
I went back to TrueNAS 12.0U2

 

[BetYourBottom]

I could see Jails being migrated by moving them into FreeBSD VMs that the user could then replace with containers at their leisure.

 

[georgelza]

I'd be keen to know what type of "performance" I could expect.
currently running my TrueNAS core 12.0U3 with 16Gb RAM. as expected TrueNAS takes it all, it's doing ok, (not very happy with my Plex performance running in a Jail).
So wondering, with the migrate to SCALE, how much resources I will be able to give the single node K8s environment,
I tried running Ubuntu in a Jail, but even before installing anything it was very sluggish, simply by remote desktop connecting to it.
G

 

[ornias]

I could see Jails being migrated by moving them into FreeBSD VMs that the user could then replace with containers at their leisure.

There are a multitude of reasons iX is never going to implement this idea.
Storage, Networking and UX would take so many man hours to get into a somewhat reasonable state... nope.