Looking to migrate/export containers out of TrueNAS Scale to native Docker host

[scotrod]

Hello! I started using TrueNAS Scale a month ago, and after two breaking updates from TrueNAS itself and TrueCharts, which broke all of my hard work so far, I'm unable to neither upgrade nor reinstall some of my containers.

I still have some of my Docker containers left on the TrueNAS and I'm looking for some sort of guide on how to migrate completely from this system and transfer my containers to a native Docker host. I know nothing about k8s and this will probably be a hassle.

I'm open to any guides/suggestions on the topic.

Thanks!

 

[MisterE2002]

Container images are immutable so it does not really make sense to migrate them (just add them again on your dedicated system).

What you are after is what you configured. This should be stored outside the the container on the NAS. If you used HostPath you already should know where the data is located so i assume it is mostly PVC. Use the truecharts script to mount the PVC and extract and transfer the data.

With some applications you can make a webbased dump to export/import this would be faster.

Note: if your target container is based on another image then the paths/user id's can be different.
There is not a "follow this guide..."

Try and validate your app setup/restore in all cases!

 

[scotrod]

Container images are immutable so it does not really make sense to migrate them (just add them again on your dedicated system).

What you are after is what you configured. This should be stored outside the the container on the NAS. If you used HostPath you already should know where the data is located so i assume it is mostly PVC. Use the truecharts script to mount the PVC and extract and transfer the data.

With some applications you can make a webbased dump to export/import this would be faster.

Note: if your target container is based on another image then the paths/user id's can be different.
There is not a "follow this guide..."

Try and validate your app setup/restore in all cases!

Thanks for the tips. As you may suggest, I started using Docker around the time I started with TrueNAS. Looks like I'm looking at the containers from the wrong angle. I thought that the containers can be exported/replicated to another hosts without any issues, which may not be true at all.

I think I will be better off just setting up everything over.

 

[MisterE2002]

To be clear. Containers are "instances" of images. The image is immutable. The container can be altered in runtime but all changes are lost on restart. Therefore the "i want to keep" stuff (should be) saved outside the container.

 

[systract]

After got burn a few times from system upgrade, I just put all my containers(except official photoprism) inside a dedicated VM.
Life so much better ever since.

 

[scotrod]

After got burn a few times from system upgrade, I just put all my containers(except official photoprism) inside a dedicated VM.
Life so much better ever since.

Do you mean a dedicated VM inside TrueNAS? My goal is to put all of my Docker containers inside ubuntu VM which sits on separate physical host, and to use NFS to mount everything to the Docker containers.

 

[scotrod]

To be clear. Containers are "instances" of images. The image is immutable. The container can be altered in runtime but all changes are lost on restart. Therefore the "i want to keep" stuff (should be) saved outside the container.

I was missing some vital knowledge about containers. When I span off TrueNAS I used Docker to create some apps and I really didn't want to dig deep in it, because I wanted to create simple things with it.
My understanding was that the all of my configuration was inside the container, which (is still possible??) but should not be the case. Looks like I have some stuff to learn before moving my stuff outside of TrueNAS.
For example, I have a Jellyfin server and I was expecting to transfer the Docker container to a new host and preserve all of my configs. If the container is setup as it should, all of the configs should stay on an external mounted volume if I'm correct?

 

[sretalla]

If the container is setup as it should, all of the configs should stay on an external mounted volume if I'm correct?

If you can locate that external location and see your config there, yes.

If you can't, it has probably defaulted to the PVC style of storage (somewhat obscured from human view).

You can, however, use heavyscript to mount the PVC(s) and then collect your config from the mounted location(s).

GitHub - Heavybullets8/heavy_script: Bash script for managing Truenas SCALE applications, automatically update applications, backup applications datasets, open a shell for containers.

Bash script for managing Truenas SCALE applications, automatically update applications, backup applications datasets, open a shell for containers. - Heavybullets8/heavy_script

github.com

(Option 2, 2, 1 for mounting PVCs)

 

[sretalla]

In general, you're much better off understanding how to use containers properly and not storing any data inside them, so your exercise is mostly going in the wrong direction and you will eventually hit the need to upgrade the container... at which point you lose your data... choice is yours.

 

[scotrod]

In general, you're much better off understanding how to use containers properly and not storing any data inside them, so your exercise is mostly going in the wrong direction and you will eventually hit the need to upgrade the container... at which point you lose your data... choice is yours.

Looks like I didn't express myself clear on my latest post. My goal is to keep the containers "clean" from now on, and to store everything externally, so the next time I have to move/upgrade the container, I don't have to worry about my data.

 

[sretalla]

Perhaps I wasn't clear enough either...

I was suggesting that SCALE can already do that (and sort-of forces it in many cases as kubernetes really enforces container ephemerality by redeploying on almost every action).

You can do that if you get it right in a VM too, but you're already there on SCALE, so maybe consider just learning how it works and check that it's all covered already before doing the work to move away.

 

[scotrod]

Perhaps I wasn't clear enough either...

I was suggesting that SCALE can already do that (and sort-of forces it in many cases as kubernetes really enforces container ephemerality by redeploying on almost every action).

You can do that if you get it right in a VM too, but you're already there on SCALE, so maybe consider just learning how it works and check that it's all covered already before doing the work to move away.

Currently, my plan is to learn how this stuff actually works. Since I am using Docker for simple and cheap stuff, I have no advantage to learn K8s because it is a complex technology and overkill for what I'm trying to accomplish. After I learn what are the different kinds of volumes and PVC (whatever that means...) I will bring up some containers on an external Docker host and mount my data via NFS - this way I don't have to worry if TrueNAS or TrueCharts decide to break something on update. And I will use just Docker, instead of K8s. I will lose smoe performance because I don't have a 10g network, but you can't have everything.

 

[MisterE2002]

Currently, my plan is to learn how this stuff actually works. Since I am using Docker for simple and cheap stuff, I have no advantage to learn K8s because it is a complex technology and overkill for what I'm trying to accomplish. After I learn what are the different kinds of volumes and PVC (whatever that means...) I will bring up some containers on an external Docker host and mount my data via NFS - this way I don't have to worry if TrueNAS or TrueCharts decide to break something on update. And I will use just Docker, instead of K8s. I will lose smoe performance because I don't have a 10g network, but you can't have everything.

The idea on the external Docker host is exactly the same. Kubernetes does not change that much. You still need to know the "volume" to map. That said, docker hub pages usually explain this.

The official images are prepared to create the -not so obvious- PVC storage. You can image this as a hidden zvol. However this is created in the ix-applications folder. This is persistent, so if you restart a image this will be attached to the next container.

sretalla answer was somewhat vague, but a normal "update" of the image (and thus next instance/container) should not delete the PVC data. However if you DELETE the container first it will also delete the PVC.
(AFAIK truecharts forced a reinstall of all items with the latest version causing a lot of angry users)

So basically. You can use PVC's but understand that you have to backup this part in some cases, or make the HostPath mapping, but you have to figure out yourself where the volume-data is stored (usually in "/config")

​

 

[scotrod]

The idea on the external Docker host is exactly the same. Kubernetes does not change that much. You still need to know the "volume" to map. That said, docker hub pages usually explain this.

The official images are prepared to create the -not so obvious- PVC storage. You can image this as a hidden zvol. However this is created in the ix-applications folder. This is persistent, so if you restart a image this will be attached to the next container.

sretalla answer was somewhat vague, but a normal "update" of the image (and thus next instance/container) should not delete the PVC data. However if you DELETE the container first it will also delete the PVC.
(AFAIK truecharts forced a reinstall of all items with the latest version causing a lot of angry users)

So basically. You can use PVC's but understand that you have to backup this part in some cases, or make the HostPath mapping, but you have to figure out yourself where the volume-data is stored (usually in "/config")

​

Correct. I'm one of those angry users and I wish to completely separate my containers from Kubernetes (which is way too complex for my purposes) and TruaCharts well... for everything.

 

[sretalla]

OK, so I'll stop trying to convince you it's easier to stay where you are and go back to help you to see my recommendation to move forward.

You can get to your data/config which is stored (by the official or TrueCharts apps) in PVCs by using heavyscript.

That script required no knowledge of kubernetes.

Run it, select option 2, then 2, then 1 and select the app PVC to mount... you can even select more before leaving the script.

The go under /mnt/ on your server and find the mounted app PVCs to get access to whatever data was there in your apps that would need migration to your new setup.

Copy the data as appropriate, then when setting up the new apps, make sure to point the volume mounts (or whatever you're going to do for storage mounts) at that respective data.

 

[scotrod]

OK, so I'll stop trying to convince you it's easier to stay where you are and go back to help you to see my recommendation to move forward.

You can get to your data/config which is stored (by the official or TrueCharts apps) in PVCs by using heavyscript.

That script required no knowledge of kubernetes.

Run it, select option 2, then 2, then 1 and select the app PVC to mount... you can even select more before leaving the script.

The go under /mnt/ on your server and find the mounted app PVCs to get access to whatever data was there in your apps that would need migration to your new setup.

Copy the data as appropriate, then when setting up the new apps, make sure to point the volume mounts (or whatever you're going to do for storage mounts) at that respective data.

Thank you! I managed to salvage some of my data using the script and your guide.

As for TrueNAS Scale... I do have a strange feeling about the way it does containerization- do big enterprises will REALLY use a NAS to manage their k8s clusters? I feel like the majority of this feature will be home users, which usually go with the docker path instead of Kubernetes, which is way too complex for most home users... Just my 5 cents on the topic.

 

[sretalla]

As for TrueNAS Scale... I do have a strange feeling about the way it does containerization- do big enterprises will REALLY use a NAS to manage their k8s clusters? I feel like the majority of this feature will be home users, which usually go with the docker path instead of Kubernetes, which is way too complex for most home users... Just my 5 cents on the topic.

I would start by pointing at the intention of SCALE (which is to "Scale-Out").

If you're going to containerize across multiple nodes, docker doesn't cut it.

You could use Docker Swarm, but nobody is doing that, Kubernetes won the VHS vs Betamax war on that one several years ago.

So, I can see why (at this point where application clustering is not yet available) you would be questioning the complication, but it's just a straightforward necessity to go where the product is designed to go.

Will enterprises use it for that? I don't know, but I expect some who trust TrueNAS for their Enterprise storage will strongly consider it.

If they don't, we'll probably lose that part of the product, so we should hope for at least some adoption.

 

[sretalla]

I also see I didn't suggest the jail-style linux container as an option for you... maybe that's a better fit for what you want and still allows sticking with SCALE (if that's a thing you care about).

Use a custom "Jail" in SCALE? (Possible? Alternatives?)

Being less familiar with Docker/Kubernetes and containers, while having good results with Jails under TrueNAS Core, I'm curious if it's possible to do the following in SCALE: Have a standalone "from scratch" container with specific CLI tools, mount points, and the ability to SSH into such a...

www.truenas.com

Look not at the first suggestion (running compose as a TruCharts App), rather the second one, which talks about jailmaker.

 

[scotrod]

I also see I didn't suggest the jail-style linux container as an option for you... maybe that's a better fit for what you want and still allows sticking with SCALE (if that's a thing you care about).

Use a custom "Jail" in SCALE? (Possible? Alternatives?)

Being less familiar with Docker/Kubernetes and containers, while having good results with Jails under TrueNAS Core, I'm curious if it's possible to do the following in SCALE: Have a standalone "from scratch" container with specific CLI tools, mount points, and the ability to SSH into such a...

www.truenas.com

Look not at the first suggestion (running compose as a TruCharts App), rather the second one, which talks about jailmaker.

I'm still doing my research for choosing Core/Scale. Since I don't run any applications on Scale anymore, I don't know if I Scale will have any advantages over Core for now (for me). And since after the latest update a lot of people are reporting NFS performance hits (my TrueNAS is nothing more but NFS storage for me right now) I wonder if I should make the switch to Core instead... Perhaps for a little more stability.

 

[sretalla]

The only point to running SCALE would be if you wanted apps. Stability and performance for storage is all in CORE... but you can't get docker there, so I guess that's how we started.

If you use the jailmaker approach in SCALE, you can get "direct" access from your apps to the storage (no need to use NFS), so maybe that's actually what you want.