Best way to run vanilla Docker?

[victor-perez]

Recently I migrated from Core to Scale, my main reason was because of docker support and because of Linux. FreeBSD and jails are nice, but my BDS knowledge is limited. For me K8's is a bit of a overkill and the current setup is not really working for. I do belief that I can make it work for me, but not without getting deep understanding of of K8's. So after searching the Internet and this forum I found 3 ways to run vanilla Docker, but I was wondering what is the best way.

1. Inside a VM, I don't really like the extra layer
2. Via the Docker-Compose App, while this works perfectly, K8's run in the background just to run the Docker-Compose App in my case.
3. Via changing the /etc/docker/daemon.json as far I understand this need to be restored after every update. ( I do found this nice post where scripts are use to change the daemon.json, so that will make it easier to restore after a update )

I could not find any way to turn off K8's and use vanilla Docker that will survive a update, but maybe there is one?

 

[Arwen]

From my reading, it appears that plain Docker support is not really a goal for TrueNAS SCALE. As you have found out, their are ways to get it to work, but none are easily supported.

Hopefully someone who has more knowledge can give you more information.

 

[tprelog]

FYI - In October 2023, TrueNAS SCALE Cobia will be released. At that time, ix-systems is switching to containerd and Docker will be removed. While it may remain possible to enable apt and install Docker again, a different (unofficial and unsupported) approach can be found using @Jip-Hop's Jailmaker script

GitHub - Jip-Hop/jailmaker: Persistent Linux 'jails' on TrueNAS SCALE to install software (docker-compose, portainer, podman, etc.) with full access to all files via bind mounts thanks to systemd-nspawn!

Persistent Linux 'jails' on TrueNAS SCALE to install software (docker-compose, portainer, podman, etc.) with full access to all files via bind mounts thanks to systemd-nspawn! - Jip-Hop/jai...

github.com

 

[nasBuilder]

Seems like a big deal, if this kubernetes process stays and keeps consuming 10% of CPU for no reason, preventing reaching C10 state and wasting electricity, it's reason for some users to consider switching to other solutions with zfs plugins etc.

 

[sretalla]

Seems like a big deal, if this kubernetes process stays and keeps consuming 10% of CPU for no reason, preventing reaching C10 state and wasting electricity, it's reason for some users to consider switching to other solutions with zfs plugins etc.

If you have another way to run your apps, why would you enable the built-in apps?

 

[victor-perez]

If you have another way to run your apps, why would you enable the built-in apps?

I think the problem is that you are now forced to use kubernetes and that is not for everyone. Also the current implantation is limited/not easy to use if you want more harder things you have to have good kubernetes knowledge. For example I want to use for some of my containers VLAN's or IPv6 the last one is not support and the first one I didn't get it work correctly.

I do understand the way of thinking, because the build-in apps are easy to use for less technical people, but for people with more technical knowledge it's made very hard to do something else. This is even more true if you try to run a docker container that is not in the app list. And for me I don't use any of the build in apps currently, just docker containers that are not in the app list. so for me kubernetes is waste of resources as @nasBuilder already pointed out.

I see that LXC support is coming somewhere in the future, so I think self I will look in to that, because systemd-container was already killed in 22.12.3

 

[sretalla]

I think the problem is that you are now forced to use kubernetes and that is not for everyone.

And if you look at the reference to @Jip-Hop 's script in the post a couple below your first one, there's an alternative which is already available and doesn't require kubernetes...

 

[Patrick M. Hausen]

Why don't you invest the time and effort to learn more about BSD and jails? Any one-stop solution that is easy to use for everyone will also limit the options for people who want to have more control over particular settings. IPv6 and networking in general are all way easier and more transparent with jails compared to what Linux offers.

 

[victor-perez]

And if you look at the reference to @Jip-Hop 's script in the post a couple below your first one, there's an alternative which is already available and doesn't require kubernetes...

This doesn't work anymore out of the box after killing systemd-container in 22.12.3

 

[sretalla]

This doesn't work anymore out of the box after killing systemd-container in 22.12.3

OK, good point.

I found that surprising as there was no hint from @morganL in other discussions on that topic including the feature request to "allow" it to continue to work.

It's actually pretty disappointing if they have done this on purpose and didn't want to explain themselves to anyone.

 

[morganL]

OK, good point.

I found that surprising as there was no hint from @morganL in other discussions on that topic including the feature request to "allow" it to continue to work.

It's actually pretty disappointing if they have done this on purpose and didn't want to explain themselves to anyone.

I wasn't aware that this was done. Will chase down the cause.

In general, we have been tightening things for security reasons......it might be that.

 

[raidflex]

I wasn't aware that this was done. Will chase down the cause.

In general, we have been tightening things for security reasons......it might be that.

I did not see this mentioned anywhere in the release notes unless I missed something...

 

[PackElend]

Hi there,
I hope that you have enough insight to get the right feeling of how application isolation can be reached or you get at least as close as possible to it.

1. "Apps"->"Launch Docker Image", it still uses Kubernetes (#1, #2) put TN primary IP on Service/Admin VLAN, App access goes through added VLAN interface -> normal user cannot reach TN as long as the app is not compromised/does not come with tools allowing to access the host
2. TrueCharts docker-compose -> same as above
3. a VM and running anything in there, quite an overhead if you run only one or only a few apps.
4. Kubernetes Apps with TurueChart MetalLB, risks are as for 1.

although I'm not sure about the not 100% isolation if you do "Apps"->"Launch Docker Image" when I read https://gist.github.com/Jip-Hop/470...malink_comment_id=4608385#gistcomment-4608385

but as docker support going to be dropped anyway: https://www.truenas.com/community/t...ight-solution-for-docker-in-scale-v23.106612/
so the hopes are that TN SCALE is going to offer

1. jails, what is hopefully coming soon: https://ixsystems.atlassian.net/browse/NAS-108019 + https://ixsystems.atlassian.net/browse/NAS-114193
2. that Bluefin Kubernetes Multi-node Clustering? : truenas means that Kubernetes Worker Nodes can be bound to VLANs

merci :) :)

 

[raidflex]

Hi there,
I hope that you have enough insight to get the right feeling of how application isolation can be reached or you get at least as close as possible to it.

1. "Apps"->"Launch Docker Image", it still uses Kubernetes (#1, #2) put TN primary IP on Service/Admin VLAN, App access goes through added VLAN interface -> normal user cannot reach TN as long as the app is not compromised/does not come with tools allowing to access the host
2. TrueCharts docker-compose -> same as above
3. a VM and running anything in there, quite an overhead if you run only one or only a few apps.
4. Kubernetes Apps with TurueChart MetalLB, risks are as for 1.

although I'm not sure about the not 100% isolation if you do "Apps"->"Launch Docker Image" when I read https://gist.github.com/Jip-Hop/470...malink_comment_id=4608385#gistcomment-4608385

but as docker support going to be dropped anyway: https://www.truenas.com/community/t...ight-solution-for-docker-in-scale-v23.106612/
so the hopes are that TN SCALE is going to offer

1. jails, what is hopefully coming soon: https://ixsystems.atlassian.net/browse/NAS-108019 + https://ixsystems.atlassian.net/browse/NAS-114193
2. that Bluefin Kubernetes Multi-node Clustering? : truenas means that Kubernetes Worker Nodes can be bound to VLANs

merci :) :)

It seems unless you have an enterprise environment, I see no reason to use Truecharts in the home unless you just run all apps on the same Vlan and do not care about any basic network segmentation. But then again it seems that apps are breaking all the time with updates as well. Is it really that much better to deal with breaking changes on an ongoing basis vs running Jails/VMs on Truenas CORE?

Linux jails seem to be the best answer for the future at least.

 

[PackElend]

It seems unless you have an enterprise environment, I see no reason to use Truecharts in the home

I have something in between. It is a system in a multi-apartment building where is a common IoT, Network domain and some user spaces. I would like to isolate things. It is done on networking equipment, now I'm about to extend it to the application.

Code:

     
     www
     |
     |
------------------------------
|  ROUTER                    |
|  FIREWALL, VLAN-GATEWAYS   |
------------------------------
     ||
     ||VLAN-TRUNK
     ||
------------------------------
|  SWITCH, VLAN CAPABLE      |
|  only L2 operation         |=== OTHER DEVICES such as APs
------------------------------
  ||                     ||   
  || TAGGED TRAFFIC ONLY ||  
  ||                     ||  
------------------------------------------------------
| eth0                   eth1         TrueNAS SCALE  |
| ||                      ||                         |
| ----------bond0-----------                         |
|            ||                                      |
|   VLAN1----------VLAN2---------VLAN3               |
|     |             |             |                  |
|   bridge1        bridge2        bridge3            |
|   |-VM |-Docker  |-VM |-Docker  |-VM  |-Docker     |
|                                                    |
------------------------------------------------------

running Jails/VMs on Truenas CORE

because I'm too much a fan of Docker and the App Catalog :).
I know the breaking change risk is high, in particular with TrueCharts but as LXC support it is worth starting with SCALE straight away.


Seems there is an option available, running docker within "Launch Docker Image" as mentioned in Persistent .Debian 'jail' on TrueNAS SCALE to install software (docker-compose, portainer, podman, etc.) with full access to all files via bind mounts. Without modifying the host OS at all thanks to systemd-nspawn and discussed in Discord | "Apps isolation from host - with docker" | TrueNAS Community and achieving app isolation on SCALE

 

[sretalla]

Seems there is an option available, running docker within "Launch Docker Image" as mentioned in Persistent .Debian 'jail' on TrueNAS SCALE to install software (docker-compose, portainer, podman, etc.) with full access to all files via bind mounts. Without modifying the host OS at all thanks to systemd-nspawn

Note as mentioned above in this thread that this option was "removed" (or at least temporarily broken) in 22.12.3...

EDIT: or the second half of it... seems maybe I spoke too soon. Doing it via a launched debian container and then from inside that may get the job done.

 

[PackElend]

EDIT: or the second half of it... seems maybe I spoke too soon. Doing it via a launched debian container and then from inside that may get the job done.

yep, I had private chat with him yesterday, we are switching now to a public discussion

 

[NugentS]

I wasn't aware that this was done. Will chase down the cause.

In general, we have been tightening things for security reasons......it might be that.

@morganL - is there any feedback on reasons?

 

[morganL]

@morganL - is there any feedback on reasons?

We're reviewing, but the issue seems to be incompatibility with Kubernetes. To make Kubernetes robust, there have to be restrictions. One or more of those restrictions is impacting systemd containers.

In the meantime, we'd recommend running a VM.

 

[NugentS]

Hmm, that would be a shame as from my PoV systemd containers are a better idea than K3S (although I acknowledge not from an IX PoV)

The problem with a VM is that I have to allocate resources to that VM, and I can then share those resources amongst any containers on that VM so the containers are limited. Systemd just shares (like K3S), but K3S has a crappy routing problem that IX don't want to fix (although I MIGHT have found a workaround to that issue, at least for some containers - still looking at that).