Best way to run vanilla Docker?

[Jip-Hop]

So far systemd containers (jailmaker) continues to work well for me after manually installing the systemd-container package.

 

[NugentS]

I concur

 

[Jip-Hop]

We're reviewing, but the issue seems to be incompatibility with Kubernetes. To make Kubernetes robust, there have to be restrictions. One or more of those restrictions is impacting systemd containers.

In the meantime, we'd recommend running a VM.

I'm trying to find the commit which removed the systemd container package, in order to understand why this was done. So far no luck looking through commits in the 22.12.3 branch. Any update from your side?

 

[morganL]

I'm trying to find the commit which removed the systemd container package, in order to understand why this was done. So far no luck looking through commits in the 22.12.3 branch. Any update from your side?

I don't think we know....it wasn't deliberate or part of our test process. Probably a side-effect of improving stability of Kubernetes.

 

[Spusuf]

I don't think we know....it wasn't deliberate or part of our test process. Probably a side-effect of improving stability of Kubernetes.

Since its a default package does that mean it is present in 23.10? Or has it been a deliberate removal for optimization sake? Because I genuinely believe this systemd jail method is going to be the most popular going forward. So it would be nice to see whether the community should account for working around (using apt, which isnt ideal) or just wait for the next release.

 

[NugentS]

@morganL

Can we have it back please?

 

[Jip-Hop]

Probably a side-effect of improving stability of Kubernetes.

I'd be surprised if the mere fact of systemd-container package being installed (and not being used) would interfere with the stability of kubernetes. Would be great to have it back for the people who use jailmaker, but I fear we're going to have to use apt-install from now on...

 

[sfatula]

I just build my docker containers on a Raspberry Pi and use the big blue button in Scale to load the image and run it. Didn't see a need for any changes on Scale.

 

[Jip-Hop]

@morganL

Can we have it back please?

Turns out, systemd-nspawn is already back in the latest nightly TrueNAS-SCALE-23.10-MASTER-20230813-042924...

So that's good news! Just not great that it can be gone from time to time. I suppose it would be better if the systemd-container package would be on this list: https://github.com/truenas/middleware/blob/master/debian/debian/control. Then we know it will be included, until removed from that file.

Edit: I think adding systemd-container to this list would be more appropriate: https://github.com/truenas/scale-build/blob/release/23.10-BETA.1/conf/build.manifest#L227

 

[Jip-Hop]

@morganL I've made a pull request to explicitly add systemd-container to the list of installed packages.

https://github.com/truenas/scale-build/pull/482

 

[morganL]

@morganL I've made a pull request to explicitly add systemd-container to the list of installed packages.

Add systemd-container to additional-packages list by Jip-Hop · Pull Request #1 · Jip-Hop/scale-build

As requested in NAS-123533. The systemd-container package, which provides the systemd-nspawn and machinectl commands, was included with SCALE releases prior to version 22.12.3. It was then gone for...

github.com

What would be a useful addition is examples of people who used this and normal Apps together without issues.

The primary challenge for our SW team is making all standard functions reliable. If someone reports an App issue, they are unlikely to report that they also use a systemd container, even if it may be the root cause.

 

[sretalla]

What would be a useful addition is examples of people who used this and normal Apps together without issues.

I run Mailcow, which doesn't have any other supported option than docker compose.

I don't like how it works in the TrueCharts compose app, so prefer to run it in a systemd container.

I note that with the systemd container running, I get a completely separate network:

Code:

root@scale2[/mnt/nvme/jailmaker]# netstat -r
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
default         pfSense         0.0.0.0         UG        0 0          0 br01
172.16.0.0      0.0.0.0         255.255.0.0     U         0 0          0 kube-bridge
172.18.0.0      0.0.0.0         255.255.0.0     U         0 0          0 docker0

I have not seen issues with either side of it (although I'm aware I need to not open host ports that are consumed already by either system, so am being careful about that).

 

[Jip-Hop]

Pull request has already been closed unfortunately :(

 

[NugentS]

If its any help - I am using both. No issues. K3S works as it is designed to do and the Scale Jail works just great too.

I consider this a complimentary solution to K3S AND a replacement for running docker in a VM

 

[NugentS]

And I would add that the routing out of the Scale Jail works properly - where routing out of K3S is still buggered although if the container allows and gives the option for "Host Networking" then that acts as a workaround

 

[Jip-Hop]

In case anyone is wondering why systemd-nspawn disappeared suddenly: I did some digging. But it's still not conclusive...

 

[sretalla]

I would point out that we were promised linux containers at some point as part of SCALE's development, so saying that "removing 3rd party packages/dependencies that our api does not actively use" is a reason not to keep functionality that delivers on the original promise is pretty weak.

 

[sretalla]

Although clearly to deliver in full on the promise of linux containers will take time, effort and integration with the GUI/API, you're being offered an almost instant win with virtually 0 effort to just not remove something that was already there in almost every version of SCALE published so far... and still not offering any kind of alternative in replacement for what the community has provided at no cost (as a stop-gap until you're ready to deliver something fully-baked, once you decide what that will be).

 

[NugentS]

How can I add many many likes to that post.

As @sretalla says - its an instant win, it works (much better than the K3S setup IMHO - at least for my purposes anyway). Doesn't crap all over the carpet (think ix-application snapshots) like K3S does (although to be fair thats a lot better than it was (I have 420 snapshots of ix-applications now, rather than 1000's, but I wonder how much of that is down to 3rd party tools in both directions as a lot are marked HeavyScript (259)), and is better than VM's (in some instances)

I have had comments on Jira tickets that have noticed the ScaleJail - comments that have been +ve from the devs - and then they got rid of it.....

 

[Jip-Hop]

Well to be fair they didn't get rid of it (it's back in the upcoming release) but it turns out they also never explicitly added it. All I'm asking is that this time it's added explicitly (since community users are using it) so that we can rely on it being there until it's explicitly removed (which we can then see coming in the commits).