Requesting docker-compose example

[morganL]

Thanks @Ixian and @ornias ... some good constructive discussion here.

My primary concern is running Docker like this precludes a user from running any Apps or slowly migrating their apps to Helm Charts. Is this the case?

Could we have default advice be start a VM.... and then this other approach be recommended for smaller systems that have zero intentions of using Apps? If so, I think it would be a very useful document which we could publish in Docs or on the forums.

 

[Basil Hendroff]

While the heavyweights thrash out a position, let's not ignore the quiet voices of @Mlovelace here and @Yorick here. Their comments seem to be resonating with a broader cross-section of respondents.

 

[morganL]

While the heavyweights thrash out a position, let's not ignore the quiet voices of @Mlovelace here and @Yorick here. Their comments seem to be resonating with a broader cross-section of respondents.

Ironically, those are recommending a greater divergence from docker.

 

[Basil Hendroff]

Ironically, those are recommending a greater divergence from docker.

Yes, that's true, but Docker-Compose is still relevant. See follow-up posts #38, #39 and #47.

 

[Yorick]

Ironically, those are recommending a greater divergence from docker.

From docker the daemon, in particular. I am looking a little deeper: Kubernetes may drop docker-shim support as early as 1.23 in late '21. This might very well slip to a later release in '22, of course. And mirantis will continue maintaining the shim. Still, native CRI seems more important to Enterprise customers than "Docker". As icing on that cake: " If Docker is working for you, moving to containerd should be a relatively easy swap and will have strictly better performance and less overhead. "

From a userland perspective, I am advocating for providing the tools to allow people to use familiar docker and docker-compose CLI: podman, podman-docker and docker-compose, with the podman.socket /var/run/docker.sock service running. Optionally also podman-compose so people can choose between docker-compose and podman-compose.

Rootless podman may or may not make it into the first iteration.

So I am advocating for:
- Use the containerd engine for k3s, because it is a) the default and b) the upstream k8s will drop docker-shim support in >= 1.23, with support being picked up by Mirantis/Docker.
- Give users familiar docker and docker-compose tools by bundling podman, podman-docker and docker-compose, podman.socket running, without actually running a docker daemon in the background. Bonus, also install podman-compose, because why not.
- Since podman and k3s are independent of each other, for an iteration on this, consider a UI switch to run podman rootless.

Edit: I am blithely assuming that podman and containerd/k3s can run alongside each other, because podman is daemonless. I have not actually done this, or looked deeper into whether there are known interop issues. That'd be good to know. If you know, please pipe up.

 

[Ixian]

From docker the daemon, in particular. I am looking a little deeper: Kubernetes may drop docker-shim support as early as 1.23 in late '21. This might very well slip to a later release in '22, of course. And mirantis will continue maintaining the shim. Still, native CRI seems more important to Enterprise customers than "Docker". As icing on that cake: " If Docker is working for you, moving to containerd should be a relatively easy swap and will have strictly better performance and less overhead. "

From a userland perspective, I am advocating for providing the tools to allow people to use familiar docker and docker-compose CLI: podman, podman-docker and docker-compose, with the podman.socket /var/run/docker.sock service running. Optionally also podman-compose so people can choose between docker-compose and podman-compose.

Rootless podman may or may not make it into the first iteration.

So I am advocating for:
- Use the containerd engine for k3s, because it is a) the default and b) the upstream k8s will drop docker-shim support in >= 1.23, with support being picked up by Mirantis/Docker.
- Give users familiar docker and docker-compose tools by bundling podman, podman-docker and docker-compose, podman.socket running, without actually running a docker daemon in the background. Bonus, also install podman-compose, because why not.
- Since podman and k3s are independent of each other, for an iteration on this, consider a UI switch to run podman rootless.

Edit: I am blithely assuming that podman and containerd/k3s can run alongside each other, because podman is daemonless. I have not actually done this, or looked deeper into whether there are known interop issues. That'd be good to know. If you know, please pipe up.

Unless I'm mistaken SCALE is already using containerd

Code:

CGroup: /system.slice/docker.service
             ├─206266 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

 

[ornias]

While the heavyweights thrash out a position

Uhmm, I donnot know which heavy-weights you refer to, but the only community heavyweights (when it comes to trackrecord, experience and contribution) here are mostly Patrick, You, Yorick and Me.
I actually don't even oppose Docker-Compose, but I want to make clear there are reasons for it not being there AND consequences.

From docker the daemon, in particular. I am looking a little deeper: Kubernetes may drop docker-shim support as early as 1.23 in late '21. This might very well slip to a later release in '22, of course. And mirantis will continue maintaining the shim. Still, native CRI seems more important to Enterprise customers than "Docker". As icing on that cake: " If Docker is working for you, moving to containerd should be a relatively easy swap and will have strictly better performance and less overhead. "

From a userland perspective, I am advocating for providing the tools to allow people to use familiar docker and docker-compose CLI: podman, podman-docker and docker-compose, with the podman.socket /var/run/docker.sock service running. Optionally also podman-compose so people can choose between docker-compose and podman-compose.

Rootless podman may or may not make it into the first iteration.

So I am advocating for:
- Use the containerd engine for k3s, because it is a) the default and b) the upstream k8s will drop docker-shim support in >= 1.23, with support being picked up by Mirantis/Docker.
- Give users familiar docker and docker-compose tools by bundling podman, podman-docker and docker-compose, podman.socket running, without actually running a docker daemon in the background. Bonus, also install podman-compose, because why not.
- Since podman and k3s are independent of each other, for an iteration on this, consider a UI switch to run podman rootless.

Edit: I am blithely assuming that podman and containerd/k3s can run alongside each other, because podman is daemonless. I have not actually done this, or looked deeper into whether there are known interop issues. That'd be good to know. If you know, please pipe up.

Thats some interesting feedback @Yorick indeed docker-compose and using docker-shim with k3s, is inherently incompatible. Although that mostly seems to be due to networking and such. So it might be fixable by iX.

So at the very least I would support moving k3s to CRI-O to keep things standardised somewhat.

Just this small step would:
1. Nuke any discussion about support of docker-compose, because it isn't there anymore
2. Enable future addition of docker-compose completely seperate from k3s (and thus allow both to run at the same time)
3. Give users some room of adding docker-compose like software themselves, without interfering with the middleware k3s.

I think that would be a great first step, regardless of which "camp" one resides in when it comes to the docker-compose discussion.

 

[Yorick]

Unless I'm mistaken SCALE is already using containerd

Code:

CGroup: /system.slice/docker.service
             ├─206266 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

Right so that's where it gets interesting. dockerd does use containerd. However afaik, k3s on SCALE currently uses docker-shim, hence the presence of dockerd. docker-ce for example installs docker-ce, containerd, docker-ce-cli and docker-ce-rootless-extras. That's without the presence of k3s.

This was likely done so there was an option in the UI to run docker containers directly. What I am arguing for is that we can have our cake - fully CRI-compliant k3s - and eat it too: Run docker containers directly via podman and podman-docker, no change needed even to the middleware because docker commands work (with an option of layering docker-compose onto this); and have k3s go directly to containerd, instead of going docker-shim -> dockerd -> containerd.

Some testing is, obviously, required, to ensure that this vision of k3s/containerd alongside podman/podman-docker/docker-compose isn't just a pipe dream. Edit: I can also see the merit of "fewer layers", and just doing podman/podman-compose, with a middleware change. But at that point the basic premise of "no more docker-shim" has been accepted, and the rest is details and ix figuring out what they want to support.

Edit: I got no dog in the fight re containerd and CRI-O, and no experience to weigh the relative merits. I'm defaulting to containerd here because it's the k3s default and the k8s FAQ describes it as an "easy lift" when coming from docker-shim. Anything that's not docker-shim is great by me. If after due diligence, CRI-O is the better choice, I shall do naught but rejoice.

 

[ornias]

Right so that's where it gets interesting. dockerd does use containerd. However afaik, k3s on SCALE currently uses docker-shim, hence the presence of dockerd. docker-ce for example installs docker-ce, containerd, docker-ce-cli and docker-ce-rootless-extras. That's without the presence of k3s.

This was likely done so there was an option in the UI to run docker containers directly. What I am arguing for is that we can have our cake - fully CRI-compliant k3s - and eat it too: Run docker containers directly via podman and podman-docker, no change needed even to the middleware because docker commands work (with an option of layering docker-compose onto this); and have k3s go directly to containerd, instead of going docker-shim -> dockerd -> containerd.

Some testing is, obviously, required, to ensure that this vision of k3s/containerd alongside podman/podman-docker/docker-compose isn't just a pipe dream. Edit: I can also see the merit of "fewer layers", and just doing podman/podman-compose, with a middleware change. But at that point the basic premise of "no more docker-shim" has been accepted, and the rest is details and ix figuring out what they want to support.

Edit: I got no dog in the fight re containerd and CRI-O, and no experience to weigh the relative merits. I'm defaulting to containerd here because it's the k3s default and the k8s FAQ describes it as an "easy lift" when coming from docker-shim. Anything that's not docker-shim is great by me. If after due diligence, CRI-O is the better choice, I shall do naught but rejoice.

This:
- Just move k3s as far as possible away from docker and use a different backend (no docker, no containerd). Which at least limits the chances of users messing with docker making mistakes and breaking k3s. I think thats a fair request and would fit within iX policies and vision: iX doesn't have to support docker, doesn't even include it, but this seperation would still allow users to do as they please outside of the k3s/SCALE bubble.
A win-win without actually supporting docker-compose :)

 

[jct]

This:
- Just move k3s as far as possible away from docker and use a different backend (no docker, no containerd). Which at least limits the chances of users messing with docker making mistakes and breaking k3s. I think thats a fair request and would fit within iX policies and vision: iX doesn't have to support docker, doesn't even include it, but this seperation would still allow users to do as they please outside of the k3s/SCALE bubble.
A win-win without actually supporting docker-compose :)

To quickly recap, lest any feelings get hurt: this is largely predicated on the reported ability of podman-compose to help Docker refugees get onboard into the system at their own pace.

I dig it.

 

[ornias]

To quickly recap, lest any feelings get hurt: this is largely predicated on the reported ability of podman-compose to help Docker refugees get onboard into the system at their own pace.

I dig it.

No it isn't at all.

The complete seperation I propose would mean one could even just run native docker, because k3s wouldn't be touching any port of it at all.

 

[jct]

No it isn't at all.

Oh — my apologies. Either way: quickstart + coexistence = gentle migration.

 

[ornias]

Oh — my apologies. Either way: quickstart + coexistence = gentle migration.

This, the lack of a gentle migration path is something that annoys myself as well and one of the prime reason why I donnot run any SCALE Apps myself on my home server. :(

 

[jct]

This, the lack of a gentle migration path is something that annoys myself as well and one of the prime reason why I donnot run any SCALE Apps myself on my home server. :(

I appreciate your sharing that. Thank you.

I think we've reached the heart of it. For all the truly heroic effort and best intentions, this Apps-first approach makes so little intrinsic sense that even its greatest champions aren't eager to dogfood it.

It's a labor of love; you're empathetic enough to prepare Apps for us rubes in the cheap seats. But not many [albeit among early adopters] are asking for that. And, well... several times now you've already threatened to take your ball and go home.

OK. Are we able to have an honest discussion about whom this is meant to serve, and what other gaps might need [my?] attention to win over your homelab and mine, without upsetting this precarious balance before it even ships?

 

[ornias]

I appreciate your sharing that. Thank you.

I think we've reached the heart of it. For all the truly heroic effort and best intentions, this Apps-first approach makes so little intrinsic sense that even its greatest champions aren't eager to dogfood it.

Well, Apps first makes sense to me. It just doesn't help me migrating that I cannot run both, but I blame myself for that as I run a hacky compose setup, not the product ;-)

It's a labor of love; you're empathetic enough to prepare Apps for us rubes in the cheap seats. But not many [albeit among early adopters] are asking for that. And, well... several times now you've already threatened to take your ball and go home.

Well, thats not completely true, I said so two times because I was repeating myself in two discussions. You can count that as one time.
And it was mostly for the fact that if iX decides to support too many options, I wouldn't even have the target audience to warrant the effort anymore. It's no a threat, it's more of a natural consequence of supporting too many options.

OK. Are we able to have an honest discussion about whom this is meant to serve, and what other gaps might need [my?] attention to win over your homelab and mine, without upsetting this precarious balance before it even ships?

I've already been working on this with iX (devs) privately and less-privately for almost a year by now. But my feedback for what it needs "to get into my homelab", isn't focussed on anything discussed here. There has been no feedback, wish or requirement by me for anything discussed here. Because I know iX's design philosophy a bit and try to work with them to itterate on the product and slowly improve what is already there.

There is so many little important (QOL) changes iX has to implement for the Apps system, that i've next to zero interest to distract them with big designchanges like there.

 

[jct]

Heh. You don't know me very well yet, but you are so my people. +d:)

Yes I'm speaking up late, and not in private. But I've been watching about as closely as one could from the outside for about that same year. I've witnessed enough persistent breakdown in the feedback loop that I wanted to have my story straight. To fully understand the implications of what I'm suggesting and why, and whether it's worth my speaking up at all.

I've concluded that I should speak up and offer my perspective. Whether about next steps or about viewing the horizon. I trust that we can all walk and chew gum while iX continues to deliver on their established release schedule.

I'll put my thoughts together, try to edit them down, ;) and strike up a new thread.

 

[brando56894]

I'm confused on all of this k8s vs docker-compose because I just used docker-compose via the shell and it worked perfectly, is this all about docker-compose being exposed in the UI? If people want to manage docker containers easily just setup the portainer UI and go from there.

 

[stavros-k]

I'm confused on all of this k8s vs docker-compose because I just used docker-compose via the shell and it worked perfectly, is this all about docker-compose being exposed in the UI? If people want to manage docker containers easily just setup the portainer UI and go from there.

Docker-compose is not actively blocked, but it's not supported by iX, that means that if it gets on the way for something k8s related, they will block it/remove it/alter its configuration/etc without any notice.

 

[brando56894]

I see, IMO this whole debate is kinda pointless. While it would be nice to have it exposed in the UI/officially supported, if you really want to be sure it's not going away anytime soon, just run docker and portainer in a vm. I work for a massive multimedia streaming company (100M subs) and we run all of our docker and k8s hosts in VMs. The physical hosts are just the hypervisors.

While the "point and click" interface in the Apps section is nice, IMO it's far too limited (currently only like 7 official plugins IIRC) and you need to learn Helm Charts in order to add any. If you just need something relatively simple, portainer in a VM is the foolproof way.

 

[truecharts]

While the "point and click" interface in the Apps section is nice, IMO it's far too limited (currently only like 7 official plugins IIRC) and you need to learn Helm Charts in order to add any. If you just need something relatively simple, portainer in a VM is the foolproof way.

I think our 50+ community apps are quite significant to mention here. ;-)