VHS or BETA?

[inman.turbo]

By design, I didn't include a third option on the poll of 'both' or 'either'. There were two reasons for this. The first was, I didn't want voters to fence-sit and take the easy way out. The second reason is, like consumers of the 80's, most didn't have the luxury of having both a VHS and BETAMAX player at home. They had to choose between one or the other format.

I agree with you though. I like the idea of freedom of choice.

I understand, but that isn't my vote ... it's part of a conversation.

Anyway my vote couldn't be completely accurate within the context of TN CORE. Because whichever one I want, I want it natively. And by natively I really mean the vanilla mplementation. So we can track bugs back to the upstream source. And follow all the original documentation and guidelines.

I just need a NAS. A good one, not that Openmediavault or whomever. I don't want this other kind of stuff in the nas gui. I want it (my app integrations) pulled down from a repository and deployed automatically, through pipelines, after testing and approval.

I much prefer kubernetes in the wild. But I would have to either have to roll it out myself, or just buy it from a cloud provider, or both.

I suppose on the NAS I would prefer docker-compose. Why? It's easier to explain to the JR admins. And it gives IX SYSTEMS Engineers more cycles to work on Storage bugs. Yeesh, what's next? Openstack? In the same repo as the NAS build?

 

[morganL]

When creating polls, I don't like to counter any post with my own biases. All points of view... mainstream, left and right are welcome as long as we all keep it family-friendly. I haven't cast a vote myself. Like US political candidates, I'll do that on the last day of the poll.

At this stage though, I think it's worthwhile considering the questions 'How did we arrive at this point?' 'Why isn't everyone on board with Kubernetes?' I'd like to play devil's advocate here and present a possible scenario. The evidence seems to back it up.

There are application packages that haven't been, and, it appears, never will be, ported to BSD. These packages are readily available on Linux platforms and easily deployed through Docker. Under FreeNAS/TNCORE, the workaround is to set up a Linux VM to run these applications. What formed in the mindset of sections of the community (I'm guilty of this!) is a natural association between Linux and Docker (Docker-Compose).

TrueNAS SCALE was announced, which was very well received by the community. Linux and Docker were mentioned and sections of the community thought 'Great! We can now skip the VM and run Docker (Docker-Compose) directly on Linux'. To their surprise and dismay. they're finding out that Docker (Docker-Compose) isn't being supported. Instead, there's a detour around Docker to Kubernetes.

View attachment 48794

It's a reasonable point that the graphic is not complete or detailed. The intention was docker containers .. which is done, but we've never made a commitment to either docker compose or docker swarm.. We had to pick one for Angelfish release in 2021. It does not preclude other additions for future release trains.

 

[ornias]

My understanding was that they have configured K3s in such a way as that changes to the cluster made through the cli and not through the angular ui, and the middleware are not persisted across reboots and updates. Is that not the case?

I was refering to managing Helm installs using Angular and such.
But yes: Even initialising the K8S cluster using the TrueNAS API or CLI is possible

Or if you are referring to the Truenas cli (and not the shell) isn't it an interactive cli? Therefore fairly useless when it comes to scripting and automation.

Both the Shell, CLI and API are available to build things like ansible shizzle.

The absolute best scenario for me would be if SCALE were simply a debian package with the ui, api and all the storage and sharing functionality of CORE (plus glusterfs, maybe ceph as well) that could then be installed on a debian box, sans the apps or VM stuff, with the user free to choose their own solutions for the rest.

Well, thats not a scenario that's going to happen. So you might just-as-well stop repeating it, because you already said so a few times, on which you already got the note that it isn't going to happen.
TrueNAS CORE and SCALE all require EXTENSIVE integration with the OS.

At this stage though, I think it's worthwhile considering the questions 'How did we arrive at this point?' 'Why isn't everyone on board with Kubernetes?' I'd like to play devil's advocate here and present a possible scenario. The evidence seems to back it up.

There are application packages that haven't been, and, it appears, never will be, ported to BSD. These packages are readily available on Linux platforms and easily deployed through Docker. Under FreeNAS/TNCORE, the workaround is to set up a Linux VM to run these applications. What formed in the mindset of sections of the community (I'm guilty of this!) is a natural association between Linux and Docker (Docker-Compose).

TrueNAS SCALE was announced, which was very well received by the community. Linux and Docker were mentioned and sections of the community thought 'Great! We can now skip the VM and run Docker (Docker-Compose) directly on Linux'. To their surprise and dismay. they're finding out that Docker (Docker-Compose) isn't being supported. Instead, there's a detour around Docker to Kubernetes.


View attachment 48794

@morganL It's time to get rid of those graphics, issue a correction and smite the PR team that create it with due haste.
I completely agree with @Basil Hendroff here that the PR team created the confusion by that docker logo right there.

Is it false information? No. Docker is there, it does support docker-containers and no-where did it say "docker-compose".
But one should know people assume a lot of things based on such an infographic.

 

[ornias]

By design, I didn't include a third option on the poll of 'both' or 'either'. There were two reasons for this. The first was, I didn't want voters to fence-sit and take the easy way out. The second reason is, like consumers of the 80's, most didn't have the luxury of having both a VHS and BETAMAX player at home. They had to choose between one or the other format.

I agree with you though. I like the idea of freedom of choice.

In that case you created a useless poll, because we all know very well iX isn't going to throw away the Helm/k8s work they did.
The correct question would've been:
"Do you also want docker-compose support"
- Yes
- No
- /care

 

[ornias]

I understand, but that isn't my vote ... it's part of a conversation.

Anyway my vote couldn't be completely accurate within the context of TN CORE. Because whichever one I want, I want it natively. And by natively I really mean the vanilla mplementation. So we can track bugs back to the upstream source. And follow all the original documentation and guidelines.

I just need a NAS. A good one, not that Openmediavault or whomever. I don't want this other kind of stuff in the nas gui. I want it (my app integrations) pulled down from a repository and deployed automatically, through pipelines, after testing and approval.

I much prefer kubernetes in the wild. But I would have to either have to roll it out myself, or just buy it from a cloud provider, or both.

I suppose on the NAS I would prefer docker-compose. Why? It's easier to explain to the JR admins. And it gives IX SYSTEMS Engineers more cycles to work on Storage bugs. Yeesh, what's next? Openstack? In the same repo as the NAS build?

SCALE Isn't a NAS, it's a clustered storage appliance OS. Focussing on hyperconvergence.
It's a wholly different product, so thats why you're not content with SCALE: It's just not for your.
- It's an Appliance OS, not a standard OS.
- It's not meant to be managed by the SHELL.
- It's not meant to have user defined services running.
- It's not meant to be a NAS.

It's meant to be a (hyperconverged) (clustered) storage appliance OS.

To be honest:
I think you should stop here and focus on other things. With all you wrote it seems super clear that TrueNAS SCALE has too many features and is too much of a "closed garden" for you. Which is fine, but I think it isn't helping the product forward to try and complain it isn't a completely different product.

Where the question "Do we want docker-compose" is, in essence, valid and moving the product forward. All additional points you make aren't going to be in SCALE because that's out of design scope by a long shot. So it isn't moving the product forward.

 

[inman.turbo]

So it isn't moving the product forward

Well, there isn't a better option for storage than IX. So that's was we use, extra features or no. And whether or not my businesse's needs or use case matters I'm sure the great deal of money we spend on these things goes a long way towards helping move this and many other products forward.

But right ... maybe I should just delete my account and leave you to it, especially since my needs don't align with your opinions?

 

[inman.turbo]

Both the Shell, CLI and API are available to build things like ansible shizzle.

Thank you. Finally some info I can use.

TrueNAS CORE and SCALE all require EXTENSIVE integration with the OS

Obviously, but there is still a point here, that some users need more flexibility and control when it comes to these additional feature. It's okay to have an Apps module, but a black box kubernetes is far from ideal. Open it up to traditional management tools and we're golden. I should be able to deploy with using kubect from anywhere, without touching the ui or even being aware of its existence.

Otherwise, a more mudular approach with an opt-out would be nice.

 

[ornias]

Obviously, but there is still a point here, that some users need more flexibility and control when it comes to these additional feature. It's okay to have an Apps module, but a black box kubernetes is far from ideal. Open it up to traditional management tools and we're golden. I should be able to deploy with using kubect from anywhere, without touching the ui or even being aware of its existence.

Otherwise, a more mudular approach with an opt-out would be nice.

It's not a black-box. Like I said about 6 times already in 2 days:
Normal Helm is available, you donnot HAVE TO use iX Apps. But that also means you need to design your own storage and rollback within k8s.

Otherwise, a more mudular approach with an opt-out would be nice.

Besides the fact of not supporting clustering at the moment, you can load all of your own Containers, Storagemanagers, Dashboards, Ingress, Networking and even things like Flux or Argo, just using normal Helm.

 

[ornias]

Well, there isn't a better option for storage than IX. So that's was we use, extra features or no. And whether or not my businesse's needs or use case matters I'm sure the great deal of money we spend on these things goes a long way towards helping move this and many other products forward.

You could just roll out your own clustered k8s solution and use something like Democratic CSI to deploy storage, that still works on SCALE quite well :)
I know the preference would be to be able to integrate it completely, but at the moment thats not planned to be supported for the initial release.

Just wait patiently to look where iX wants to go with the product, clustered k8s isn't even supported yet due to some issues with it.

But right ... maybe I should just delete my account and leave you to it, especially since my needs don't align with your opinions?

Well, maybe not delete your account. That would definately be overdoing it. But what you are basically saying:
"I want it all, I want it now and I donnot care about your company vision of doing everything using a GUI"

I think thats not helpfull for anyone at all. We know quite clearly by now that the current design for the initial release of SCALE isn't fitting your usecase. A shame ofcoarse, but that is NOT going to change, no mater how often you repeat it.

And if I take the devs correctly, they are definately open to suggestions to allow some more options but "an option to completely disable their k8s deployment, while still guaranteeing it survives upgrades" is not even close to being considered.

 

[inman.turbo]

You could just roll out your own clustered k8s solution and use something like Democratic CSI to deploy storage, that still works on SCALE quite well :)

That is basically what we do now, except we bring our own hypervisor ( basically just qemu/kvm with libvirt ) and virtualize TN. I was hoping SCALE would eliminate the need for that, but it doesn't. SCALE saves a VM on each node, but sadly we would have to write new SCALE API client wrappers for all our management tools.

Which is no less difficult than working with Red Hat or Canonical on better ZFS management for Linux. However storage just isn't something we do. We prefer leave it to the professionals.

I want it all, I want it now and I donnot care about your company vision of doing everything using a GUI

Nah, not what I'm saying. I care plenty about IX's vision and what they are doing. I just care less about what you think of what I have to say about it than it seems you would like me to.

Obviously they can't cater to the concerns of just a single voice, however I'm sure they take everything into consideration, and they may have a better idea than mine which helps similar use cases but is also acceptable and feasible for them (and only they would know what is or isn't).

 

[ornias]

Nah, not what I'm saying. I care plenty about IX's vision and what they are doing. I just care less about what you think of what I have to say about it than it seems you would like me to.

I don't assume anything I just respond.

only they would know what is or isn't.

Thats your assumption.
*Looks back at being involved with nearly all k8s related issues SCALE and bugs for a year*

 

[morganL]

Its worth noting that the iXsystems decision to embrace Kubernetes is a forward-looking business decision with the intention of being where business users are looking to invest their money and talent. Docker's enterprise situation was in disarray and had moved to Mirantis. Kubernetes was widely adopted by the major industry players. https://www.itprotoday.com/containers/whos-winning-container-software-market

This is not to diminish Docker's initial contributions, but sometime startups under VC control have trouble with Open Source business models. We could not make the bet on Mirantis and Docker, when they were also selling Kubernetes themselves.

We also wanted to embrace home and small business users, and hence invested in the Apps framework and have encouraged others to participate, build their own Apps and Catalogs. The goal is to keep improving that so it is simple, robust, maintainable and has a powerful library.

Ideally, Docker Compose would be portable to Kubernetes, just like individual containers. Helm charts are similar and more powerful, but they aren't the same. THe translation tool, Kompose is not yet mature and perhaps the problem will always be too complex to be easy. However, we fully support that effort and would be open to anything TrueNAS can do to make that process easier.

For users that need to use Docker and Compose, then the current approach would be a VM. KVM is very robust and will support that well. However, its also a good idea to document your needs in this forum thread and give us information on why you want Docker and Compose for the long term. Is it critical for small systems with less VM hosting power? Is Docker Swarm also needed? Is there anything that can be added to Kubernetes/Apps that would reduce the need for Docker Compose?

Thanks to @Basil Hendroff for starting the discussion.

 

[Yorick]

Is Docker Swarm also needed?

I doubt it. I am running a swarm right now and it’s working well enough, but I struggle to see it in a scaled enterprise environment. I’ll likely have to move the thing to some form of k8s if it ever becomes enough of a business to hire someone, because people will plain walk away when faced with the prospect of having “3 years of docker swarm” on their resume.

I struggle to understand the reason to use Docker as the k8s backend here. It’s too late to change that for release, but maybe not too late to de-emphasize it in the marketing and plan to change it down the road? If the idea is to run docker containers, that can be done without a Docker backend. Podman for example gives a CLI for interacting with containers, as others already pointed out. By installing podman-docker, podman becomes a drop in replacement, no changes to existing code needed.

What people want to do with docker-compose can be done with podman. Since v3 podman supports docker-compose, and of course there is also podman generate and podman compose

I’d opt for not enshrining docker further. That means docker-compose support via the OCI tools that are available.

Kubernetes will drop support for the docker backend in >= 1.23 - it’s a bit concerning to see a new product utilizing k3s being developed that uses docker as the backend, even as support for the shim will be picked up by Mirantis/Docker. CRI-O or containerd sound like better options for longevity. I’d lean towards containerd only because it’s the default in k3s and so will be well supported by the upstream project.

Don't Panic: Kubernetes and Docker

Update: Kubernetes support for Docker via dockershim is now removed. For more information, read the removal FAQ. You can also discuss the deprecation via a dedicated GitHub issue. Authors: Jorge Castro, Duffie Cooley, Kat Cosgrove, Justin Garrison, Noah Kantrowitz, Bob Killen, Rey Lejano, Dan...

kubernetes.io

From k3s docs: “k3s includes and defaults to containerd. Why? Because it’s just plain better. If you want to run with Docker first stop and think, “Really? Do I really want more headache?””

Article on podman with docker-compose: https://fedoramagazine.org/use-docker-compose-with-podman-to-orchestrate-containers-on-fedora/

TL;DR: Please change the backend as soon as feasible. The longer you wait, the more painful this will become - and it’s an inevitable change. The docker backend’s days are numbered. Change the marketing to emphasize OCI, once the change has been made: This will resonate more with Enterprises than “docker”.

I like the idea of docker-compose support for home users, as long as that back-ends to podman-docker, not docker. Just in the interest of OCI compliance and longevity. I use docker and like it; I don’t think it’s the right choice for scale.

 

[Basil Hendroff]

give us information on why you want Docker and Compose for the long term.

First my credentials...I'm a user of container technology. I know next to nothing about container architecture. I know a little about Docker and Docker-Compose to be dangerous. I know next to nothing about Kubernetes and Helm Charts.

To try to understand the Why from my perspective is probably best summed up by the table below. This is very much a rough and ready reckoner; nothing more. What I did was to consider two popular applications that could be served through container orchestration. I went to their support forums and searched for the four terms in the top row of the table and noted how many search results were returned.

search terms	docker	docker-compose	kubernetes	helm chart
Nextcloud	500+	500+	78	20
WordPress	1220	92	153	5

Note that the search engine in the Nextcloud forum will report 500 results and then respond with 'There are more results. Please narrow your search criteria.'

For me. what I read into this are two things:

1. There's more self-help support material around Docker and Docker-Compose for these applications.
2. I'm likely to get more support from forum members around Docker and Docker-Compose for these applications.

It's a small sample set, but I wouldn't be surprised if the results extrapolate to other applications that require container orchestration.

 

[Patrick M. Hausen]

I know a little about Docker and Docker-Compose to be dangerous.

"I have a docker-compose.yml and I am going to use it!"

 

[Ixian]

There are some really user-hostile responses in this and other threads on the subject. If the answer is "my way or the highway" how is that good for Ix systems? Because they need this community and having a group of users who talk down to everyone and brook no opinions other than their own will fracture it.

 

[Ixian]

I did a short write up on how to use docker-compose, etc. with TrueNAS SCALE here: https://www.truenas.com/community/threads/requesting-docker-compose-example.94684/page-2#post-655496

Note that I'm currently testing the nightlies (21.08) though I don't know that it wouldn't work with 21.06 either.

 

[Basil Hendroff]

I doubt it. I am running a swarm right now and it’s working well enough, but I struggle to see it in a scaled enterprise environment. I’ll likely have to move the thing to some form of k8s if it ever becomes enough of a business to hire someone, because people will plain walk away when faced with the prospect of having “3 years of docker swarm” on their resume.

I struggle to understand the reason to use Docker as the k8s backend here. It’s too late to change that for release, but maybe not too late to de-emphasize it in the marketing and plan to change it down the road? If the idea is to run docker containers, that can be done without a Docker backend. Podman for example gives a CLI for interacting with containers, as others already pointed out. By installing podman-docker, podman becomes a drop in replacement, no changes to existing code needed.

What people want to do with docker-compose can be done with podman. Since v3 podman supports docker-compose, and of course there is also podman generate and podman compose

I’d opt for not enshrining docker further. That means no docker-compose support, instead looking at the OCI tools that are available.

Kubernetes will drop support for the docker backend in 1.20 - it’s a bit concerning to see a new product utilizing k3s being developed that uses docker as the backend. CRI-O or containerd sound like better options for longevity. I’d lean towards containerd only because it’s the default in k3s and so will be well supported by the upstream project.

Don't Panic: Kubernetes and Docker

Update: Kubernetes support for Docker via dockershim is now removed. For more information, read the removal FAQ. You can also discuss the deprecation via a dedicated GitHub issue. Authors: Jorge Castro, Duffie Cooley, Kat Cosgrove, Justin Garrison, Noah Kantrowitz, Bob Killen, Rey Lejano, Dan...

kubernetes.io

From k3s docs: “k3s includes and defaults to containerd. Why? Because it’s just plain better. If you want to run with Docker first stop and think, “Really? Do I really want more headache?””

Article on podman with docker-compose: https://fedoramagazine.org/use-docker-compose-with-podman-to-orchestrate-containers-on-fedora/

TL;DR: Please change the backend as soon as feasible. The longer you wait, the more painful this will become - and it’s an inevitable change. The docker backend’s days are numbered. Change the marketing to emphasize OCI, once the change has been made: This will resonate more with Enterprises than “docker”.

I like the idea of docker-compose support for home users, as long as that back-ends to podman-compose, not docker. Just in the interest of OCI compliance and longevity. I use docker and like it; I don’t think it’s the right choice for scale.

The more I think about this post, the more I'm warming to it. Thinking about it. I really don't need Docker. Whenever I come across Docker examples I'm interested in, I run them through Composerize and convert them to Docker-Compose format anyway. As long as I can run Docker-Compose, and it is supported, I'd be happy.

 

[Ixian]

The more I think about this post, the more I'm warming to it. Thinking about it. I really don't need Docker. Whenever I come across Docker examples I'm interested in, I run them through Composerize and convert them to Docker-Compose format anyway. As long as I can run Docker-Compose, and it is supported, I'd be happy.

Podman is better in many ways I agree and I've used it on RHEL systems with compose files no problem but I believe it's still in testing mode for Debian 11, which SCALE is based off of.

It's not a bad idea though, as SCALE develops.

 

[ornias]

iX systems always has had a "GUI first, official API and CLI second" approach and has never had much official support for external CLI tools.

iX also always picked solutions that are interesting for their enterprise users and tried to make those more-or-less accessable to their home-users.

So I don't really see why iX is getting flag for doing what they always did, because it just happens to be something people don't (yet) have experience with. You did't see people complainigng this much with jails and plugins. Why? Because most people didn't know those so hadn't made up their minds yet.

It's all pretty useless feedback, because no one is actually able to give a good reason on why they need it. It's all based on "I already know this" or "more examples on the internet".

That being said:
People seem to forget that they aren't the audience that iX is making money on: Enterprises are. And like Morgan said: They do not use docker-compose at all.

---

But lets ignore all that for a moment:
How many people here even tried the new solutions like Helm and SCALE Apps yet and tried to actively(!) participate in improving the user experience on those?
With TrueCharts we are actively working on an aggregated improvement list for TrueNAS SCALE Apps and to be honest: we don't really see much constructive feedback on improvements at all.

What actual things do you guys find missing in the current system, BESIDES that it just isn't the same as your current system?

I've been quite thorough to go through docker-compose files for TrueCharts to try and assure feature-parity when I can.

There are 2 area's that keep being a bit different:
-mount-only storage vs managed-PVC and mounted storage
- Reverse-Proxy setups vs Ingress
(those are also the area's kompose shits itself though)

But that considered I'm hard pressed to find significant area's that are inherently incompatible.

In short:
Are the people crying out for docker-compose support, actually having a problem or just want your solution because "muh solution"? Because thats the feeling I get.

To try to understand the Why from my perspective is probably best summed up by the table below. This is very much a rough and ready reckoner; nothing more. What I did was to consider two popular applications that could be served through container orchestration. I went to their support forums and searched for the four terms in the top row of the table and noted how many search results were returned.

search terms	docker	docker-compose	kubernetes	helm chart
Nextcloud	500+	500+	78	20
WordPress	1220	92	153	5

Yeah, the problem here is that you are looking at the official locations. Thats not how Helm or K8S support is handled. It's a cultural difference mostly.
For K8S resources look at Bitnami or K8S-At-Home instead. ;-)

---
Problem first vs Solution first:
As might be clear from this, I'm a strong proponent of defining the actual problems first and finding solutions for those afterwards.
In the almost 100 messages this week about this I only got one solid "problem":
- Documentation (either external resources or by iX) is not on-par. (By @Basil Hendroff)

Documentation:
You guys know what: I actually agree. Documentation for everything K8S SUCKS BALLS. and iX docs for SCALE often barely exist or are not indepth enough for users to figure out how to port more complex apps like Pihole (which prefers/requires port 53 forwards).

iX also neglected for ages to remove the many docker references from their PR and still hasn't had a good guide about migrating from docker-compose to their big-blue-button (that wouldn't even be possible because many ports cannot be forwarded using that tool.

However:
In the last weeks @StavrosMadK and I, have been working quite hard to get some actually solid quick-start guides out for TrueNAS SCALE Apps. The goal is to make it easier for a new user(!) to start using SCALE Apps than to start using docker-compose. But, at the same time, make it more clear for docker-compose users to figure out where to copy-and-paste specific settings from their docker-compose file.

Daily(!) walkthrough video's have been releasing for a while now and more will follow upcomming week(s):

https://www.youtube.com/playlist?list=PLIrnTS8nNNfSfDrBC96qnMYBzFFGalFJ-

Those video's are mostly aimed to be viewed in comparison to writhen quick-start guides though, those are somewhat available here:

01 - First time Apps setup - TrueCharts

Project Documentation for TrueCharts

truecharts.org

(minus video's that aren't released yet ofc ;-) )