VHS or BETA?

[Basil Hendroff]

iXsystems has embraced Kubernetes as the way forward for SCALE. Refer to the thread TrueNAS enables Container Storage and Kubernetes. I'm sensing a lot of community pain though with the lack of Docker and Docker-Compose support on SCALE. While SCALE is still in a beta phase, I thought it might be an interesting and useful exercise to gauge community opinion on events that are unfolding.

The VHS/BETA format war is an interesting look at a company trying to dictate the market and not listening to what the public really wants.

Quote from Betamax vs VHS: The Story of the First Format War

Are we experiencing a format war here?

Cast a vote and remain anonymous and/or post an opinion and lose your anonymity.

EDIT: I'll keep this poll going for the month of August.

 

[morganL]

Its probably worth noting that SCALE will include Kubernetes (clustered next year) as part of the general proposition for TrueNAS's enterprise customers. That is a given.

So the question that is more interesting is whether a simple Docker option is worthwhile. Obviously, it can be run in a VM today. If Docker is added natively, it would be at the exclusion of Kubernetes and the Apps Catalog. Is that trade-off worthwhile for many users.... or should we focus on just making Apps better?

 

[ornias]

I'm sensing a lot of community pain though with the lack of Docker and Docker-Compose support on SCALE.

For the most part I serieusly doubt if these people ever would've given SCALE Apps a fair chance. It's a HUGE upgrade from CORE plugins and that's what it is supposed to be.
When I take into account the amount of people using TrueCharts and the fact that it's a few very vocal people, I doubt that there is such a "lot" of community pain. It's a small group of people that keeps repeating themselves on every thread about SCALE Apps imho.

While SCALE is still in a beta phase, I thought it might be an interesting and useful exercise to gauge community opinion on events that are unfolding.

BETA means that no new features of this scale (pun intended) will be added, so I donnot really see value in asking it now. It would be more worthwhile to ask when RC or Release hits and Apps are a bit more consolidated, because another docker option isn't going to make release anyway.

So what you're doing now is asking people to compare a RELEASE grade product (docker-compose) with a BETA product (SCALE Apps and Containers), which I can already almost guarantee the results from. Actually: Even I would vote docker-compose at this stage.

Are we experiencing a format war here?

No, we're seeing a "few vocal users complaining about an unfinished product"-war.


About the Ease-Of-Use:
The Big-Blue-Button is pretty close in layout to a docker-compose file and compatible with most simple(!) single containers people might want to run.

For more advanced containers (complicated networking, hardware mounting, multiple containers, etc) one would need to create an App. Which I think is not okey. It's perfectly possible to have even those options integrated in the Big-Blue-Button.
I'm actually opting to create a TrueCharts build-a-bear App equivalent to the Big-Blue-Button for RELEASE, but currently we have other priorities.

However:
TrueCharts Apps are, and should be, a LOT easier to deploy than docker-compose:
"Name, next, next, next, submit" is all thats needed for a basic setup.

To help users with those early first steps AND with more advanced setup options like Ingress, we are also building quick-start guides and walkthrough video's (which are released daily comming week):

01 - First time Apps setup - TrueCharts

Project Documentation for TrueCharts

truecharts.org

https://www.youtube.com/playlist?list=PLIrnTS8nNNfSfDrBC96qnMYBzFFGalFJ-

Conclussion:
I think the most important things we need to learn (and ARE learning!) from the vocal feedback is:
1. Not everyone is going to be statisfied.
2. Even people that are never going to be statisfied might have a point which we should take serieusly to improve the product(s)

For TrueCharts that means we actually started improving our documentation significantly for 21.08, invested heavily in quick-start guidance, added more advanced features for power users and completely reworked Apps for 21.06 to allow Apps to be functional with only minimal user configuration (mostly just the name).

Allowing Docker-Compose is, ofc, fine... But will create a split in the community when it comes to support.
In all honestly: With such a split, I don't even know if I myself would be willing to spend the hours a week I'm now to support TrueCharts anymore. It's worthwhile because of the huge target audience it serves. With all sorts of alternatives being added and that audience reduced, it's a lot less interesting to build Apps.

Simply put:
- Adding Docker-compose -> Less people using Apps -> Less people building Apps -> more users using docker compose
One could just as well nuke the Apps system right away and be done with it.

About my possible bias:
I started TrueCharts because I didn't agree with how iX created their official Apps and "big-blue-button", because it abstracted away too much freedom from the users. The same problem that, in my opinion, already plagued the CORE Plugins.

I actually disagree with a LOT of design decisions from iX systems and I think i'm also quite vocal if I do not agree. So if there is one person that isn't biased and still know his shit, it would be me. ;-)

 

[Patrick M. Hausen]

The same problem that, in my opinion, already plagued the CORE Plugins.

The main problem with plugins is - in ny opinion - raising wrong expectations.

Nextcloud - the one most frequently giving people problems - is by far non-trivial. Naive users seem to expect the plugin to be updated in a timely fashion compared to Nextcloud's releases. Which obviously it is not. Not complaining, I perfectly understand how much personpower that would require. So the Nextcloud admin backend is screaming "there's an update available!" and the plugin is not delivering.

Then there's major version upgrades. Suddenly there's MySQL 8 in the jail requiring complex administration tasks to get it to start up again, then migrate the database - you know all this, of course. I bet a larger amount of beer that most users running into problems with the plugin are not even aware they are running a database server.

That's why I see the plugins as a dead end. Either commit to a select few that are "official" and put the work and the money to them including seamless automatic upgrades and everything. Or abandon plugins altogether. We have jails which are a powerful container mechanism. Improve the documentation. Team up with e.g. Klara systems for tutorial videos. Leave it to the community to come up with scripts to set up jails in a more user friendly way. Hat tip to @danb35 and @Basil Hendroff among others for their incredible work in that field.

That's what I would do. Plugins in their current state are a maintenance nightmare. Specifically for the inexperienced user they are addressing! Yes, frustrating.

I am all with @ornias with respect to "apps" on SCALE. While I don't have the time to take the deep dive - even the time reserved for open source work is already planned for different projects and then there's that EuroBSDCon talk - I am willing to just wait until things are sorted out and give them a try every now and then. Please keep at it. Your work is appreciated and valuable.

Kind regards,
Patrick

 

[ornias]

The main problem with plugins is - in ny opinion - raising wrong expectations.

This:
Its a thin line to walk:
If you add too many advanced settings people complain because #Scared
If you donnotpeople complain because #Freedom

Team up with e.g. Klara systems for tutorial videos.

I wouldn't touch coorporation with Allan "Prototype" Judge with a long pole to be honest. He is known to overpromise and underdeliver. Not sure about his company. But Afaik we are still waiting for about 2 years for some of his ZSTD-on-ZFS promises.
The way he abused the work of others on ZSTD-on-ZFS for his own private gain (both financially and in terms of PR) is also uber disgusting.

I am all with @ornias with respect to "apps" on SCALE. While I don't have the time to take the deep dive - even the time reserved for open source work is already planned for different projects and then there's that EuroBSDCon talk - I am willing to just wait until things are sorted out and give them a try every now and then.

I think most of your issues are solvable by Apps. Besides the things addressing Nextcloud.
Honestly: I think nextcloud is not well suited for running without headaches or professional knowhow. It's the only App in The TrueCharts catalog I truelly worry about with updates :(

It's also not really the database either, i've had no issues with other apps with databases. Nextcloud just as a history of bad database practices and is a insanely arrogant dev team imho.

 

[Basil Hendroff]

Nextcloud just as a history of bad database practices and is a insanely arrogant dev team imho.

Unfortunately, the same is true of WordPress. It's pretty terribly maintained and is full of holes

 

[inman.turbo]

So if there is one person that isn't biast and still know his ****, it would be me

You may have opinions, maybe even good ones. Mine are irrelevant to my bottom line. Personally I would love for Apps to work for me. But for that I need to be able to interact with it the same way as any other k8s or K3s cluster. Using Kubectl, Ansible, terraform, etc. And I need to be able to manage how many control planes and worker node, etc. Also the docker runtime is actually a problem. For "enterprise", I am beholden to the "industry standards", and we require OCI compliance.

We use docker and docker compose during development, also in a lot of pipelines and ci, but in production we use containerd.

 

[inman.turbo]

Importantly, I don't think someones preference for "using what they know" makes them worthy of redicule. Nor do I value their point of view any less. Also it's important to note that in some cases the choice of which stack to use just doesn't exist, or it is compulsory. Or maybe for some a change just isn't feasible, or doesn't make business sense.

And some are less technical than others, and that's fine too: we all need the tools we need.

 

[ornias]

Personally I would love for Apps to work for me. But for that I need to be able to interact with it the same way as any other k8s or K3s cluster. Using Kubectl, Ansible, terraform, etc.

Actually, it is possible to interact with it using the CLI and hence Ansible and such are perfectly possible to implement :)

And I need to be able to manage how many control planes and worker node, etc.

Well currently only a single node is supported anyway, it's not k8s-clustering read. So I think you're asking too much from this release. It's not even clear if next release (2022-2023) will support non-scale worker nodes or not.

Also the docker runtime is actually a problem. For "enterprise", I am beholden to the "industry standards", and we require OCI compliance.

Agreed, I totally donnot get why they went this way. It's like the marketing department wanted to put the name docker on it and hence the engineers had to use the worst(!) k8s backend they could figure.

We use docker and docker compose during development, also in a lot of pipelines and ci, but in production we use containerd.

Yeah thats my workflow too, mostly.

 

[inman.turbo]

I honestly think that if apps isn't going to be OCI compliant, and isn't going to scale or be able to be managed the way one would expect to be able to scale and manage kubernetes, docker-compose or perhaps firecracker jails would be a better choice for easy to deploy one-off apps requiring nas storage.

 

[morganL]

It's worth noting that kompose (the tool that convert docker compose files to Kubernetes yaml or Helm charts) is still in development and there's been a recent release last month. https://github.com/kubernetes/kompose/releases

It would be interesting for a developer to review their progress and identify the major outstanding issues. Perhaps this focus might solve the problems for a majority of users?

 

[morganL]

I honestly think that if apps isn't going to be OCI compliant, and isn't going to scale or be able to be managed the way one would expect to be able to scale and manage kubernetes, docker-compose or perhaps firecracker jails would be a better choice for easy to deploy one-off apps requiring nas storage.

The intention is that SCALE will be OCI compliant and will be able to scale. The question is whether SCALE should natively enable docker support and enable docker-compose. If there are any OCI compliance issues, then please document them as bugs.

 

[inman.turbo]

It's worth noting that kompose (the tool that convert docker compose files to Kubernetes yaml or Helm charts) is still in development and there's been a recent release last month. https://github.com/kubernetes/kompose/releases

It would be interesting for a developer to review their progress and identify the major outstanding issues. Perhaps this focus might solve the problems for a majority of users?

I've actually used Kompose in the past and it is a pretty good tool. The user still needs to know a little about what it's doing for them though, particularly when it comes to persistence.

 

[inman.turbo]

Actually, it is possible to interact with it using the CLI and hence Ansible and such are perfectly possible to implement :)

My understanding was that they have configured K3s in such a way as that changes to the cluster made through the cli and not through the angular ui, and the middleware are not persisted across reboots and updates. Is that not the case?

 

[inman.turbo]

Or if you are referring to the Truenas cli (and not the shell) isn't it an interactive cli? Therefore fairly useless when it comes to scripting and automation.

 

[inman.turbo]

The intention is that SCALE will be OCI compliant and will be able to scale. The question is whether SCALE should natively enable docker support and enable docker-compose. If there are any OCI compliance issues, then please document them as bugs.

That is good to know. Clearly then it's worth keeping a closer eye on the project

 

[inman.turbo]

The absolute best scenario for me would be if SCALE were simply a debian package with the ui, api and all the storage and sharing functionality of CORE (plus glusterfs, maybe ceph as well) that could then be installed on a debian box, sans the apps or VM stuff, with the user free to choose their own solutions for the rest.

 

[Basil Hendroff]

When creating polls, I don't like to counter any post with my own biases. All points of view... mainstream, left and right are welcome as long as we all keep it family-friendly. I haven't cast a vote myself. Like US political candidates, I'll do that on the last day of the poll.

At this stage though, I think it's worthwhile considering the questions 'How did we arrive at this point?' 'Why isn't everyone on board with Kubernetes?' I'd like to play devil's advocate here and present a possible scenario. The evidence seems to back it up.

There are application packages that haven't been, and, it appears, never will be, ported to BSD. These packages are readily available on Linux platforms and easily deployed through Docker. Under FreeNAS/TNCORE, the workaround is to set up a Linux VM to run these applications. What formed in the mindset of sections of the community (I'm guilty of this!) is a natural association between Linux and Docker (Docker-Compose).

TrueNAS SCALE was announced, which was very well received by the community. Linux and Docker were mentioned and sections of the community thought 'Great! We can now skip the VM and run Docker (Docker-Compose) directly on Linux'. To their surprise and dismay. they're finding out that Docker (Docker-Compose) isn't being supported. Instead, there's a detour around Docker to Kubernetes.

 

[Basil Hendroff]

The absolute best scenario for me would be if SCALE were simply a debian package with the ui, api and all the storage and sharing functionality of CORE (plus glusterfs, maybe ceph as well) that could then be installed on a debian box, sans the apps or VM stuff, with the user free to choose their own solutions for the rest.

By design, I didn't include a third option on the poll of 'both' or 'either'. There were two reasons for this. The first was, I didn't want voters to fence-sit and take the easy way out. The second reason is, like consumers of the 80's, most didn't have the luxury of having both a VHS and BETAMAX player at home. They had to choose between one or the other format.

I agree with you though. I like the idea of freedom of choice.

 

[Mlovelace]

The intention is that SCALE will be OCI compliant and will be able to scale. The question is whether SCALE should natively enable docker support and enable docker-compose. If there are any OCI compliance issues, then please document them as bugs.

When I read "intention is that SCALE will be OCI compliant" I think why didn't they just use podman, but I'm JAFO here.