VHS or BETA?

[Ixian]

This isn't a zero-sum game. If SCALE's default is to provide "enterprise" clients Kubernetes deployment pipelines, fine. Simple single app deployment built on top of that meant for users who want a way to deploy simple apps with as few clicks as possible? Great. Happy it's there, appreciate the work. Doesn't mean we can't have flexibility for other options. I don't see the problem.

There aren't winners and losers in this. Unless the concern is not enough people will bother with Truecharts? In which case, I don't know what to tell you. I tried it, appreciate the work, got several apps working fine - it's not for me. For starters, I don't want to rely on a single, community-maintained resource for curated apps, nor is the docker-deploy button a good alternative. If SCALE actually takes a hard line on "this is the only option" as far as apps are concerned then that's when I stop using it.

Same with the whole "put it in a VM" option. That's dumb and misses the point. We can do that with CORE today if we want. I don't want to deal with the overhead of a VM, NFS mounts, and all the other faff involved.

 

[ornias]

This isn't a zero-sum game. If SCALE's default is to provide "enterprise" clients Kubernetes deployment pipelines, fine. Simple single app deployment built on top of that meant for users who want a way to deploy simple apps with as few clicks as possible? Great. Happy it's there, appreciate the work. Doesn't mean we can't have flexibility for other options. I don't see the problem.

There aren't winners and losers in this.

I do: Money.
It isn't free to support or add something.
iX has decided to go for the Apps system and k8s because there is synergy between the enterprise and home solutions (helm and apps).

Unless the concern is not enough people will bother with Truecharts? In which case, I don't know what to tell you. I tried it, appreciate the work, got several apps working fine - it's not for me.

Yeah I got that impression when you gave up after 3 hours.
But we DID take that feedback serieusly and provided more documentation on setting things up.

For starters, I don't want to rely on a single, community-maintained resource for curated apps, nor is the docker-deploy button a good alternative.

I agree that that button is pretty mediocre at best.

If SCALE actually takes a hard line on "this is the only option" as far as apps are concerned then that's when I stop using it.

Like I said 8 times this week you can just use your own custom Helm charts using k8s-at-home for example. So there ARE options.

Same with the whole "put it in a VM" option. That's dumb and misses the point. We can do that with CORE today if we want. I don't want to deal with the overhead of a VM, NFS mounts, and all the other faff involved.

yeah, but you also don't want helm (the DIY/Enterprise alternative to Apps). you just want... welll... what you want.
Thats fine, but a company isn't going to change direction just because a few users shout veryhard that they want something. They need solid reasoning.

 

[Ixian]

I spent longer than 3 hours - what happened was, after getting initial help, I figured it out and stopped bugging the Discord channel. Your attitude towards people you help is kind of awful, btw. I get the impression you mean well and you're certainly knowledgeable and passionate on the subject but you need to work on being a better ambassador if you expect users to embrace "the way". Just my observation, not trying to hurt your feelings.

K8s-at-home is a great community as well but in the end it's still a fairly narrow curated selection of apps. And it's not entirely clear what Ix's stance on custom Helm charts is - I think we're all more than clear what your stance is, but the two aren't the same unless I am missing a connection.

This debate is starting to feel pointless.

 

[ornias]

you need to work on being a better ambassador if you expect users to embrace "the way"

I'm not here to discuss my person. Nor am I an Ambassador for anything. Nor do I view TrueCharts or SCALE Apps as "The way".

K8s-at-home is a great community as well but in the end it's still a fairly narrow curated selection of apps.

With Bitnami combined they have almost all popular applications in their repositories.

And it's not entirely clear what Ix's stance on custom Helm charts is

I've a strict policy to not forward private conversations to the public, to prevent accidents where someone said to much to me and I then throw it into the public. As that would be annoying for everyone.

But:
1. @morganL talked about compose multiple times, that tool creates Helm Charts, not SCALE Apps... Turning Helm-Charts into ;-)
2. I know what the iX code does. Simply put: When it comes to how things are loaded into k3s it's mostly just helm commands internally.

- I think we're all more than clear what your stance is, but the two aren't the same unless I am missing a connection.

I don't think that my stance maters in terms of support or not? Actually, I don't actually think I even HAVE a stance on it. Because it's not my decision to make.

In short, based on the facts:
- It's comparable with direct access to the jails, it's not officially supported... But it is expected to "just work" and bug will definately be taken into account. (the different between technical-support and assistance-support)

But: again making it so personal. it's annoying.
You can repeat "not trying to hurt your feelings.", yet you yourself seems to have gotten some sort of weird intrinsic drive to make things personal all of a sudden.

This debate is starting to feel pointless.

When people don't answer thorough responses, that is indeed that tends to happen. Such a shame really.

Anyhow:
Evading arguments, turning the subject onto someones person instead of the actual subject and starting to call it pointless just because you've nothing beter to say... Welcome to the #IgnoreList.

 

[Ixian]

Mutual on ignore and we can all move on. Best of luck to you.

 

[stavros-k]

If SCALE actually takes a hard line on "this is the only option" as far as apps are concerned then that's when I stop using it.

There are options. You can use native helm commands and deploy any app you want. However you want. You will lose some of the benefits of the Rollbacks, Apps Dashboard and some features that ix also has when using the UI.
Just remember to avoid having namespaces starting with `ix-`.

I'm not familliar with helm/k8s either. But I read through the documentation of those, and also iX docs for creating an app.
And whatdoyaknow, I already ported some apps and maintaining them on TrueCharts, fixing small bugs etc.

Yes the learning curve is steep, so does the docker-compose was at first without any videos from tubers.
Yes there are low or nonexistent tutorials on tube for helm/k8s, but their official docs are great.

The only extra docs you need to create apps for scale is the iX UI yaml file. (questions.yaml). Which, when you get the grasp of it, becomes easy.

Anyway, personally, I'm happy iX picked k8s/k3s/helm, because they created a challenge for me to learn more. If you don't care to learn more, they maybe oneclick apps would be better for you and let others figure how to fix bugs or add features.

I'm always open to learn new things, especially when this thing is an industry standard.!

 

[jct]

To try to understand the Why from my perspective is probably best summed up by the table below.

FWIW, the stories I see summarized in that table are

• More people came to the forum expressing problems with Docker than with K8s
• A rough cut shipped years earlier on Docker; informed focus turned to K8s more recently

Thus it's also reasonable to recast your conclusions, in order:

1. Available self-help material is all over the map, full of historical minutiae and blind alleys
2. Help you receive will more readily come from people who missed the train when key contributors left the station

I don't know that either hot take is close enough to tell us anything useful. Charts can be funny.

Yes there are low or nonexistent tutorials on tube for helm/k8s, but their official docs are great.

I so want that to be true! I have yet to find a page on kubernetes.io that answers more questions than it raises. Being the Kubernetes Chamber of Commerce means that they're duty-bound to present every intersection without fear or favor. Getting started? First choose your path among four perfectly equal container runtimes, and five perfectly equal network plugins. Exclusively by name, without any clues to as to reputation, active development, or area of focus.

Outside, there's the opposite problem. These decisions aren't political; they're tribal and episodic. I imagine it's like stepping out of the airport in Bangalore: help is more than available; it's nigh unavoidable. Everyone knows one particular route to a favorite spot, and is strenuously urging you into their car. Get in; they'll explain on the way. No mention of preconditions, assumptions, or tradeoffs. Here's how to lock into GKE. EC2. vSphere Tanzu. You could also use a Raspberry Pi somehow!

Funny thing, though: after reading dozens of these, I noticed a pattern where nearly no one addressed on-premises storage. So that became my focus. And I found, then bit hard on those notorious SCALE dev notes. Preconfigured, well-lighted Kubernetes with integrated, robust storage out-of-the-box and room to grow? Yespleez! I figured I'd have myself up-to-speed by the time its clustering was in place.

But when they closed the API endpoint to kubectl (and thus to helm) I started to sense trouble. There keeps being one more detour to address first — just like at kubernetes.io.

Podman for example gives a CLI for interacting with containers, as others already pointed out. By installing podman-docker, podman becomes a drop in replacement, no changes to existing code needed.

You, kind sir, have just vaulted to the front of my reading list. Cheers!

 

[ornias]

But when they closed the API endpoint to kubectl (and thus to helm) I started to sense trouble. There keeps being one more detour to address first — just like at kubernetes.io.

Slight note: They didn't close anything.
They just only allow it on the host terminal using SSH, because they want to prevent people accidentally clustering installs, which isn't supported. running kubectl from another system isn't the only (or even standard!) way of using kubectl.

Also: their storage solution (with rollback etc) for k8s isn't even available on Helm directly, so unless you use Apps there is no " integrated, robust storage out-of-the-box". But if you do it's there and quite solid.

 

[jct]

Slight note: They didn't close anything.
They just only allow it on the host terminal using SSH, because they want to prevent people accidentally clustering installs, which isn't supported. running kubectl from another system isn't the only (or even standard!) way of using kubectl.

Please tell me that this shell game is only a stopgap hack until someone has a chance to address the issue with more surgical/granular permissions and guardrails. Running kubectl from another system is Rather Flipping Important for CI/CD pipelines.

Also: their storage solution (with rollback etc) for k8s isn't even available on Helm directly, so unless you use Apps there is no " integrated, robust storage out-of-the-box". But if you do it's there and quite solid.

Again: why? The more I read, the more it seems to confirm that this is configurable in-band policy. External wrappers and middlemen don't appear to be a natural fit for the ecosystem. Especially if you're correct that this is really only a wacky speed bump to remind us of the honor system, rather than being an actual guard rail protecting us from self-injury.

I have workarounds — for now. See also: Zeno's halfway paradox. But from what you and iX have been writing, there's no particular reason that this platform should be fighting against this use case. Or that I should feel like I'm fighting against the platform by pursuing it. Unless I'm missing something fundamental, this is all seemingly in-scope and shows tremendous promise. (Reminder: I've followed you to the gates of Helm; I'm not trying to sanctify Docker Compose or anything.)

If it's a question of timing and priorities, I get it and can wait. If it's closer to dart-throwing and dogma I do not, and cannot.

 

[ornias]

Please tell me that this shell game is only a stopgap hack until someone has a chance to address the issue with more surgical/granular permissions and guardrails. Running kubectl from another system is Rather Flipping Important for CI/CD pipelines.

It's not a hack, actually blocking kubectl by firewall on singlenode systems is pretty usual. iX also doesn't support CI/CD pipelines at this stage anyway.

Next release of SCALE, which is also planned to include clustered k8s support (though plans may change at this stage), is also planned to remove the firewall rule blocking API access.
It has nothing to do with permissions and such, it's done to prevent users trying to cluster it, which will cause insane issues with SCALE Apps.

Again: why? The more I read, the more it seems to confirm that this is configurable in-band policy.

What are you talking about?
Storage provisioners are quite advanced and complicated and ZFS picked the industry standard for ZFS support:

GitHub - openebs/zfs-localpv: A CSI driver for dynamically provisioning persistent Node Local Volumes for Kubernetes via the Linux ZFS storage mgmt stack.

A CSI driver for dynamically provisioning persistent Node Local Volumes for Kubernetes via the Linux ZFS storage mgmt stack. - openebs/zfs-localpv

github.com

Solid rollback is also not easy at all, it's actually the most complicated thing to do in a solid way when it comes to K8S storage.
I also find it quite a tone for someone that is just starting to read into this.

External wrappers and middlemen don't appear to be a natural fit for the ecosystem.

Uhmm, iX systems ALWAYS only supported using API and middleware to change things. Thats no change at all and actually part of their design philosofy.

Especially if you're correct that this is really only a wacky speed bump to remind us of the honor system, rather than being an actual guard rail protecting us from self-injury.

Huh? Are you mixing your comments about the kubectl thing with the storage thing now? those are two TOTALLY different problems.

I have workarounds — for now. See also: Zeno's halfway paradox. But from what you and iX have been writing, there's no particular reason that this platform should be fighting against this use case.

No one is fighting agains using Helm directly, you can even use the storage provisioner. You just cannot use the rollback system designed by iX, because it's handeled by their middleware for Apps. Feel free to use your own storageprovisioner and rollback-provider when running Helm, you are totally free to do that and it works just like you would expect. (For example: I experimented with gluster storage provisioners as a PoC).

- If you are using Helm directly: You are supposed to do everything yourself.
- If you are using Apps, iX Systems has created a great ecosystem with handholding and guardrails

Look at it positively: if you are using helm direcly, iX isn't messing with ANYTHING you do yourself. They are giving total freedom and don't interfere. Isn't that fantastic?

Or that I should feel like I'm fighting against the platform by pursuing it.

But you're not fighting agains anything, you get perfectly fine stock-helm support! It's just like how it would be if you installed it yourself!

Unless I'm missing something fundamental, this is all seemingly in-scope and shows tremendous promise. (Reminder: I've followed you to the gates of Helm; I'm not trying to sanctify Docker Compose or anything.)

If it's a question of timing and priorities, I get it and can wait. If it's closer to dart-throwing and dogma I do not, and cannot.

I've seen no interest at all from any person at iX to completely redesign their rollback and storagesolution on SCALE. Not even for the second or third release of SCALE. So don't expect it.


TLDR:
Want DIY? Use Helm!
Want guardrails? Use Apps!

 

[jct]

Huh? Are you mixing your comments about the kubectl thing with the storage thing now? those are two TOTALLY different problems.

Sorry for any confusion. I believe that in the above message I focused 100% on one issue where we seem to keep talking past each other in violent agreement.

I understand that the K8s API supports access controls (in-band policy; preconfiguration). Direct access to the API endpoint does not necessarily convey unrestricted Godzilla powers; they can be shaped and controlled. By design. Somehow users with CLI access manage not to recursively delete the root directory. And they respond well to a thoughtfully curated default environment. That's kind of the primary differentiator among platforms.

You keep telling me that SCALE does not (currently?) make use of K8s' built-in policy levers to protect anyone. Instead, iX has devised their own external layered workarounds:

• For fear that someone might try to join another node, access to the API must be redirected through ssh.
• For fear that someone might destabilize the storage or networking of Apps, they must be linearly cataloged and submitted through a separate wrapper API or GUI.

I'm not surprised to hear that this is true for any given prerelease build, or even MVP. I am continually surprised to keep seeing it repeated without progress or acknowledgement of the ambiguity as to whether this is a decision or a consequence. A product goal state or a transitional support inconvenience.

Because there's no inherent incompatibility between native Helm charts and thoughtfully preconfigured default storage and networking policy. There's no inherent risk of a cluster being arbitrarily extended or federated. Likewise, there's no inherent tension between catalogs of Apps and other namespaces/pods which could be hosted in the same well-manicured environment.

There is a practical incompatibility to these needs right now — as implementation details? All development involves trade-offs, and it is sometimes advantageous to reinvent the wheel. But I've been unable to garner a response on whether the native in-band approach was even considered and — if so — was it tabled for expediency, or was it firmly ruled out on technical and/or strategic grounds?

Authorization. Guard rails. I refer to these concepts outside of the world of GUI wizards and wrappers. Of course they apply to APIs, and they apply throughout Kubernetes. (But oh hey a hammer...?)

The fact that we're still stuck on the premise of this question rather than answering it tells me mostly that 1) I write super weird. But potentially also 2) there's a looming, disturbing possibility that the decision to go it alone was run forward and locked in without an inside advocate having ever truly walked stakeholders through evaluating the pros and cons of this and other major K8s concepts.

I'm not asking anyone to relitigate, let alone adjust plans at this very late stage. But I'm not the only one stuck on this dissonance between marketing and product design. Any insight into that process will help to inform my planning. And so I'm very grateful for the opportunity to pick your brain.

Look at it positively: if you are using helm direcly, iX isn't messing with ANYTHING you do yourself. They are giving total freedom and don't interfere. Isn't that fantastic?

In an early Slackware sort of way, yes. Meanwhile: everything else about TrueNAS beckons toward the curated, lighted path for delighting the 80%. Plus I know this guy who's really vocal about the support costs of nonstandard solutions... ;)

 

[ornias]

I understand that the K8s API supports access controls (in-band policy; preconfiguration). Direct access to the API endpoint does not necessarily convey unrestricted Godzilla powers; they can be shaped and controlled. By design. Somehow users with CLI access manage not to recursively delete the root directory. And they respond well to a thoughtfully curated default environment. That's kind of the primary differentiator among platforms.

yes and at the moment iX doesn't support direct kubectl access. So it didn't expose it to remote machines either.

You keep telling me that SCALE does not (currently?) make use of K8s' built-in policy levers to protect anyone. Instead, iX has devised their own external layered workarounds:

No I did not say so, they actually do (partly even on my advice to harden the stack a bit more), they just also donnot expose kubectl.

• For fear that someone might try to join another node, access to the API must be redirected through ssh.
• For fear that someone might destabilize the storage or networking of Apps, they must be linearly cataloged and submitted through a separate wrapper API or GUI.

Yes.

I'm not surprised to hear that this is true for any given prerelease build, or even MVP. I am continually surprised to keep seeing it repeated without progress or acknowledgement of the ambiguity as to whether this is a decision or a consequence. A product goal state or a transitional support inconvenience.

It's planned to be removed when clustered k8s releases in a future release of SCALE. It's a decision, not a consequence (as there indeed are other ways of dealing with this).

Because there's no inherent incompatibility between native Helm charts and thoughtfully preconfigured default storage and networking policy.

Thats true and that's also why running Helm charts besides apps is perfectly fine.

There's no inherent risk of a cluster being arbitrarily extended or federated. Likewise, there's no inherent tension between catalogs of Apps and other namespaces/pods which could be hosted in the same well-manicured environment.

There is, the code iX wrote is inherently incompatible with multiple nodes. Their middleware is going to get confused.

Simply put: they put NO CODE AT ALL, in place to deal with multiple nodes and their storage is going to freak out the second you try it because the storage is inherently single-node at this stage.

This will al be fixed in a future version of SCALE. But SCALE k3s is designed only to be single-node compatible. No code is in place to support any kind of clustering at this moment. That is the decision iX made (not to write said code) and you'll just have to accept that.

There is a practical incompatibility to these needs right now — as implementation details? All development involves trade-offs, and it is sometimes advantageous to reinvent the wheel. But I've been unable to garner a response on whether the native in-band approach was even considered and — if so — was it tabled for expediency, or was it firmly ruled out on technical and/or strategic grounds?

Like I said: No clustering or outside access was within scope for the initial release of SCALE. The design document for it wasn't even finished last time I heard (a month or 2 back). It's not supported because it is not within scope for the initial release.

Authorization. Guard rails. I refer to these concepts outside of the world of GUI wizards and wrappers. Of course they apply to APIs, and they apply throughout Kubernetes. (But oh hey a hammer...?)


The fact that we're still stuck on the premise of this question rather than answering it tells me mostly that 1) I write super weird.

yes, you ramble a lot. It's hard to read and you keep making side journeys and not actually sticking to one subject.

But potentially also 2) there's a looming, disturbing possibility that the decision to go it alone was run forward and locked in without an inside advocate having ever truly walked stakeholders through evaluating the pros and cons of this and other major K8s concepts.

No, clustering and external kubectl access was simply not within scope.

I'm not asking anyone to relitigate, let alone adjust plans at this very late stage. But I'm not the only one stuck on this dissonance between marketing and product design. Any insight into that process will help to inform my planning. And so I'm very grateful for the opportunity to pick your brain.

Insight: Any form of external kubectl access and/or k8s clustering is NOT WITHIN SCOPE for the initial release of SCALE

In an early Slackware sort of way, yes. Meanwhile: everything else about TrueNAS beckons toward the curated, lighted path for delighting the 80%. Plus I know this guy who's really vocal about the support costs of nonstandard solutions... ;)

This has ALWAYS been the case for all iX Systems products.

TLDR:
You keep repeating how you had prefered another solution for access to the kubectl externally and how it would've been possible to have iX storage be completely compatible with stock k8s.

- Both where not within scope for the initial release of SCALE.
- Storage: you can already use the provisioner just fine from Helm. Their rollback system is just custom,changing this is not planned and should not be expected.
- external kubectl access: Planned for a next release together with storage clustering (as that requires external kubectl access anyway)

 

[Basil Hendroff]

While the sample set is small, the poll as it stands, suggests that there is a fairly even split down the middle.

@morganL In more recent discussions in this and the sister thread Requesting docker-compose example, it appears the two sides have iterated towards some sort of equilibrium. It's all a bit techy for me, but the vibe seems positive. I hope that iX are keeping a close eye on these two threads. There are some very clever community members who have approached the problem from very different starting points, but now appear to be reaching some sort of consensus.

PS I do apologise to @hhinma who started the OP in the sister thread and has found their thread railroaded.

 

[ornias]

@morganL In more recent discussions in this and the sister thread Requesting docker-compose example, it appears the two sides have iterated towards some sort of equilibrium. It's all a bit techy for me, but the vibe seems positive. I hope that iX are keeping a close eye on these two threads. There are some very clever community members who have approached the problem from very different starting points, but now appear to be reaching some sort of consensus.

It's not really a consensus about docker-compose. It's more a consensus that using docker as a backend for k3s needlessly complicates things for everyone. The consensus wouldn't directly lead to any improved support for docker-compose in practice, it just removes a seperate issue with k3s.

However:
My primary issue with this thread is that iX has WAY more important QoL issues with Apps to fix before this should get ANY priority from anyone at iX.

 

[Kailee71]

For me at least, ideally, we'd be able to get complete LXC containers, so proper whole-systems, as well as or even instead of docker/k8s/etc. I think people who really need stuff can just as well administer a "VM" these days, and to me at least, having a whole system available is much more valuable than "just" apps, and not just because of the whole portmapping issues... Just my $0.02...

Kai.

 

[ornias]

For me at least, ideally, we'd be able to get complete LXC containers, so proper whole-systems, as well as or even instead of docker/k8s/etc. I think people who really need stuff can just as well administer a "VM" these days, and to me at least, having a whole system available is much more valuable than "just" apps, and not just because of the whole portmapping issues... Just my $0.02...

Kai.

Yeah I also REALLY want LXC containers going... I've read there was a way to do so with k8s, i'll have to read up on that :)

 

[morganL]

While the sample set is small, the poll as it stands, suggests that there is a fairly even split down the middle.

View attachment 48885

@morganL In more recent discussions in this and the sister thread Requesting docker-compose example, it appears the two sides have iterated towards some sort of equilibrium. It's all a bit techy for me, but the vibe seems positive. I hope that iX are keeping a close eye on these two threads. There are some very clever community members who have approached the problem from very different starting points, but now appear to be reaching some sort of consensus.

PS I do apologise to @hhinma who started the OP in the sister thread and has found their thread railroaded.

Thanks, yes, we've been monitoring the discussions and appreciate all the feedback.
It's not dissimilar to what we expected.
Docker was the 1st player to popularize containers and so has a large share of existing deployments
Kubernetes was the 1st open source player to make container workloads operate at scale and with Enterprise characteristics .. it had the business momentum

There's not much demand for Docker Swarm, but there is demand for Docker Compose. More specifically, there is strong interest in how to migrate existing apps described by Compose to Kubernetes.

For short term, we recommend a VM for Docker. In medium term, we'll review the ideas and see what we can propose. The team is busy now completing 21.08.

 

[Kailee71]

While the sample set is small, the poll as it stands, suggests that there is a fairly even split down the middle.

View attachment 48885

It's probably a bit late, but... could a third option be added to the poll - full LXC containers?

 

[morganL]

It's probably a bit late, but... could a third option be added to the poll - full LXC containers?

Just write-up you use-case...and why they are needed. These are on the radar for after Kubernetes is solid.

 

[ornias]

It's probably a bit late, but... could a third option be added to the poll - full LXC containers?

Those are something else entirely, K8S and Docker-Compose share a usecase, which LXC doesn't.
LXC is a fine system, but not a replacement for docker-compose or k8s,