Jailer
Not strong, but bad
- Joined
- Sep 12, 2014
- Messages
- 4,977
8 years for your first post? I think you win the award for the longest lurker.
-
Important Announcement for the TrueNAS Community.
The TrueNAS Community has now been moved. This forum will now become READ-ONLY for historical purposes. Please feel free to join us on the new TrueNAS Community Forums
How did we get to such a ludicrous position with containerisation?
- Thread starter marshalleq
- Start date
lmao no idea why I hadn't posted anything previously. Presumably I'd registered for a purpose at that point.8 years for your first post? I think you win the award for the longest lurker.
Looking at the timing...it's strangely appropriate that I posted in this thread, because it's almost likely a direct descendent of why I registered in the first place.
Based on timing I registered in anticipation test-driving the FreeNAS 10 beta and evaluating whether it would be a fit for me.
I've long been happy with FreeNAS and core, as far back as the m0n0wall derived days. It's always delivered exceptional reliability and performance for its primary purpose of being a great storage server. I've always found FreeBSD far more comprehensible and controllable than the ever-shifting sands of the linux world (of course some of that shifting is for good...but so much just feels like change for the sake of change).
Over time I've gradually wanted to run a few additional services on trueNAS, and while this has mostly been straightforward (at least initially...the tide is turning some here prompting another reconsideration of Scale) the idea of "easy-button" plugins had my interest 8 years ago and still does today (hence dropping in on this thread. But I also don't really trust easy-button plugins unless I know what they're doing, they have a track record of reliability, and I can have a sense of control of things like snapshotting and recovery.
10/Corral obviously never approached that so I stayed away. SCALE...is still dealing with growing pains where that promise isn't reached (at least not with a track record that I can trust). Truecharts is pushing a lot of stuff out there...but so much of how they choose to do things and their rationale...I just don't trust. Things like "don't press the stop button or things break" as a known issue, continued sniping about "standards" vs. "weird ix"...How the heck am I supposed to trust an app catalog that's seemingly at war with the platform that it runs on (and seems to be running away from given the recent "demotion" of Truenas to tier3 support)?
Over the last couple of years in particular, some things which I had run reliably in a jail moved forward with breaking changes that required moving to VM. And more and more I'm seeing software releases where the preferred and first-party distribution is something like docker, and I'm more frequently and visibly dependent on "a guy" more knowledgeable than me making unofficial FreeBSD releases work than ever. And over time I've built up a handful of stupid workarounds that I wouldn't expect to need if I migrate, but would need to undo/reimplement in a more normal way.
But hey now at least there's a jail option to use if/when plugins still aren't where they need to be after 8 years. And now they run on a platform I enjoy using less (but at least stuff runs).
Bhyve not quite getting there for stability really set this whole thing back almost a decade.
marshalleq
Explorer
- Joined
- Mar 12, 2016
- Messages
- 88
marshalleq
Explorer
- Joined
- Mar 12, 2016
- Messages
- 88
Yes we do, but I do think that the advertised use cases (e.g. plex) - being that that is an official app that they offer support for, is perhaps muddying this water. I'd be happy to just put it all in a VM - I actually think IX would be better off saying they don't do apps in this sense cause it's an embarrassing mess and they may as well end most of it. Then I'd agree that kubernetes makes sense for commercial, but then they'd have to make it REAL kubernetes not this weird hybrid thing they've got - which I assume is to appeal to that same market that perhaps we're saying isn't their core.... or wasn't. Not exactly a clear message they're sending is it.
I'm a bit unclear what you mean by "advertised use cases" of TrueNAS sandboxes. They're officially unsupported beyond "we probably won't break this functionality that we're making available to power users". iX describes them as:
From an iXsystems perspective, they're broadly promoted as "like Core jails" more than anything. And as an "unofficial" suggestion they point to jailmaker - which again doesn't mention Plex anywhere at the top-level. There's one mention of plex/jellyfin as an example of passing storage through to the sandbox.
Have people mentioned running plex in a sandbox on these forums? probably? Does that constitute an advertised use case from iX? Probably not.
The point of the sandbox feature is to be a lightweight replacement for linux VMs with better ability to share and access system resources for "most" use cases. Light enough that - if you want - you can run a bunch of single-purpose jails/sandboxes in parallel if that's your jam.
They aren't being "promoted" as replacements for apps because they aren't (using sandboxes as an app framework would bring more packaging and maintenance work in-house, which makes no sense.).
It feels like you're overthinking things a bit, especially NOT coming from a CORE background (which is still the majority of installations)
- There's a large legacy CORE base that is used to having "jails" (and also have been trained to NOT trust click-to-run plugins across the long term.). Sandboxes provide something ranging from "essential tool" to "safety blanket" to segments of this audience.
- There's a vocal group of users who "just want to run docker" or otherwise want to use their NAS/Server in ways that are outside of the realm of the "appliance" that iX delivers (also circling back to group 1)
- At the most basic, sandboxes are simply a way to provide an appliance, but with a controlled way for the end-user to do what they want on it without compromising all consistency, reliability, etc. benefits that go into making an appliance.
Or put more simply...
Leaving aside implementation details...apps are very clearly going to stay as Docker/Kubernetes-style containers. This seems clear even if you ignore technical implementation details - if only because so much of the work is done upstream by developers of the apps (using your example, there's an official plex docker container). So (ideally) the "App/Plugin" is wrapping a "nice" interface and some minor QA around an officieal developer container.
Sandboxes don't deliver anywhere approaching that sort of built-in developer support. The images that the jailmaker pulls from are OS images, not app images, and that's by far the dominant paradigm of what's distributed for nspawn. They're persistent, "easy" to set up and tweak, and can run "whatever you want". You could distribute apps in them - but why would you want to? CORE plugins were essentially that, and ended up as a bit of a mess.
iX is very clearly marketing/presenting them as "hey SCALE has jail-like functions now too, CORE holdouts".
Leaving aside implementation details...apps are very clearly going to stay as Docker/Kubernetes-style containers. This seems clear even if you ignore technical implementation details - if only because so much of the work is done upstream by developers of the apps (using your example, there's an official plex docker container). So (ideally) the "App/Plugin" is wrapping a "nice" interface and some minor QA around an officieal developer container.
Sandboxes don't deliver anywhere approaching that sort of built-in developer support. The images that the jailmaker pulls from are OS images, not app images, and that's by far the dominant paradigm of what's distributed for nspawn. They're persistent, "easy" to set up and tweak, and can run "whatever you want". You could distribute apps in them - but why would you want to? CORE plugins were essentially that, and ended up as a bit of a mess.
iX is very clearly marketing/presenting them as "hey SCALE has jail-like functions now too, CORE holdouts".
marshalleq
Explorer
- Joined
- Mar 12, 2016
- Messages
- 88
I think you got confused about Truenas official Apps and associated catalogs (not Jailmaker as you seem to be suggesting above), because you were saying that we all see our own use cases, and then the rest of what you say is not really on topic I think. This was not meant to be a docker vs whatever other containerisation you happen to like discussion. So I am simply supporting my argument by pointing out that the way that the use cases for scale are presented is not exclusive to the enterprise / small business but includes the home user use case (e.g. Plex) which doesn't lend itself well to kubernetes etc. In this use case, it isn't really working well. You could potentially justify that if you want to argue the IX target market is enterprise 'ish' and thats why they're sticking to kubernetes then fair enough, but enterprise wouldn't be running plex. See my point? And then if enterprise were using it, they'd be doing some other kind of kubernetes anyway, not this hybrid type thing we've got.
Further, I agree, running multiple jails with multiple dockers is ridiculous, which is exactly my point and exactly why I wrote it. Yet if you accept that there are plenty of us that want docker and are used to having docker with it's individually abstracted apps, this is again my point. Why must we have all this extra complication simply because for some reason IX don't want to run docker.
But anyway since you raise it, I did see an official survey from IX on redit a few weeks ago, for what features we wanted next - the response was Docker + Compose was most desired by a small margin. I attribute that to the user base, also I didn't see it so I didn't vote. I think this also speaks to the user base quite accurately in that it's probably about 50/50 explaining why there's a bit of passion around the topic. :D
And to quote another person on this forum, "IMHO, there's something to be said for having jails that can start and stop independently of the others in addition to potentially having different versions of mono (as radarr goes dotnet and sonarr doesn't yet), python or whatever is needed. (yes I know you can theoretically mix versions in a single jail, but, blergh... not for me)". Couldn't agree more - seems like just additional, unnecessary complexity for the sake of what exactly I don't know yet.
I think the fact that they offer support for these containers and referring to them as official in that support thread (which I think used to be a label against them in the App Store thingy previously too) with plenty of not enterprise containers is evidence enough, but there have also been emails I've received, interviews to some extent and I'm sure we could dig enough of it up to make the point. Further worsening the wound and perhaps it's just bad English, but IX actually advertise that docker is included with scale in multiple places. Anyway, I am trying to figure out what your motivation is for arguing this point. Perhaps you can enlighten me, it doesn't seem related to the topic, but I could be wrong.
Sure, but not really the topic of this thread I think?
I think you're trying to say that possibly why ix isn't offering docker (despite advertising that they do) is because you think it would be too much maintenance for them? Which may be true, but they could add docker unsupported like the jailmaker script just as easy.
I think you're overthinking things a bit. This has nothing to do with the topic of the thread as far as I can see. Feel free to enlighten me though, I'm quite open to being wrong.
Yes, I think I suggested this in my original post.
This WAS one of my questions in the original post. This may be a reason why. It still doesn't however explain why IX is resisting docker.
And finally, this is quite a long thread now, my apologies if my rushed typing has come across offensive anywhere, it is not intended and would be a mistake. I love these good discussion where we can all learn from each other.
Marshalleq.
- Joined
- Dec 30, 2020
- Messages
- 2,134
But where did you got this ridiculous idea?
The typical use case is to install docker(-compose) in a jail and run all containers from there.
marshalleq
Explorer
- Joined
- Mar 12, 2016
- Messages
- 88
Because I was expecting it to run like all the other containerised apps that all the other systems have, including TrueNAS apps. The precedent is humongous. I can't think of any case outside of VM's that do it like this. Unless Core does - I don't know about core, I didn't think it did but perhaps it does. What else do you know of where you have to install a virtualised something inside of a virtualised something besides VM's? But also because that's the topic of this thread which I did say at the beginning. It's for those of us that want docker. It wasn't desired to be a discussion between which is best, or what way is the right way to do it or anything like that. That's because there are always different use cases - something you pointed out earlier.
But since you ask, when you take docker out of it, it's essentially like LXD right. Which you can build your own little whatevers inside. The fact that we are throwing docker inside there is really a bit silly even if it does work - taking some creative thought here I doubt it was initially intended to be for that. For that a good architecture would actually be to put the docker engine straight on the host like jailmaker and kubernetes have been. Which gets us right back again to the topic of this post.
But since you ask, when you take docker out of it, it's essentially like LXD right. Which you can build your own little whatevers inside. The fact that we are throwing docker inside there is really a bit silly even if it does work - taking some creative thought here I doubt it was initially intended to be for that. For that a good architecture would actually be to put the docker engine straight on the host like jailmaker and kubernetes have been. Which gets us right back again to the topic of this post.
- Joined
- Aug 19, 2017
- Messages
- 1,556
Jails aren't virtualization, its containment, built on chroot. It builds walls around a service, much like what iOS does with "apps".
Dockerspins up a new kerne and builds a lightweight virtual machine, giving the maintainer complete control over the environment. Jails are superior when you have control of the whole system, like Apple does. And the sales pitch for docker is that you can build and deploy services on any unknown architecture.
Docker
Last edited:
- Joined
- Nov 25, 2013
- Messages
- 7,776
No. Docker provides the illusion of a container built on top of cgroups and namespaces.
Docker Explainer: Namespaces & Control Groups
Ever wondered what is going on under the hood when running a container? This quick explainer definitely won't give you the full picture, but it is at least a good place to start!
syvarth.com
To be clear, I was literally responding to this.
You explicitly brought up the idea of Jailmaker as app and container replacement.
So in summary, it seems like you brought up jailmaker, but don't really have a concept of what it (or the analogous BSD jails that are central to doing non-storage stuff on core).
Jails and sandboxes are lightweight ways to create an independent namespace and run stuff into it (optionally with various extra security/isolation measures that might impose some additional performance cost). iX has decided for reasons (i.e. kubernetes removing support for docker) that they don't want docker as part of Truenas - they're creating an appliance and the reliability of that appliance depends on consistency - not installing a bunch of overlapping and possibly conflicting stuff.
They're a way that iX can give core users what they're used to that scale is lacking AND also allow scale folks to run "whatever they want" (which docker is a popular option) in a low-overhead way without interfering with the appliance that they've created (and its reliability).
Sandboxes allow for the end-user to create isolated environments for stuff (like docker, as an example) to run in its own environment, with its own dependencies, segregated from the host - and to do so without needing to partition off RAM and CPU resources for each sandbox.
(Most of) The whole point of sandboxes (caveat - we all see our own uses...) is to provide a less resource-intensive quasi-VM environment that doesn't require one to partition off specific CPU/RAM resources.
