How did we get to such a ludicrous position with containerisation?

[marshalleq]

I mean, we have the 'apps' section which has two catalogs as far as I know, one that has limited apps and one that makes some annoying decisions for you, both of which have at times some pretty serious problems and limitations as documented elsewhere in this forum.

Then we have the custom apps which is where we add something that is not on the list, usually where people refer to the place where they can 'finally get what they want, docker'. But it runs on an engine which is not docker and is encapsulated by Kubernetes causing more problems as documented elsewhere in this forum. Among these problems a particular killer for me is not being able to set a hostname. And I'll just throw out another example of the term ludicrous by mentioning the limitation on ports lower than 9000.

Then we have VM's which seems to be the last ditch effort for people to get sanity for example with docker, which does work well but has other distinct disadvantages, as documented elsewhere on this forum (RAM, Disk, Shares, Permissions, Accounts etc)

Now, excitement abuzz, we have Jailmaker, the latest solution to this containerisation problem being thrown around, IX-Systems have officially added the service built in to TrueNAS and say that it isn't currently supported, but they promise upgrades won't break it. Yay people say, now I can run docker that way!
But it's also fraught with problems which may not be well documented so here are a few:

• It's a container in a container requiring double entry of all shares
• It's a container in a container requiring triple entry of all permissions
• it's a container in a container requiring triple entry of all accounts
• If we create separate jails for all our apps (say we have 20 apps), we now must run 20 simultaneous instances of docker engine, WTF.
• If we don't create separate jails for all our apps, we're basically the same as running a VM with some different trade offs (Ram, Disk, Portability)

So, if this jailmaker script was suddenly unofficially endorsed, I assume to give some relief to people wanting a resolution to this ludicrous containerisation problem, possibly to entice some TrueNAS CORE people to come over to the dark side of linux as well because it 'sounds like' BSD jails, all so that we can run docker, why don't IX Systems just listen to the silent screams everywhere and just add docker engine?

Don't get me wrong, Jailmaker is great for some smaller use cases, but it isn't docker. And it has no docker hub. Sure it has the potential to replace TrueCharts, but it will end up probably exactly the same because IX Systems aren't in the business of ensuring we have a massive App Library.

So here we are again with no REAL solution, just a WORKAROUND and now we just get to juggle one more option with different tradeoffs and we just get to choose which one we will tolerate the most for a particular scenario.

I mean what happened to TrueNAS in the past that made IX-Systems so resistant to docker? Do they just have a bee in their bonnet about it? Or was there some 'event' that happened that made IX-Systems resist docker so much? Perhaps it's a strategy they have we don't know about? Or perhaps it WAS the strategy with clustering etc which is now no longer applicable? Does anyone know?

Please note: This is not intended to be a Kubernetes, vs Jailmaker vs VM vs Docker or anything else discussion. This is intended to be a discussion for those of us wishing there was proper docker support along with perhaps some understanding of how we got to here and why.

Thanks,

Marshalleq

 

[garm]

culture is the answer.. android anarchy vs iOS meritocracy
the good and the light is

Code:

nas% iocage create $_jailName
nas% iocage exec $_jailName 'pkg install <appName>'

 

[asap2go]

Before jails ever get popular we'll go to wasm and finally the jvm again.

 

[Stux]

There’s a few rewsons why many groups are attempting to replace docker with alternatives,

podman or containerd.

Sometimes it’s because of the root requirements, or the docker socket. Or just because they don’t like the corporate behind it.

Suffice to say, there is a trend to minimize dependency on docker itself, but not on the fundamental container technology.

Kubernetes also has that same trend internally.

Meanwhile, docker is fairly intrusive and I fully understand ix’s desire to not have it in the base os.

The systemd-spawn/jailmaker tech basically allows to have a lightweight alternate Linux distribution/installstion overlayed over the base os.

Which is a perfect solution to allow native level docker installation, without actually contaminating the base os.

And it is t just docker. It could be lxc/lxd/incus or podman.

Or all of the above.

Or Kubernetes!

Or whatever.

Meanwhile, I understand why Ix uses a k3s system to run their apps.

K8s/k3s provide an interesting capability at the cost of complexity and cpu usage.

BUT there are things you can do in compose, which can’t dieectly be migrated to k8s.

 

[Etorix]

I suspect that the reason is NOT to be found in the technical realm.

SCALE was supposed to… scale out, with distributed apps and distributed storage. To handle distributed apps, iXsystems went with Kubernetes, and got some fire because this is too complicated for home users who just want containers with standard Linux recipes (not to mention the infamous "x % CPU just idling"). To handle distributed storage, iXsystems went with Gluster. So far, so consistent.
Then Gluster went to nowhere and iX was left with a fishy OS which will NOT scale and an over-complicated way to run containers. Jailmaker, if it can grow to a full-fledged jail system for SCALE, can be a way to de-emphasise, or even phase out, Kubernetes.

 

[Patrick M. Hausen]

Then Gluster went to nowhere and iX was left with a fishy OS which will NOT scale and an over-complicated way to run containers. Jailmaker, if it can grow to a full-fledged jail system for SCALE, can be a way to de-emphasise, or even phase out, Kubernetes.

I guess I'll just go pet my FreeBSD servers for a while ... What a clusterf* in the literal sense.

 

[Stux]

I suspect that the reason is NOT to be found in the technical realm.

SCALE was supposed to… scale out, with distributed apps and distributed storage. To handle distributed apps, iXsystems went with Kubernetes, and got some fire because this is too complicated for home users who just want containers with standard Linux recipes (not to mention the infamous "x % CPU just idling"). To handle distributed storage, iXsystems went with Gluster. So far, so consistent.
Then Gluster went to nowhere and iX was left with a fishy OS which will NOT scale and an over-complicated way to run containers. Jailmaker, if it can grow to a full-fledged jail system for SCALE, can be a way to de-emphasise, or even phase out, Kubernetes.

Heh.

No reason to not run the apps ecosystem in a jail ;)

Well aside from not being able to nfs mount.

 

[Patrick M. Hausen]

Well aside from not being able to nfs mount.

There are local loopback/nullfs mounts on Linux?

 

[Stux]

There are local loopback/nullfs mounts on Linux?

I believe a limitation of the sandbox system is you need to mount any nfs mounts on the host system, not in the sandbox.

Docker does the same thing if you think about it. Mounting on the host and then looping into the guest. Which in docker is really just another chrooted fs.

 

[garm]

Before jails ever get popular we'll go to wasm and finally the jvm again.

You still have to deploy wasm's somewhere

Meanwhile, I understand why Ix uses a k3s system to run their apps.

Personally I think they started out with HA in mind (Scale?) and then things started derailing.. and now they are stuck in a bad configuration with heavy focus on patching together a debian based Unraid for the masses who want to run plex and missing the core value of the original business idea, an open source NAS built on the planets best file storage tech.

clusterf* in the literal sense.

 

[Patrick M. Hausen]

I believe a limitation of the sandbox system is you need to mount any nfs mounts on the host system, not in the sandbox.

Two points:

1. If there is a local nullfs equivalent, you definitely don't want NFS. NFS does not provide 100% POSIX file system semantics. This is one of the main features that make jails on FreeBSD so much superior to VMs. You can run a MariaDB server in one jail and mount the directory/dataset with the socket into N other jails, then have them communicate with the DB over a Unix domain socket instead of over TCP. To name just one example.

2. NFS or local nullfs mount - of course you need to mount from the host when the jail/container/whatever is started. That's the point. But why is this a problem? Just do it.

 

[Stux]

So what is the current replacement for glusterF*?

 

[Stux]

Two points:

1. If there is a local nullfs equivalent, you definitely don't want NFS. NFS does not provide 100% POSIX file system semantics. This is one of the main features that make jails on FreeBSD so much superior to VMs. You can run a MariaDB server in one jail and mount the directory/dataset with the socket into N other jails, then have them communicate with the DB over a Unix domain socket instead of over TCP. To name just one example.

2. NFS or local nullfs mount - of course you need to mount from the host when the jail/container/whatever is started. That's the point. But why is this a problem? Just do it.

Yes, you can null mount host directories into the sandbox.

But what happens when you want to mount an nfs mount off another system?

I guess you just add a startup mount command directly to the jails fs.

 

[sfatula]

I have exactly zero issues with custom apps, never have had any. You can get around the limitation of ports for a specific app that need < 9000 by using it's own IP. I've not needed hostname but that can be an issue if your app needs it yes. They are rock solid, bulletproof even.

I'll be very unlikely to even consider using the new "jails" as I don't see any need, at least yet.

 

[Patrick M. Hausen]

But what happens when you want to mount an nfs mount off another system?

You mount from the host. In CORE there is iocage fstab for that.

 

[garm]

So what is the current replacement for glusterF*?

ceph I presume

 

[Stux]

You mount from the host. In CORE there is iocage fstab for that.

Which is missing currently from jailmaker.

Guessing similar functionality could be added.

 

[mattsteg]

Don't get me wrong, Jailmaker is great for some smaller use cases, but it isn't docker. And it has no docker hub. Sure it has the potential to replace TrueCharts, but it will end up probably exactly the same because IX Systems aren't in the business of ensuring we have a massive App Library.

I think we all see our own use cases by default. Saying "it's not docker" seems silly - it doesn't pretend to be. And the whole conceit of running multiple jails with multiple docker's is ridiculous. If you want to run docker, set up a jail for it. Also not really clear how it would "replace" truecharts in an immediate sense.

I think the history of jails and plugins on CORE is a good reminder of what jails are really good for (easily spinning up simple to moderate stuff you roll yourself) vs. what docker and "plugins" do or should excel at (replicating another setup in a clean and repeatable way across a large community, including across upgrades).

 

[sretalla]

Which is missing currently from jailmaker.

But aren't you missing bind mounts? are they somehow not enough?

 

[Stux]

But aren't you missing bind mounts? are they somehow not enough?

Yes. I think its enough, and I also think I was conflating and getting confused between BSD and Linux terminologies.

Jailmaker / systemd-nspawn does everything I *need*