Is TrueCharts really the future for TrueNAS or is plain docker going to be removed in the future?

[hiro5id]

After struggling with porting my stuff from docker-compose to True Charts, I am beginning to give up. I think i'm going to have to abandon TrueNAS. When I started using FeeNAS it was attractive because it uses ZFS being non proprietary and not locking me in. I was excited when I heard announcement FreeNAS move to Linux, and support docker as a first class feature. But very saddened to find out that I must use yet another layer, True Charts, and Kubernetes. And on top of that it forces me to use that ecosystem and doing plain docker is not an option, officially. Kubernetes is overly complicated for my needs, I just want a Linux based ZFS NAS that can also run docker & docker-compose. (I know that recently there was a release of a True Scale addon that allows to use docker-compose, but even that has limitations, for example I need to specify subnet that is the same as the subnet on which the NAS runs on) It is very frustrating to hear that TrueNAS may remove docker-compose altogether and make it even harder to use in future updates. It's one thing to not officially support it, it is quite another to be actively working against people who don't want to use True Charts + Kubernetes.

At least if I knew that un-officially docker will remain in TrueNAS in the future, it might make me want to stay, as I can hack each version to run what I need. But i'm hearing rumors that it might go away.

With that, I am looking at migrating my stuff to Open Media Vault. It supports ZFS and docker-compose without any fuss.

I'm sad because I have used TrueNAS since the early days, and I have been a big promoter of it to everyone that would listen to me. But the way decisions have been made about implementing docker support is just too rigid, and I don't want to be forced into another abstraction such as True Charts which is specific to TrueNAS. I need my stuff to be cattle not pets. If my TrueNAS system somehow breaks, or worse, TrueNAS goes away, I need to be able to start up my stuff on another machine easily and quickly without needing to install TrueNAS. ZFS and plain docker allows me to be portable. But coupling myself to True Charts and on top of that complications from Kubernetes is just too much.

 

[Glowtape]

Has there been some new statements? I can only find this so far from the truecharts username:

For the same reason no guarantees are given by iX Systems either way (removing or keeping) when it comes to docker-compose.
It simply just "happens to be there" due to the chosen container backend for kubernetes.

You can use plain Docker on TrueNAS Scale right now. You just need to remember backing up and restoring the /etc/docker/daemon.json and reenabling the Docker service every OS upgrade. That's what I'm doing now.

That said, if it gets removed, I'm also out. I find the idea of Kubernetes for single node systems kinda offending, because I don't see what it offers over pure Docker in that context, other than wasting some more energy (backplane thread keeping a CPU core at full blast).

 

[bigjme]

This may not help but I am planning to move over from a dedicated docker machine and pulling everything into truenas scale to consolidate hardware. Like yourself, I am using docker-compose right now and refuse to move to Kubernetes as it is just too much for what I need

My intention (I have tested this on a spare machine), is to run the TrueCharts docker-compose app and have it install Portainer CE
I will then use Portainer to keep to using docker-compose. I have tried this with Vlans etc. and it all does seem to work

I have not tested performance compared to bare metal and it is an extra step I'd prefer not to need but it does run
Running a LibreSpeed docker and testing via Chrome on another machine, I can easily max out a normal gigabit network connection but this is still docker-in-docker which is not really recommended

Regards,
Jamie

 

[hiro5id]

The problem I’m finding with that is that I’ve read there is a limitation on the network settings in that you cannot specify a Docker network subnet that is the same as your host’s subnet. I need that capability for my workloads. With plain Docker-compose I do not have that limitation.

And of course, it just feels dirty to have the Docker-in-Docker situation. The extra unnecessary layers drives me crazy. When I need to squeak out every little bit of performance from my consolidated hardware I cannot afford to lose performance over unnecessary layers.

It is precisely the reason I was happy to see FreeNAS move to linux, because it meant that I no longer had to run Docker within a VM like before. One extra layer removed Docker running on Linux natively was exactly what I wanted. Only to be hugely disappointed that we’re being forced into going through yet another layer, Kubernetes and True Charts. Admittedly not as thick of a layer as a VM, but a layer nonetheless.

I’m aware of TrueNAS team’s stance on this, but I’m really hoping they see these threads that this is a genuine problem for a large majority of us, and hope that they reconsider leaving docker-compose in the system. If not officially, then at the very least un-officially.

This may not help but I am planning to move over from a dedicated docker machine and pulling everything into truenas scale to consolidate hardware. Like yourself, I am using docker-compose right now and refuse to move to Kubernetes as it is just too much for what I need

My intention (I have tested this on a spare machine), is to run the TrueCharts docker-compose app and have it install Portainer CE
I will then use Portainer to keep to using docker-compose. I have tried this with Vlans etc. and it all does seem to work

I have not tested performance compared to bare metal and it is an extra step I'd prefer not to need but it does run
Running a LibreSpeed docker and testing via Chrome on another machine, I can easily max out a normal gigabit network connection but this is still docker-in-docker which is not really recommended

Regards,
Jamie

 

[Arwen]

This clearly demonstrates the problem with containers today, their are way too many. Or too many management methods. A single smaller vendor can not support all types. They have to choose one or a sub-set of them to support. Even IBM likely does not support them all.

While it would be nice to support docker & docker compose, iX made what they think is a good decision for their Enterprise customers. We freebee users simply get the benefit, (or lack of benefit), from those decisions. I am not trying to justify it, but to remind people "their are too many container choices today".

As for trying to get iX to officially include docker & docker compose, that would then require both a business case, and resources to implement & test, each and every TrueNAS SCALE release.

 

[morganL]

After struggling with porting my stuff from docker-compose to True Charts, I am beginning to give up. I think i'm going to have to abandon TrueNAS. When I started using FeeNAS it was attractive because it uses ZFS being non proprietary and not locking me in. I was excited when I heard announcement FreeNAS move to Linux, and support docker as a first class feature. But very saddened to find out that I must use yet another layer, True Charts, and Kubernetes. And on top of that it forces me to use that ecosystem and doing plain docker is not an option, officially. Kubernetes is overly complicated for my needs, I just want a Linux based ZFS NAS that can also run docker & docker-compose. (I know that recently there was a release of a True Scale addon that allows to use docker-compose, but even that has limitations, for example I need to specify subnet that is the same as the subnet on which the NAS runs on) It is very frustrating to hear that TrueNAS may remove docker-compose altogether and make it even harder to use in future updates. It's one thing to not officially support it, it is quite another to be actively working against people who don't want to use True Charts + Kubernetes.

At least if I knew that un-officially docker will remain in TrueNAS in the future, it might make me want to stay, as I can hack each version to run what I need. But i'm hearing rumors that it might go away.

With that, I am looking at migrating my stuff to Open Media Vault. It supports ZFS and docker-compose without any fuss.

I'm sad because I have used TrueNAS since the early days, and I have been a big promoter of it to everyone that would listen to me. But the way decisions have been made about implementing docker support is just too rigid, and I don't want to be forced into another abstraction such as True Charts which is specific to TrueNAS. I need my stuff to be cattle not pets. If my TrueNAS system somehow breaks, or worse, TrueNAS goes away, I need to be able to start up my stuff on another machine easily and quickly without needing to install TrueNAS. ZFS and plain docker allows me to be portable. But coupling myself to True Charts and on top of that complications from Kubernetes is just too much.

I'm not sure where you are getting your rumors from?

Docker containers is supported through the UI.

Docker-Compose is currently a TrueCharts App. It's possible we could make it an official App.... does that address your needs?

However, its not possible to run Kubernetes and Docker Compose natively at the same time... and avoid confusion/issues.

It is also possible to run a Linux VM with Docker Compose as well. Some limitations.
It's also possible to run Docker on a separate machine with CSI driver.

If the issue is networking... I'd suggest writing up your requirements and posting here before making a suggestion in the "report-a-bug". We want to solve issues, but there are some constraints.

 

[bigjme]

I know this is not an issue for me but I would love an official app for this. While TrueCharts is great and are amazing in itself, I would like to see something officially supported and configured how iX would prefer, something you can install without the overhead of the truecharts catalogue download, even if it just installed and had portainer baked in (portainer with docker-compose options)

 

[morganL]

I know this is not an issue for me but I would love an official app for this. While TrueCharts is great and are amazing in itself, I would like to see something officially supported and configured how iX would prefer, something you can install without the overhead of the truecharts catalogue download, even if it just installed and had portainer baked in (portainer with docker-compose options)

I like the idea... can you make a suggestion and give people the bug-ID to upvote it.

The TrueCharts team did the work to make it possible...... we'd like the community to recognise their contribution and support them.

 

[hiro5id]

I'm not sure where you are getting your rumors from?

Docker containers is supported through the UI.

Docker-Compose is currently a TrueCharts App. It's possible we could make it an official App.... does that address your needs?

However, its not possible to run Kubernetes and Docker Compose natively at the same time... and avoid confusion/issues.

It is also possible to run a Linux VM with Docker Compose as well. Some limitations.
It's also possible to run Docker on a separate machine with CSI driver.

If the issue is networking... I'd suggest writing up your requirements and posting here before making a suggestion in the "report-a-bug". We want to solve issues, but there are some constraints.

Thank you for chiming in.

The issue is that it’s extra abstraction layers to go through. Even the Docker-compose in TrueCharts is still a Docker-in-Docker situation. And I have read that there is a few caveats such as not being able to define a subnet that is the same as the host’s subnet. Which is exactly what I’m currently doing with plain vanilla Docker-compose.

I am aware that I can run Docker within a VM. That’s what I have been doing back on old FreeNAS. But it always bothered me because disk IO was unnecessary slow. And also I can’t directly mount a ZFS dataset into the Docker container. This is the reason I was happy to see FreeNAS move to linux, because it meant that I no longer had to run Docker within a VM like before. One extra layer removed, Docker running on Linux natively was exactly what I wanted. Only to be disappointed that we’re being forced into going through yet another layer, Kubernetes and True Charts. Admittedly not as thick of a layer as a VM, but a layer nonetheless, which brings its own caveats that I don’t want to list because I want to keep this brief and to the point.

Currently I have disabled the whole True Charts and Kubernetes thing on my TrueNAS SCALE and, I hacked a few files so I can run Docker-compose natively. It works beautifully. The problem is that with each update I have anxiety that it will go away and I won’t be able to hack it anymore to do that because it’s obviously not officially supported.

 

[bigjme]

You can have docker images with static ips or dhcp from your main network in the docker-compose container
I have 4 docker networks setup within the compose app; one on my local network ip (the same subnet truenas is on), and 3 for various vlans on my network

These can be setup within portainer or the way I do it is console into the docker-compose app and run the following:
docker network create -d macvlan --subnet=192.168.x.0/24 --gateway=192.168.x.1 -o parent=br0 Internal
docker network create -d macvlan --subnet=192.168.x.0/24 --gateway=192.168.x.1 -o parent=br0.20 Vlan20
docker network create -d macvlan --subnet=192.168.x.0/24 --gateway=192.168.x.1 -o parent=br0.21 Vlan21
docker network create -d macvlan --subnet=192.168.x.0/24 --gateway=192.168.x.1 -o parent=br0.22 Vlan22

br0 in my case is a bridged adapter setup on the Truenas host. I have tested Internal using DHCP and static IP's from my router and all work without issues. I have also tested the Vlan's although only with static ip's and all was fine there as well

The docker-compose app is configured in a way that allowed itself full access to the main TrueNas nic's and should therefore have no issues doing anything you can on a bare metal host (that I am aware of at least)

 

[bigjme]

Also, this is the link to the official docker-compose feature request:
https://ixsystems.atlassian.net/browse/NAS-118888

 

[hiro5id]

Even IBM likely does not support them all.

But Kubernetes is not another kind of container. Nor is True Charts. I work at IBM. What are you asserting? About a specific device? Or IBM cloud? Because we’re taking about an appliance here with TrueNAS. I feel like it’s comparing apples and oranges. IBM cloud supports all kinds of containers, especially Docker. Kubernetes is just abstraction on top of Docker. Yes Kubernetes can orchestrate other containers too. Docker is the most popular at the moment.

 

[jgreco]

I work at IBM. What are you asserting?

I suggest you go re-read what @Arwen wrote. It's pretty clearly stated. It is not practical to support all the things that people might like to be supported. Linux has a massive issue with reinventing the wheel and trying to outclever the next guy. The assertion that "even IBM likely does not support them all" is pretty clear.

 

[hiro5id]

You can have docker images with static ips or dhcp from your main network in the docker-compose container
I have 4 docker networks setup within the compose app; one on my local network ip (the same subnet truenas is on), and 3 for various vlans on my network

These can be setup within portainer or the way I do it is console into the docker-compose app and run the following:
docker network create -d macvlan --subnet=192.168.x.0/24 --gateway=192.168.x.1 -o parent=br0 Internal
docker network create -d macvlan --subnet=192.168.x.0/24 --gateway=192.168.x.1 -o parent=br0.20 Vlan20
docker network create -d macvlan --subnet=192.168.x.0/24 --gateway=192.168.x.1 -o parent=br0.21 Vlan21
docker network create -d macvlan --subnet=192.168.x.0/24 --gateway=192.168.x.1 -o parent=br0.22 Vlan22

br0 in my case is a bridged adapter setup on the Truenas host. I have tested Internal using DHCP and static IP's from my router and all work without issues. I have also tested the Vlan's although only with static ip's and all was fine there as well

The docker-compose app is configured in a way that allowed itself full access to the main TrueNas nic's and should therefore have no issues doing anything you can on a bare metal host (that I am aware of at least)

I'll have a look. When I watched this https://www.youtube.com/watch?v=QXooywQSfJY he mentioned that it is one of the limitations that you should not define a network that is the same as the host.

But is it as simple as being able to clone a git repo, and do `docker-compose up` in a terminal without any fuss? That's what I do now with vanilla docker-compose.

 

[bigjme]

It can be but you'd have to jump into the docker-compose app to do it, so one extra step

The network can be configured within the compose apps themselves

 

[hiro5id]

I suggest you go re-read what @Arwen wrote. It's pretty clearly stated. It is not practical to support all the things that people might like to be supported. Linux has a massive issue with reinventing the wheel and trying to outclever the next guy. The assertion that "even IBM likely does not support them all" is pretty clear.

I'm not looking to argue. I was asking for clarification. Because we are not talking about obscure stuff here, docker/docker-compose, Kubernetes, Helm... it's all de-facto standard stuff that IBM supports every one of those.

 

[hiro5id]

It can be but you'd have to jump into the docker-compose app to do it, so one extra step

The network can be configured within the compose apps themselves

Ok like I said i'll have a look. Not sure if you saw this, as I edited my last response and you may have missed it: When I watched this https://www.youtube.com/watch?v=QXooywQSfJY he mentioned that it is one of the limitations that you should not define a network that is the same as the host.

EDIT: Ah I misspoke..... it's not supposed to conflict with KUBERNETES networks... not the Host as I originally thought :).

Perhaps this has life afterall.... i'll need to take a look more deeply.

If this works out then, like @morganL said before, making Docker-Compose TrueCharts App an official app then that would probably help.

Thanks for the tips.

 

[bigjme]

I did just watch that video, I believe the reason they said not to is if you define an actual network, I.e. One that docker will provide dhcp for, it will cause ip conflicts between docker and your lan. The same as doing it conflicting with the kubernetes network range

What I stated above attached the docker to the nic itself, so your router would provide dhcp if a static was not provided, not docker itself, this should prevent dhcp conflicts

I hope that makes sense and helps a little?

 

[jgreco]

I'm not looking to argue. I was asking for clarification. Because we are not talking about obscure stuff here, docker/docker-compose, Kubernetes, Helm... it's all de-facto standard stuff that IBM supports every one of those.

I'd wager a small sum that IBM doesn't support jails, which are the precursor technology to Docker. Arwen's point is, as I read it, that there's a crapton of stuff that could be supported that some subsegment of the userbase would like to see supported, but that it is simply impractical for a small development team at a small NAS vendor to support more than a carefully selected few. It is disingenuous to say "Because we are not talking about obscure stuff here", because at some point someone WILL come up with a compelling reason that they want to see support for something that is fundamentally incompatible with other possible options, and that you will define as "obscure". Docker isn't really a one-size-fits-all technology; they've also had to include KVM in order to cope with a different kind of use case. Based on past observation, iXsystems is simply not going to be able to accommodate all the various things people might want to see supported.

 

[hiro5id]

I'd wager a small sum that IBM doesn't support jails, which are the precursor technology to Docker. Arwen's point is, as I read it, that there's a crapton of stuff that could be supported that some subsegment of the userbase would like to see supported, but that it is simply impractical for a small development team at a small NAS vendor to support more than a carefully selected few. It is disingenuous to say "Because we are not talking about obscure stuff here", because at some point someone WILL come up with a compelling reason that they want to see support for something that is fundamentally incompatible with other possible options, and that you will define as "obscure". Docker isn't really a one-size-fits-all technology; they've also had to include KVM in order to cope with a different kind of use case. Based on past observation, iXsystems is simply not going to be able to accommodate all the various things people might want to see supported.

I know what you are saying, of course iXsystems cannot support everything. For the scope of this particular thread though, we are talking about docker-compose. Which is certainly not something totally out of left field. Docker is already there because Kubernetes needs it -- unless it's decided to switch to a different container run time. Since docker is already on the system, it's not really going off into uncharted territory by asking for docker-compose as they are all in the same family.

I digress anyway, it's not a productive topic, I don't wish to argue. Sorry that I waded into that I should have ignored it.

I am thankful for some of the suggestions up top, I will give another try with the TrueCharts SCALE - Docker-Compose app, and see if I can make it work for my simple needs, without adding too much overhead and complications.