VHS or BETA?

[brando56894]

Just wanna throw in my two cents since now I can finally use SCALE on my server since my HBA bug has been fixed. I'm in the process of setting everything up and noticed that there wasn't a GUI option available for docker-compose, which was slightly disappointing since I have 2 compose files written for my setup, one using Traefik as my reverse proxy and one using Nginx as my reverse proxy. I'm running SCALE on my home server, it's largely a media server which runs the usenet suite of apps (you guys know what I'm doing ), plex, jellyfin, nextcloud, and a webserver for remote access.

I'm a Linux System Engineer, but I have little experience with k8s and really little need for it in my home setup, docker is good enough for me. I shelled into my box and noticed that docker-compose is installed, but I'm assuming it's non-functional, or is it just not exposed in the UI?

Edit:

I'm confused on all of this k8s vs docker-compose argument because I just used docker-compose via the shell and it worked perfectly, is this all about docker-compose being exposed in the UI? If people want to manage docker containers easily just setup the portainer UI and go from there. From what I can tell there are very little management operations available in the current Manage Docker Containers section other than update and delete, so IMO using portainer is kinda necessary at least at this point.

The only containers that failed for me were Plex (because I have it setup in k8s) and nginx-proxy because the TrueNAS UI is running on port 80 and 443.

 

[Basil Hendroff]

I'm confused on all of this k8s vs docker-compose argument because I just used docker-compose via the shell and it worked perfectly, is this all about docker-compose being exposed in the UI? If people want to manage docker containers easily just setup the portainer UI and go from there. From what I can tell there are very little management operations available in the current Manage Docker Containers section other than update and delete, so IMO using portainer is kinda necessary at least at this point.

There are quite a few threads on SCALE and Docker/Docker-Compose within the forum. If you study these, you'll notice comments like 'You can do it, but it's not supported', 'Use of Portainer is not recommended', etc. I guess for me, the fundamental issue, and the reason I started this thread, is that I'm hoping for acknowledgement from the establishment that Docker-Compose will be supported in some form as SCALE matures preferably not via a VM, but natively. Whether Docker-Compose is exposed in the GUI or not is independent of the support issue.

 

[ornias]

the reason I started this thread, is that I'm hoping for acknowledgement from the establishment that Docker-Compose will be supported in some form as SCALE matures preferably not via a VM, but natively.

Aren't you quite aware that iX generally doesn't give acknowledgement what is and isn't going to be in future versions before even the first version is launched? So, while I appreciate the effort you put into it, I'm not so sure why you put your hopes up before even the first release is done? You know it isn't going to give the response you hope for, which is also basisically what iX responded "we're watching it and taking feedbackinto account but no promises yet".

---
That being said: After discussing the scentences in the devnotes that stated kubernetes could be disabled in the future, iX has decided to delete the old devnotes. They where left mostly by accident and didn't represent the current project state :)

Hope this atleast solves that end-less confusion a bit :)

 

[brando56894]

There are quite a few threads on SCALE and Docker/Docker-Compose within the forum. If you study these, you'll notice comments like 'You can do it, but it's not supported', 'Use of Portainer is not recommended', etc. I guess for me, the fundamental issue, and the reason I started this thread, is that I'm hoping for acknowledgement from the establishment that Docker-Compose will be supported in some form as SCALE matures preferably not via a VM, but natively. Whether Docker-Compose is exposed in the GUI or not is independent of the support issue.

Yeah, I can see where you would want their blessing that it's not going to be killed sometime in the future if you're running TrueNAS in an enterprise setup, but for me, a home user, this doesn't matter to me at all. I agree that running everything in a VM isn't the best approach and native access would be preferred, I've always had performance issues with NFS compared to native access to the files. On the other hand, I work for a massive multimedia streaming company that you or someone you know probably subscribes to (I think we just hit 100M subs) and about 80% of our infrastructure is run in VMs, but kubernetes and docker, and we have zero issues serving tons of content. Granted we're running RedHat Virtualization Manager and oVirt (essentially the same thing), and this is Debian based, but it's still Linux at its core so the differences are minimal.

 

[Basil Hendroff]

A reminder that this poll will close in a few days time on 1-Sep UTC+0. Get your vote in while you still can.

 

[Basil Hendroff]

The poll will close in around 35 hours. This will be your last opportunity to cast a vote.

 

[j_r0dd]

The intention is that SCALE will be OCI compliant and will be able to scale. The question is whether SCALE should natively enable docker support and enable docker-compose. If there are any OCI compliance issues, then please document them as bugs.

Now that I'm balls deep in k3s I don't see a need for docker at all, but until upstream overlayfs is added for ZFS we are kind of stuck using docker as the runtime for k3s instead of containerd.

 

[inman.turbo]

but until upstream overlayfs is added for ZFS we are kind of stuck using docker as the runtime for k3s instead of containerd.

Thanks for mentioning this.

 

[ornias]

Now that I'm balls deep in k3s I don't see a need for docker at all, but until upstream overlayfs is added for ZFS we are kind of stuck using docker as the runtime for k3s instead of containerd.

Huh? OpenEBS should work on both just fine and k8s itself migrated to storageplugins like openEBS years ago.
I really not see what overlayfs has to do with it.

 

[j_r0dd]

Huh? OpenEBS should work on both just fine and k8s itself migrated to storageplugins like openEBS years ago.
I really not see what overlayfs has to do with it.

Currently a ZFS dataset is created for each layer in a container image because of the lack of overlayfs. This is causing massive slowdowns over time and from what I’ve read that overlayfs is needed for the containerd runtime. There’s a jira ticket regarding this as well k3s and OpenZFS GitHub issues in regards to overlayfs support for ZFS.

 

[ornias]

Currently a ZFS dataset is created for each layer in a container image because of the lack of overlayfs. This is causing massive slowdowns over time and from what I’ve read that overlayfs is needed for the containerd runtime. There’s a jira ticket regarding this as well k3s and OpenZFS GitHub issues in regards to overlayfs support for ZFS.

Ahh that explains things a lot more, thank you!
I wasn't thinking about supporting ZFS for container storage, only for data storage, but indeed obviously both need to be stored that way.

 

[Basil Hendroff]

This poll has now closed. The poll ran from 14 Aug 2021 UTC+0420 to 1 Sep 2021 UTC+1. It ran for just over two weeks and garnered 34 votes over that time.

A couple of observations:

1. During the polling period, votes for Kubernetes and Helm charts never surpassed votes for Docker and Docker Compose.
2. While the sample set isn't large, there is sufficient evidence, based on the poll result and discussion thread, for a case to support, at the very least, Docker Compose in some form.

 

[Jip-Hop]

Shoot I missed the poll! Would have voted for Docker and Docker Compose. I'm using macvlan networks for my Docker containers at home. That way they all get their own ip address, I don't need to mess with ports (no conflicts) and I can manage policies for each container on my router (only some containers should be routed through the VPN on the router). This all works great on my Synology NAS, using docker-compose.

Please correct me if I'm wrong, but this setup currently seems impossible to replicate using Kubernetes/Helm charts. Hence the desire to use Docker (with docker-compose).

So it's not only the fact that there are more resources and examples on how to use Docker containers out there. Docker also offers features that the alternatives don't.

I hope there will eventually be a way to run docker-compose alongside the official Apps, without them (possibly) interfering with each other. From reading Requesting docker-compose example I go the impression that it may be possible to 'untangle' them...

I got introduced to Docker and docker-compose by my Synology NAS. Now I'd like to upgrade to TrueNAS SCALE :) But unfortunately Docker on TrueNAS SCALE isn't what I was expecting... Of course I could stick with Synology, but I don't want to miss out on the full disk encryption, ECC ram and the fact that TrueNAS SCALE is open source!

 

[Ixian]

There's a number of issues:

The most serious (IMHO) is that TrueNAS uses the docker zfs storage driver, which seems like a "duh" choice, I know - the zfs storage driver leverages native zfs snapshotting, etc. - but you need to know what you are doing if you are managing docker on your own because their are a number of caveats you need to be aware of with volume management, etc. There's also a well-known performance problem with the docker zfs storage driver and uncached layers that's been around for years: https://github.com/moby/moby/issues/31247 . I don't know that it affects what Ix is doing with K3s/Apps but it will affect you if you run docker/docker-compose "native" on your own.

To get around the latter, most users (those with other Linux/OpenZFS systems) just create a zvol, format it with ext4, and mount the docker data directory on it using the Overlay2 FS driver (you can use bind mounts to store your persistent container data wherever you like) but that's not an option with SCALE - for one you can't format a zvol from the CLI (SCALE expects an external OS - such as one you attached a zvol to through iscsi, or a VM zvol - to handle this) and even if you do manage to format it there's no "official" way to mount a non-zfs volume in Truenas (same for Core and SCALE). And Overlay2/etc. don't work on top of ZFS filesystems, you are stuck using the zfs storage driver with Docker.

So that's a problem, one that at the very least requires heightened care managing and cleaning up volumes, etc. and at worst would have you staring at scripting workarounds that will likely break with any change.

Another issue is the version of Docker-Compose that SCALE ships with is 1.25, which has some drawbacks/lack of support for certain functions. The current stable version of Compose is 1.29.2. You can't remove the one SCALE ships with - the apt package has dependencies to the truenas package itself and you will break your install. So you can't install a newer version of compose with pip/etc.

Ix may just get rid of the Docker-Compose package completely (actually, I have no idea why they include it today, it isn't used anywhere I know of), which would sort of be a benefit as you could then potentially install a newer version - but now we're back to workarounds outside the middleware/GUI which as we all know is a big no-no for Truenas.

All in all there are just too many drawbacks to running docker/docker-compose the DYI route, today. I've switched to testing on a VM, which works, though it has the usual problems with NFS mounts for storage (you can basically forget about using NFS to mount persistant storage for apps that have databases/etc.) and the always-fun permission/uid/gid syncing dance. Also, in the current beta, Virtualization support is still underdeveloped - you can't import/export virtual disks, you can't pass through USB devices (except by passing through the entire PCI controller), and so on, though at least that is being worked on, with several open Jira issues.

 

[Patrick M. Hausen]

I'm using macvlan networks for my Docker containers at home. That way they all get their own ip address, I don't need to mess with ports (no conflicts) and I can manage policies for each container on my router (only some containers should be routed through the VPN on the router).

You know you get this with jails on CORE, too?

 

[Ixian]

You know you get this with jails on CORE, too?

The two are so different otherwise I don't see how it's relevant.

 

[Jip-Hop]

You know you get this with jails on CORE, too?

That's great! One more reason to want it on SCALE too, haha ^^

 

[Jip-Hop]

I made a Jira ticket requesting future compatibility with Docker. Hopefully it will draw some attention. Would be great if using Docker on TrueNAS SCALE becomes less risky...

even if you do manage to format it there's no "official" way to mount a non-zfs volume in Truenas

No official way, but mounting the ext4 formatted zvol with a pre-init script via "Add Init/Shutdown Script" would work right?

 

[Ixian]

I made a Jira ticket requesting future compatibility with Docker. Hopefully it will draw some attention. Would be great if using Docker on TrueNAS SCALE becomes less risky...



No official way, but mounting the ext4 formatted zvol with a pre-init script via "Add Init/Shutdown Script" would work right?

Yes, I did this for a while, but found it problematic - timing-wise pre-init is too soon and post-init too late.

Possibly the better route would be to inject the mount in to the source of /etc/fstab like has been proposed for modifying the daemon.json file but that is getting even further off the supported route.

What I do today is just use the default ix-applications docker data directory on zfs and run a simple garbage collection docker that I've scheduled to prune nightly. Since I use bind mounts for all my persistent container volumes there's no risk doing this and it keeps the docker volume layers cleaned up so in turn zfs snapshots for each layer don't pile up.

I snapshot my container bind volumes - I put them all in an appdata directory - via Truenas's built in snapshot and replication tools. Very simple to rollback a container if necessary that way.

For the cleanup docker, I use this one: clockworksoul/docker-gc-cron - though there are others.