Docker on TrueNAS scale

[destate]

First of all, I'd like to acknowledge the number of posts on this forum regarding docker-compose and other docker related topics on TrueNAS scale. I've read through those and I still have questions.

I'd like to deploy a few services on my TrueNAS Scale machine such as Syncthing and an NVR service. I'd like to focus on both security/isolation of those services as well as the portability of the setup in case I want to deploy these solutions on different hardware down the road (likely not running on top of TrueNAS scale in future deployments). For these reasons, as well as my familiarity with docker, I'd like to deploy these services as docker containers via dockerfiles and/or docker-compose files. I really want to specify those services as dockerfiles because I value the documentation/version control aspects of having a small (and portable) file defining my deployment.

Now we get to TrueNAS scale: it sure looks like the options I'm defining in a "custom app" are options that could be defined in a docker-compose file, so I naively thought there would be some docker-compose action running under the hood. But alas, upon taking a peek at how the apps run, I don't see any docker at all. Just Kubernetes (which I'll admit up front, I know absolutely nothing about).

Is there any way for me to specify my "app" via a docker-compose file and have the TrueNAS Scale environment manage the "app"? And by manage, I mean ensure it gets started when the system boots and ensure it remains running (or at least notices when it fails to run or crashes)? I know I could do all sorts of manual things like ssh'ing in and setting up my own crons/scripts, but I'm interested to know if there are any more options native to this whole "Scale" paradigm.

Lastly, just a quick note: Yes I know there are "apps" that already exist for syncthing and NVR applications. I could just click "install" on those and be done with this. But again, for the sake of portability away from the TrueNAS scale environment one day as well as my own human-readable documentation of how these services are configured, I'd prefer to custom roll a yaml file.

 

[LarsR]

With Cobia, Docker was removed from the base OS. Docker was just used as the container runtime. The apps system on scale was always k3s and docker as backend. But k3s switched form docker to containerd as runtime so docker was removed. There's no easy way to do what you want, but there are some ways to run docker on truenas.
1. Set up a linux vm and run your apps there
2. The Docker-Compose app from truecharts
3. The jail-maker script https://github.com/Jip-Hop/jailmaker

 

[majorgear]

Docker in a VM is probably the best way to go.

 

[NugentS]

1. Set up a linux vm and run your apps there
2. The Docker-Compose app from truecharts
3. The jail-maker script https://github.com/Jip-Hop/jailmaker

I do 1 & 3
So far I much prefer 3 - which works extremely well

 

[destate]

Regarding option #2, I've seen others propose the docker-compose app from truecharts, but I don't see that app. Did that get removed from the truecharts catalog?

 

[LarsR]

The docker-compose App has been depreciated as far as i know

 

[NugentS]

OK - then how about the portainer app on Truecharts?

 

[destate]

Yep, looks like portainer is on there. However, I'm a bit confused as I've never used Portainer before. Once I've logged into the web interface of portainer, how can I tell it to run my docker-compose file? And how can I have portainer always make sure this container is running (even starting the container when TrueNAS boots)?

 

[NugentS]

Create a stack file - thats essentially the docker compose

 

[destate]

Ope... I get it... Sorry that took me a minute. For any others like me out there:

- I started the portainer app and ran it with a GID matching "docker" so it could access the docker.sock. I also added "Additional App Storage" (type: Host path) to map /var/run/docker.sock to /var/run/docker.sock in this portainer node.
- I logged into the Portainer web UI, made an account, and noticed that it was already connected to my kubernetes instance and can therefore run/manage/view nodes in there. I needed to link it to my docker daemon already running on my NAS, so I added a new "Environment" (type: docker), used the "socket" method to connect to my docker instance, gave it a name, and then I was connected. This gave me the ability to run/manage/view my docker containers through portainer.
- Lastly, I can create "stacks" in that "environment" which just point to my docker-compose files.

Very cool stuff. Thanks for your patience, @NuggetS!

 

[MisterE2002]

Ope... I get it... Sorry that took me a minute. For any others like me out there:

- I started the portainer app and ran it with a GID matching "docker" so it could access the docker.sock. I also added "Additional App Storage" (type: Host path) to map /var/run/docker.sock to /var/run/docker.sock in this portainer node.
- I logged into the Portainer web UI, made an account, and noticed that it was already connected to my kubernetes instance and can therefore run/manage/view nodes in there. I needed to link it to my docker daemon already running on my NAS, so I added a new "Environment" (type: docker), used the "socket" method to connect to my docker instance, gave it a name, and then I was connected. This gave me the ability to run/manage/view my docker containers through portainer.
- Lastly, I can create "stacks" in that "environment" which just point to my docker-compose files.

Very cool stuff. Thanks for your patience, @NuggetS!

You mean, you use this on pre-cobia sytems?
I assume Portainer is just that: a web manager and not docker-in-kubernetes.

So only option 1 and 3 remain.
Option 1: Can not pass other datasets to vm?
Option 3: jail-maker is a hack because iX does not want to implement jails.

 

[danb35]

jail-maker is a hack because iX does not want to implement jails.

I don't think this--especially the "because" part--is correct. iX, and Morgan in particular, have been open to the concept but understandably cautious. Whether, and when, something like jails will show up in SCALE, I don't know, but it seems that a hard "they don't want to implement them" overstates the situation.

 

[MisterE2002]

I don't think this--especially the "because" part--is correct. iX, and Morgan in particular, have been open to the concept but understandably cautious. Whether, and when, something like jails will show up in SCALE, I don't know, but it seems that a hard "they don't want to implement them" overstates the situation.

Sure, but the ticket is open for a long time.

Is my conclusion about solution 2 correct? Otherwise i still do not fully understand all technicalities about container (engines).
I really prefer to use bind mounts and not ugly loopback share (samba/nfs) loopbacks and keep a throw-away host.
So, i think my only option is option 3?

 

[danb35]

Sure, but the ticket is open for a long time.

Yes, hence my saying they're "understandably cautious." I don't think it's going to happen soon, but I also don't think they're philosophically opposed to the idea.

 

[tprelog]

jail-maker is a hack because iX does not want to implement jails.

The ticket for this was recently accepted.

[NAS-119787] - iXsystems TrueNAS Jira

ixsystems.atlassian.net

The most recent comment by @Kris Moore states

Some good news for those waiting on this item. We will be including systemd-nspawn in the next major release, 24.04 (Dragonfish). We will be recommending folks utilize it with the jailmaker tool. (https://github.com/Jip-Hop/jailmaker)

Docs pages will be made available in the near future. This will be CLI only functionality, but if we see enough users utilizing we will consider it for a full blown UI feature at some later release.

 

[soleous]

I was just reading about nspawn and I find it interesting for sandboxing, however I moved freebsd jails onto docker a while ago for single applications. I have now moved containers to kubernetes due to SCALE, but I still feel the need to use docker and docker-compose for single applications in my workflow.

With kubernetes runtime moved from docker to containerd, its freed up docker. It would be nice to have a system simular to what I'm seeing for "Sandboxes" but for docker-compose files.

I'm super curious why this wasn't planned for docker especually as nspawn is introduced as a new feature?

 

[CJRoss]

As someone not familiar with nspawn, will that give me the full network availability like jails did? Currently I have several different plex jails which all show up as their own network and IP. This works better than VMs because I don't have to put hard limits on their cpu or memory usage.

 

[Kailee71]

Yes, you get a choice of how you network your jails. Mine all get their own individual ip adresses through dhcp. Static is also no problem. I have a seperate physical NIC available for my jails, which I assign no IP to, then create a bridge on that, also without IP, and then the jails connect to that bridge and pull individual IP as mentioned. Works a treat.

To come back to the OP; I also am extremely satisfied with Jailmaker and suggest to try it out. Bung a container on, install docker (or whatever you like), and you're smiling.

 

[monovitae]

Between the two can you elaborate on why you prefer 3 so far?

I do 1 & 3
So far I much prefer 3 - which works extremely well

 

[Kailee71]

For me it's because the jails are super lightweight, extremely easy and quick to set up and remove, and give (nearly) all the flexibility of having a VM including GPU "passthrough". Networking is also great with individual bridge devices created and listed in the UI for monitoring. The only things I would love to see included is to have datasets created for each rootfs on creation (for individual snapshots), UI integration and possibly have the networking setup a little more streamlined for non-standard setups (but if you only need straightforward setups you're good to go as is).

But even in its current state Jailmaker really is a great addition to Scale - thank you especially to @Jip-Hop and @Kris Moore for your efforts!