docker compose?

[dragon2611]

I see SCALE has docker-compose which I'm currently using to run some containers after getting scale running on a QNAP TS-653D (Upgraded to 20gb ram)
This was mostly done out of laziness as I copied the container config from the configs I was using on QTS, but also I'd rather not use K3s since whilst it's lightweight for kubernetes it still adds a fair bit of CPU usage and overhead and at the moment i'm not doing anything where I feel the extra overheads of running Kubernetes are justifiable

My question is mostly is it planned for docker / docker-compose to remain available on the unit for the foreseeable future especially as I think K3 defaults to another container runtime thesedays.

I should be able to script starting of the containers on reboot (Well for ones not an encrypted dataset at least)

 

[truecharts]

I see SCALE has docker-compose which I'm currently using to run some containers after getting scale running on a QNAP TS-653D (Upgraded to 20gb ram)
This was mostly done out of laziness as I copied the container config from the configs I was using on QTS, but also I'd rather not use K3s since whilst it's lightweight for kubernetes it still adds a fair bit of CPU usage and overhead and at the moment i'm not doing anything where I feel the extra overheads of running Kubernetes are justifiable

My question is mostly is it planned for docker / docker-compose to remain available on the unit for the foreseeable future especially as I think K3 defaults to another container runtime thesedays.

I should be able to script starting of the containers on reboot (Well for ones not an encrypted dataset at least)

SCALE does not officially support docker-compose.
There are a few tricks/hacks available, but it's not an officially supported feature.

For the same reason no guarantees are given by iX Systems either way (removing or keeping) when it comes to docker-compose.
It simply just "happens to be there" due to the chosen container backend for kubernetes.

 

[Ixian]

We aren't going to get much more of an answer out of Ix Systems than has already been provided (i.e. it's not "officially supported").

The Docker installation itself seem likely to stick around in some form since Ix does provide a way to run docker containers, albeit wrapped in K3s vs. leveraging the docker subsystems. Whether they keep docker-compose is anyone's guess.

You can use Linuxserver.io's Docker-Compose image as a docker container instead - I realize for folks new to this the idea of running Compose itself from a docker container might take a bit to wrap your heads around but it's perfectly functional and saves you from updating anything in the SCALE boot image: https://docs.linuxserver.io/images/docker-docker-compose

All you need to do is create a symlink to it (they provide instructions) and it runs the latest Compose just like it was native. You'll need to re-create the symlink each time SCALE updates (or script it) but it is simple enough.

 

[dragon2611]

Fair enough, I noticed after upgrading from RC2 to Release docker.json is no longer in /etc/docker I suspect it was created when I played with apps the first time, not a major biggie although I may recreate it so I don't fill up the root with container images.

Kubernetes is nice enough and i may replace the compose files as some-point with kubernetes configuration, although that still leaves the problem of CPU usage.

 

[Ixian]

Fair enough, I noticed after upgrading from RC2 to Release docker.json is no longer in /etc/docker I suspect it was created when I played with apps the first time, not a major biggie although I may recreate it so I don't fill up the root with container images.

Kubernetes is nice enough and i may replace the compose files as some-point with kubernetes configuration, although that still leaves the problem of CPU usage.

That's because it resets the App configuration when the system is upgraded. If you had unset the app pool before (or never set it) it won't initialize the various bits needed to support Apps, such as docker.

All you need to do is set, then unset, the App pool. This will recreate systemctl entries as well as the stock daemon.json file among others.

All you need to do then is mod the daemon.json to your needs (for most people this just amounts to removing iptables=false from it) and re-enable docker via systemctl enable docker and you're off and running. It'll stay this way, including during reboots, until you either A) Update SCALE to a new release or B) Set an app pool again, which will re-initialize the default settings (including resetting daemon.json).

Personally, I keep the pool un-set and by extension the Kubernetes-based App system disabled when I run vanilla docker/docker-compose on SCALE.

 

[truecharts]

You can use Linuxserver.io's Docker-Compose image as a docker container instead - I realize for folks new to this the idea of running Compose itself from a docker container might take a bit to wrap your heads around but it's perfectly functional and saves you from updating anything in the SCALE boot image: https://docs.linuxserver.io/images/docker-docker-compose

All you need to do is create a symlink to it (they provide instructions) and it runs the latest Compose just like it was native. You'll need to re-create the symlink each time SCALE updates (or script it) but it is simple enough.

Nice tip!
We might be able to do something cool with that as an App :-D

 

[Ixian]

Nice tip!
We might be able to do something cool with that as an App :-D

That could be interesting. I've been running it via docker for a while, it (Compose) is actually perfect for hosting in a container of its own

 

[bitbit]

That's because it resets the App configuration when the system is upgraded. If you had unset the app pool before (or never set it) it won't initialize the various bits needed to support Apps, such as docker.

All you need to do is set, then unset, the App pool. This will recreate systemctl entries as well as the stock daemon.json file among others.

All you need to do then is mod the daemon.json to your needs (for most people this just amounts to removing iptables=false from it) and re-enable docker via systemctl enable docker and you're off and running. It'll stay this way, including during reboots, until you either A) Update SCALE to a new release or B) Set an app pool again, which will re-initialize the default settings (including resetting daemon.json).

Personally, I keep the pool un-set and by extension the Kubernetes-based App system disabled when I run vanilla docker/docker-compose on SCALE.

Longtime docker-compose user migrating to Scale. At some point I'll likely dive down the Kubernetes rabbit hole. But for now, removing iptables=false from /etc/docker/daemon.json, enabling the docker service (systemctl enable docker), and this Init command should do the trick. Cheers!

Code:

curl -L --fail https://raw.githubusercontent.com/linuxserver/docker-docker-compose/master/run.sh -o /usr/local/bin/docker-compose && chmod +x /usr/local/bin/docker-compose && ln -sf /usr/local/bin/docker-compose /usr/bin/docker-compose && service docker restart && docker-compose -f /$YOUR_PATH/docker-compose.yml up -d --remove-orphans

 

[truecharts]

Longtime docker-compose user migrating to Scale. At some point I'll likely dive down the Kubernetes rabbit hole. But for now, removing iptables=false from /etc/docker/daemon.json, enabling the docker service (systemctl enable docker), and this Init command should do the trick. Cheers!

Code:

curl -L --fail https://raw.githubusercontent.com/linuxserver/docker-docker-compose/master/run.sh -o /usr/local/bin/docker-compose && chmod +x /usr/local/bin/docker-compose && ln -sf /usr/local/bin/docker-compose /usr/bin/docker-compose && service docker restart && docker-compose -f /$YOUR_PATH/docker-compose.yml up -d --remove-orphans

Please be aware this cannot be combined with SCALE Apps, in case you want to migrate on a later date.
We're currently working on finding a solution for the docker compose issue, that won't require these hacks. but we're not ready to be more specific at this time

 

[bitbit]

Please be aware this cannot be combined with SCALE Apps, in case you want to migrate on a later date.
We're currently working on finding a solution for the docker compose issue, that won't require these hacks. but we're not ready to be more specific at this time

Great to hear a solution may be in the works!

 

[bitbit]

TrueCharts has added docker-compose to their catalog. A huge THANK YOU to the TrueCharts team for their docker-compose implementation and all of their contributions to the community!!

• Return docker to its default state. For me that meant rm /etc/docker/daemon.json and systemctl disable docker
• Remove any applicable Init Scripts you previously added to enable a vanilla docker environment
• Reboot
• Enable TrueNAS Apps by choosing a pool
• Add the TrueChart Catalog
• Install docker-compose from the TrueCharts catalog
• Add your compose file to the Container Configuration and it will autostart (i.e. /mnt/jet/appdata/docker-compose/docker-compose.yml)
  Alternately, you can enter the docker-compose app shell and spin up a docker-compose stack (i.e. docker-compose -f /mnt/jet/appdata/docker-compose/docker-compose.yml up -d)

The below docker-compose.yml example will setup a Portainer container. All additional containers can then be configured within Portainer. As always, your mileage may vary.

Code:

version: '3'
services:
  portainer:
    image: cr.portainer.io/portainer/portainer-ce:latest
    container_name: portainer
    restart: always
    networks:
      - portainer
    ports:
      - "8000:8000"
      - "9443:9443"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /mnt/jet/appdata/portainer:/data

networks:
  portainer:
    external: false

Now that my containers are successfully running on TrueNAS SCALE, I'll be looking to migrate most of them to TrueNAS SCALE Applications. I could use some guidance on migrating existing, persistent app data from a dataset to TrueChart apps. Cheers!

 

[Ixian]

Props to the TrueCharts team for doing this! Going to give it a spin today.

 

[murzik]

Not sure what I am doing wrong. But can't get it to work. Gets stuck on deploying..

 

[bitbit]

Not sure what I am doing wrong. But can't get it to work. Gets stuck on deploying..

For support using TrueChart apps, please check out their Discord channel.

 

[Ixian]

• For my purposes, I then setup an external network (docker network create proxy). This may need to be added to a TrueNAS Init Script to ensure persistence.

The version of Compose this uses is the latest, 1.29.2, so you can actually tell Compose to create the networks in addition to referencing external ones.

i.e.:
The below docker-compose.yml example will set up 2 networks when docker-compose up is run and removes them when Compose is stopped (downed).

Code:

version: "3"

networks:

  external_proxy:
    name: external_proxy
    driver: bridge
    ipam:
      config:
        - subnet: 192.168.90.0/24

  internal_proxy:
    name: internal_internal
    driver: bridge
    ipam:
      config:
        - subnet: 192.168.92.0/24

In the above case I'm creating two bridge networks, but you can use other types like Macvlan, etc. I'm also specifying ip ranges, but you can leave that as the default too. Basically, any Docker network you can create externally you can just define in the Compose file and they'll be created/destroyed as needed.

There's also an option in this app to define the Compose file you want it to start when the app starts. I don't see a straightforward way to have it launch multiple Compose files but that can be done from the CLI.

 

[Ixian]

Now that my containers are successfully running on TrueNAS SCALE, I'll be looking to migrate most of them to TrueNAS SCALE Applications. I could use some guidance on migrating existing, persistent app data from a dataset to TrueChart apps. Cheers!

The simplest way would be to use bind mounts on the host, which all the Truechart app configs I've seen support. If you are already using bind mounts (instead of docker managed volumes) with Compose you can just re-use those.

With bind mounts, of course, you are just leveraging your file system, meaning you have to create the directories for your persistent storage, change permissions appropriately, etc. but some folks prefer that level of control. There are also implications for clustering, etc. if you go this route but for many users that's probably not a big concern, particularly with media-server oriented apps.

Otherwise, an option with Truecharts apps is to use PVC (PersistantVolumeClaims) which, to oversimplify a bit, are managed persistent storage volumes that are created with the app but persist, hence the name.

If you go the PV route you can fire up the app (so the volume gets created and claimed) then from the cli run k3s kubectl describe pv and in the VolumeAttributes you should find the path to it on the filesystem (it'll be inside the pool location you specified for Apps when you set it up). You can copy existing data there - make sure you change any permissions on your data as needed (i.e. to the guid/puid the App runs at, etc. - it's an option in the app config, with Truecharts apps).

There's more to persistent volumes but hopefully that gets you started.

 

[bitbit]

The simplest way would be to use bind mounts on the host, which all the Truechart app configs I've seen support. If you are already using bind mounts (instead of docker managed volumes) with Compose you can just re-use those.

With bind mounts, of course, you are just leveraging your file system, meaning you have to create the directories for your persistent storage, change permissions appropriately, etc. but some folks prefer that level of control. There are also implications for clustering, etc. if you go this route but for many users that's probably not a big concern, particularly with media-server oriented apps.

Otherwise, an option with Truecharts apps is to use PVC (PersistantVolumeClaims) which, to oversimplify a bit, are managed persistent storage volumes that are created with the app but persist, hence the name.

If you go the PV route you can fire up the app (so the volume gets created and claimed) then from the cli run k3s kubectl describe pv and in the VolumeAttributes you should find the path to it on the filesystem (it'll be inside the pool location you specified for Apps when you set it up). You can copy existing data there - make sure you change any permissions on your data as needed (i.e. to the guid/puid the App runs at, etc. - it's an option in the app config, with Truecharts apps).

There's more to persistent volumes but hopefully that gets you started.

Great stuff here! Thank you for the guidance. I'll give the migration to PVC a go later today.

 

[Ixian]

Few other things I've noted:

By default /mnt, /root, and /cluster are exposed from the host to the Compose app. Those should cover most use cases however it can get a little tricky if you need to access other host resources.

For example, one of my test containers needs read-only access to /etc/timezone. This doesn't exist in the Compose app and /etc isn't passed through by default from the host so initially that will fail. You can bind /etc/timezone in read-only mode to the Compose app though, which in turn will make it available to bind to a container, so this shouldn't be an issue in general.

Where things might get tricky is with GPU passthrough. I don't have a GPU in my test setup so I am not sure if I can pass through /dev/dri or if there are other impediments to getting GPU passthrough to work with docker-compose containers run through this App, or if that will conflict with other Apps, etc.

I'm also still working through how containers stood up via the Compose App can interact with other Apps. Since it's using the K3S network stack you can ping apps via their internal kubernetes domain name i.e. ping sabnzbd.ix-sabnzbd.svc.cluster.local works from the Compose shell to reach an app named sabnzbd. I assume this means we can link Compose container apps with SCALE apps but haven't done much testing.

And I'm not sure how it is leveraging the host docker socket, host resource allocation, etc.

Still, exciting to see this and hopefully it turns out to be a good solution for running Compose.

 

[bitbit]

Also a great solution to run cloudflared as a reverse proxy. Below is an example docker-compose file and Cloudflared config.yaml. You'll also need your CLOUDFLARED_UUID.json and cert.pem files. Check out their documentation on how to set it up.
cloudflared.yml

Code:

version: '3'
services:
  cloudflared:
    image: cloudflare/cloudflared:2022.1.3-amd64
    container_name: cloudflared
    restart: always
    command: tunnel run ${CLOUDFLARED_UUID}
    networks:
      - proxy
    volumes:
      - ${CONFIG}/cloudflared:/home/nonroot/.cloudflared/
networks:
  proxy:
    external: false

config.yaml

Code:

tunnel: CLOUDFLARED_UUID
credentials-file: /home/nonroot/.cloudflared/CLOUDFLARED_UUID.json

ingress:
  - hostname: plex.example.com
    service: http://plex.ix-plex.svc.cluster.local:32400
  - hostname: overseerr.example.com
    service: http://overseerr.ix-overseerr.svc.cluster.local:5055
  - service: http_status:404

Ideally, we'd point Cloudflared to Traefik but my initial attempt was unsuccessful

Code:

tunnel: CLOUDFLARED_UUID
credentials-file: /home/nonroot/.cloudflared/CLOUDFLARED_UUID.json

# forward all traffic to Reverse Proxy w/ SSL and no TLS Verify
ingress:
  - service: https://traefik.ix-traefik.svc.cluster.local:443
    originRequest:
      noTLSVerify: true

 

[Ixian]

Also a great solution to run cloudflared as a reverse proxy. Below is an example docker-compose file and Cloudflared config.yaml. You'll also need your CLOUDFLARED_UUID.json and cert.pem files. Check out their documentation on how to set it up.
cloudflared.yml

Code:

version: '3'
services:
  cloudflared:
    image: cloudflare/cloudflared:2022.1.3-amd64
    container_name: cloudflared
    restart: always
    command: tunnel run ${CLOUDFLARED_UUID}
    networks:
      - proxy
    volumes:
      - ${CONFIG}/cloudflared:/home/nonroot/.cloudflared/
networks:
  proxy:
    external: false

config.yaml

Code:

tunnel: CLOUDFLARED_UUID
credentials-file: /home/nonroot/.cloudflared/CLOUDFLARED_UUID.json

ingress:
  - hostname: plex.example.com
    service: http://plex.ix-plex.svc.cluster.local:32400
  - hostname: overseerr.example.com
    service: http://overseerr.ix-overseerr.svc.cluster.local:5055
  - service: http_status:404

Ideally, we'd point Cloudflared to Traefik but my initial attempt was unsuccessful

Code:

tunnel: CLOUDFLARED_UUID
credentials-file: /home/nonroot/.cloudflared/CLOUDFLARED_UUID.json

# forward all traffic to Reverse Proxy w/ SSL and no TLS Verify
ingress:
  - service: https://traefik.ix-traefik.svc.cluster.local:443
    originRequest:
      noTLSVerify: true

Interesting. I haven't migrated to Cloudflared yet because I already use a reverse proxy (Traefik) and I filter out non-Cloudflare origins from my firewall/IDS (Opnsense) already but it's on the list. I'd want to do the same and have Traefik still be the ingress point vs. defining everything in Cloudflared because Traefik has more flexible filtering and is easy to integrate with Identity management (I use Authelia).

What you are doing should work - are you sure the Traefik app is listening on 443 internally, not just externally? Since you are using the internal name.