Apps SSL with Let's Encrypt and Caddy

alugowski

Dabbler
Joined
May 8, 2019
Messages
32
This post is about how I set up easy, fully-automatic SSL for my TrueNAS Apps and Docker containers. I'm assuming a home/small business deployment.

The official instructions and support are great, but only work with CloudFlare and Route53 domain providers. Everyone else is on their own. Some folks buy a certificate, some run CertBot on their own and upload the cert to TrueNAS, some do manual DNS challenges every few months.

An easier way is to just use a reverse proxy that acquires and manages certificates for you, using ACME. The two easiest are Caddy and Traefik. TrueCharts is well integrated with their Traefik chart, but that chart has no way to enable Traefik's ACME. So let's go with Caddy.

All we need is to run Caddy via Launch Docker image and :
  • use official 'caddy' image
  • mount a /config volume (Caddy stores its own config here that persists between docker image restarts and upgrades)
  • mount a /data volume (Caddy stores the certificates here)
  • mount a directory with your Caddyfile to /etc/caddy
  • Forward container ports 80 and 443 to the node. I used 9080 and 9443. Have your router forward ports 80 and 443 to your TrueNAS box's IP and ports 9080 and 9443.
  • Configure your Caddyfile to reverse proxy the containers you want to expose to the outside.
For the volumes, I have a 'caddy' directory with config, data, etccaddy subdirs and just mount those. This is what it looks like in the UI:

Screen Shot 2022-08-30 at 4.57.52 PM.png

Screen Shot 2022-08-30 at 4.58.48 PM.png

Screen Shot 2022-08-30 at 4.58.36 PM.png


I also use the "kill" scaling policy to avoid race conditions on the writable volumes.
Screen Shot 2022-08-30 at 4.59.04 PM.png

Everything else is the default.

In my `etccaddy` directory I have the Caddyfile that configures Caddy. There are many tutorials out there on how to write this, but here's what mine looks like:

Code:
# TrueNAS Reverse Proxy setup. The TrueNAS host has IP 192.168.1.10
# Services have port forwarding configured for any service that needs to be accessible from outside
# The Caddy container handles ports 80 and 443 as forwarded by the router and again by TrueNAS.

# Configure ACME
{
    # Test configuration using Let's Encrypt staging environment.
    # Comment out when happy with your config to get a real certificate:
    acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
    email your@email.com
}

truenas.example.com {
    # Simple proxy example:
    # Static web server hosted by static-file-server Docker container
    handle_path /static* {
        reverse_proxy 192.168.1.10:11080
    }

    # Syncthing (official docker image)
    # Based on https://docs.syncthing.net/users/reverseproxy.html
    handle_path /syncthing/* {
        # The syncthing docs include this line, but it does what handle_path already does
        #uri strip_prefix /syncthing
        reverse_proxy http://192.168.1.10:9384 {
            header_up Host {upstream_hostport}
            # Caddy warning message says this is unnecessary:
            #header_up X-Forwarded-Host {host}
        }
    }
    redir /syncthing /syncthing/
}

# PhotoPrism served as a subdomain. Uses a CNAME record that points 'pp' to truenas.example.com to reuse the DDNS entry.
pp.example.com {
    reverse_proxy 192.168.1.10:12342
}


This uses the Let's Encrypt STAGING environment for testing. When you're done testing comment out the `acme_ca` line to get real certs.

This example shows how to put an app at a path (truenas.example.com/static, and truenas.example.com/syncthing/) as well as at a subdomain (pp.example.com). The ports you see here are the ones that each app has been configured to export to the node. Since TrueNAS won't let you use port numbers below 9000, you usually have to use a port number other than the app default.
 

alugowski

Dabbler
Joined
May 8, 2019
Messages
32
NextCloud

I chose to run the official NextCloud docker image instead of the bundled app, mostly due to being burned by the old official FreeNAS plugin. The two options are to run the regular image which bundles its own Apache server, and the fpm image that depends on supplying your own HTTP server.

The regular image (tag: latest) or the TrueNAS NextCloud plugin probably just requires a single `reverse_proxy` directive, like the PhotoPrism in my example.

The upside of the FPM image (tag: fpm) is that we already have an HTTP server so let's use it. NextCloud needs the HTTP server to both call its FastCGI and to serve static content. For the latter we need to mount the same volume for NextCloud's /var/www/html to Caddy so it can serve static content. This seems to work great:

Code:
# Nextcloud
# Using the FPM version of NextCloud.
# An alternative is to use the default version that includes its own pre-configured Apache.
# In that case only a simple reverse_proxy is enough, but it means an extra hop for every request.
# Based on https://caddy.community/t/example-docker-nextcloud-fpm-caddy-v2-webserver/9407
nc.example.com {
    redir /.well-known/carddav /remote.php/dav 301
    redir /.well-known/caldav /remote.php/dav 301

    header {
        # enable HSTS
        Strict-Transport-Security max-age=15552000;
    }

    # .htaccess / data / config / ... shouldn't be accessible from outside
    @forbidden {
        path /.htaccess
        path /data/*
        path /config/*
        path /db_structure
        path /.xml
        path /README
        path /3rdparty/*
        path /lib/*
        path /templates/*
        path /occ
        path /console.php
    }
    respond @forbidden 404

    # Proxy FastCGI. The port is chosen randomly. Not using the NextCloud default of 9000.
    php_fastcgi 192.168.1.10:9903

    # Serve static content
    # Requires mounting the same NextCloud 'html' volume both here and in the NextCloud container.
    # It's mounted read-only here.
    root * /var/www/html
    file_server
}
 
Last edited:

alugowski

Dabbler
Joined
May 8, 2019
Messages
32
Caddy reload without restarting

Caddy can be told to reload the Caddyfile without restarting. This is handy when experimenting with your services to avoid having to stop/start the container after a change.

I wrote this script to do so (requires sudo):

Code:
#!/usr/bin/env bash
# Run caddy reload in the caddy container to avoid having to stop/start it

# view container namespaces:
# k3s kubectl get namespaces
# Caddy appears as ix-caddy
NAMESPACE="ix-caddy"

# view pods in namespace:
# k3s kubectl get -n <NAMESPACE> pods
# returns a header line then the caddy pod is in the first column
POD="$(k3s kubectl get -n $NAMESPACE pods | tail -n 1 | awk '{print $1}')"

# Run a shell in the container:
# k3s kubectl exec -n <NAMESPACE> --stdin --tty <POD> -- /bin/bash

k3s kubectl exec -n $NAMESPACE --stdin --tty $POD -- caddy reload --config /etc/caddy/Caddyfile --adapter caddyfile
 

truecharts

Guru
Joined
Aug 19, 2021
Messages
788
An easier way is to just use a reverse proxy that acquires and manages certificates for you, using ACME. The two easiest are Caddy and Traefik. TrueCharts is well integrated with their Traefik chart, but that chart has no way to enable Traefik's ACME. So let's go with Caddy.

To be clear, and like explained before, normally on kubernetes people also don't use Traefik ACME but use the industry-standard "CertManager" system. So even if we are going to support non-SCALE certificates (which is not out of the realm of possibilities either), it would be using kubernetes secrets for use with things like certmanager, as that's the standard.

Traefik ACME has basically been gimped on purpose to upsell their enterprise offering.
So the TLDR is more like: 'Traefik makes it unfeasable for us to enable Traefik-ACME'

We'll make a note to support key certificates end of this year though. That should allow users to use the normal kubernetes tools for this as they please :)
 

alugowski

Dabbler
Joined
May 8, 2019
Messages
32
To be clear, and like explained before, normally on kubernetes people also don't use Traefik ACME but use the industry-standard "CertManager" system. So even if we are going to support non-SCALE certificates (which is not out of the realm of possibilities either), it would be using kubernetes secrets for use with things like certmanager, as that's the standard.

Traefik ACME has basically been gimped on purpose to upsell their enterprise offering.
So the TLDR is more like: 'Traefik makes it unfeasable for us to enable Traefik-ACME'

We'll make a note to support key certificates end of this year though. That should allow users to use the normal kubernetes tools for this as they please :)

First google result for "TrueCharts CertManager" is a pull request for how TrueCharts is getting rid of CertManager. https://github.com/truecharts/charts/issues/189

If you have a writeup somewhere feel free to share. I honestly couldn't find one. I doubt I'm the only one.
 

truecharts

Guru
Joined
Aug 19, 2021
Messages
788
First google result for "TrueCharts CertManager" is a pull request for how TrueCharts is getting rid of CertManager. https://github.com/truecharts/charts/issues/189

If you have a writeup somewhere feel free to share. I honestly couldn't find one. I doubt I'm the only one.

What we where saying is that we're open to allow self-defined secrets for certificates in the SCALE GUI again.
(which includes cert-manager)

Still: CertManager is the kubernetes industry standard, which we already support on normal Helm installs for a while by now.
 

ukos

Dabbler
Joined
Jul 22, 2018
Messages
12
Thank you for the introduction to caddy on TrueNas Scale it helped my to get my caddy set up.
I was previously running a debian VM with nginx as the ingres on my old TrueNas Core. Like you, I also had a lot of trouble with the native nextcloud jail on core and I am now pleased to finally switch back from bsd to linux.

When searching for a cloud-native solution, I came across your post on Traefik/letsencrypt. After taking a look at @alugowski github repo with the charts beeing >300 commits behind upstream, I realized that such a (hacky) solution will soon get out-of date.

Coming from "pure" nginx, I do get that caddy can simplify the process of certbot and server but we will see if it can also handle the ssh connection forwarding needed for my gitea server. Nevertheless, good tutorial!
 
Last edited:
Top