Dynamic DNS + Let's Encrypt on SCALE apps

[sbor23]

Question:
Is it possible to use dynamic DNS (eg. duckdns.org) and Let's Encrypt for truecharts apps?
If yes, what is preferred way?
I don't necessarily want to expose the TrueNAS UI, so just the apps for now.

Background:
For some years I've been running OMV, running some apps like Jellyfin and Nextcloud using docker-compose behind a caddy reverse-proxy for generating Let's encrypt certificates using this: https://github.com/lucaslorentz/caddy-docker-proxy . Caddy uses ACME HTTP challenge in this case.
This takes place in my home network behind NAT and I don't have a static public IP from my ISP. Hence I use duckdns.org which is running on my router. Ports 80 and 443 are forwarded to the box.

So far so standard, one would think.

Now I'm currently evaluating TrueNAS SCALE as a replacement for the OMV box, having all the goodies like ZFS and the very compelling kubernetes with the truecharts catalog. I've followed the truecharts guide to the point where we need to register a ACME DNS-Authenticator with a public domain from Cloudflare or route53.
This is where I'm stuck, because I don't see official support for dynamic DNS users.

Edit: I first didn't realize that DDNS is possible with a custom domain in Cloudflare. I guess purchasing a domain and setup Cloudflare is a viable way.

Option 1: Use Traefik
Now I've seen similar posts, but this is about a situation where we have a static IP, so we can point a public domain to it: https://www.truenas.com/community/threads/use-traefik-to-generate-lets-encrypt-certificates.100881/
My intuition was also to just let Traefik handle the Let's encrypt part but apparently that's not easily possible as it's an Ingress controller etc.
I've found these instructions for Traefik + kubernetesCRD + TLS but it seems complicated and I have no idea if it would work with truecharts.

Traefik CRD TLS Documentation - Traefik

Learn how to use Traefik Proxy w/ an IngressRoute Custom Resource Definition (CRD) for Kubernetes, and TLS with Let's Encrypt. Read the technical documentation.

doc.traefik.io

Option 2: Wait for more ACME DNS-Authenticators to be enabled by TrueNAS SCALE.
There is a ticket open for this reason: https://jira.ixsystems.com/browse/NAS-115350
Lego would support DDNS providers such as DuckDNS: https://go-acme.github.io/lego/dns/duckdns/

Option 3: Workaround to run acme.sh locally and import the cert via truenas API
This blog describes how it would be possible to do the whole ACME challenge separately (in a shell script, could be docker too I guess), somehow push the cert to the TrueNAS using the API, automate the process via cron jobs etc:

Let’s encrypt Certificate with AMCE Shell Script (amce.sh) on TrueNAS SCALE – SYSADMIN102™

sysadmin102.com

However this seems like a maintenance hell to me, hence I'm reluctant to go this route. There must be a better way...

Unfortunately I'm stuck here and as of now TrueNAS is not viable for me as a NAS for this reason. Which is a shame because I like the concept and the package a lot.
And I have to say I'm a bit surprised to be stuck at this point, because I would consider my situation very common for home users.

 

[morganL]

Option 2: Wait for more ACME DNS-Authenticators to be enabled by TrueNAS SCALE.
There is a ticket open for this reason: https://jira.ixsystems.com/browse/NAS-115350
Lego would support DDNS providers such as DuckDNS: https://go-acme.github.io/lego/dns/duckdns/

While we encourage upvoting on this feature request, I don't think it makes sense to wait for it.
Option 1 or Option 3 are the current options.

As Yogi Berra said "when you come to a fork in the road, pick one".

 

[sbor23]

Thanks for the reply @morganL confirming my observations.
Seems that I didn't miss any obvious, alternative solutions.
Another workaround until Option 2 (hopefully) happens might be to use self-signed certs for my DDNS.

I leave it open for a bit in case some people have more suggestions.

 

[grayson]

Option 4: Run Letsencrypt + certbot in docker, so you can have dynamic IP with domain name.

 

[sbor23]

Option 4: Run Letsencrypt + certbot in docker, so you can have dynamic IP with domain name.

Would you care to elaborate?
How would this integrate with TrueNAS / apps so the applications can be exposed using k8s?

I imagine this works if you want to run a good old docker(-compose) setup, like I was doing with OMV. But I don't want to work around the system here, in fact I hope TrueNAS with k3s/Truecharts to be a bit more "fire and forget" than a manual docker-compose setup.

 

[Black_Duck]

A variation of Option 4 above that I use:
Certbot is already installed in the TrueNAS base, so no need for docker. Use the certbot parameter "--config-dir" to point to a permanent storage location.
Prerequisites:

• Your Domain or your DDNS is set up (Dynamic DNS can be set up in "System Settings > Services")
• Add storage location for certificates to your pool (in my example filesystem is "pool/applications/certbot"
• Change TrueNAS HTTP port to 81 (or whatever) in "System > General > GUI" - hopefully you're using https anyway.

Setup:

1. Create certificate. This is a once off process
   1. Login into TrueNAS shell:
   2. certbot certonly --dry-run --standalone --preferred-challenges http-01 --email youremail@domain.com -d your.domain.name --config-dir "/mnt/pool/applications/certbot/letsencrypt"
   3. If all went well, then generate the certificate by removing the "--dry-run" parameter.
   4. Certificate now located at /mnt/pool/applications/certbot/letsencrypt/live
2. Set up regular renewal of certificate
   1. Go to TrueNAS Cronjob setup : "System > Advanced" and "Add" to Cron Jobs
      • Description: Certificate Renewal
      • Command: certbot renew --quiet --no-self-upgrade --standalone --preferred-challenges http-01 --config-dir "/mnt/pool/applications/certbot/letsencrypt"
      • Run As User: root
      • Schedule: Weekly
   2. SAVE
3. For any app requiring access to certificates: configure a Host Path: "/mnt/pool/applications/certbot/letsencrypt" to mount point "/etc/letsencrypt". Certificates are at '/etc/letsencrypt/live"

 

[CPT82]

A variation of Option 4 above that I use:
Certbot is already installed in the TrueNAS base, so no need for docker. Use the certbot parameter "--config-dir" to point to a permanent storage location.
Prerequisites:

• Your Domain or your DDNS is set up (Dynamic DNS can be set up in "System Settings > Services")
• Add storage location for certificates to your pool (in my example filesystem is "pool/applications/certbot"
• Change TrueNAS HTTP port to 81 (or whatever) in "System > General > GUI" - hopefully you're using https anyway.

Setup:

1. Create certificate. This is a once off process
   1. Login into TrueNAS shell:
   2. certbot certonly --dry-run --standalone --preferred-challenges http-01 --email youremail@domain.com -d your.domain.name --config-dir "/mnt/pool/applications/certbot/letsencrypt"
   3. If all went well, then generate the certificate by removing the "--dry-run" parameter.
   4. Certificate now located at /mnt/pool/applications/certbot/letsencrypt/live
2. Set up regular renewal of certificate
   1. Go to TrueNAS Cronjob setup : "System > Advanced" and "Add" to Cron Jobs
      • Description: Certificate Renewal
      • Command: certbot renew --quiet --no-self-upgrade --standalone --preferred-challenges http-01 --config-dir "/mnt/pool/applications/certbot/letsencrypt"
      • Run As User: root
      • Schedule: Weekly
   2. SAVE
3. For any app requiring access to certificates: configure a Host Path: "/mnt/pool/applications/certbot/letsencrypt" to mount point "/etc/letsencrypt". Certificates are at '/etc/letsencrypt/live"

I can agree on this!!!! Since I migrated from Core to Scale I had to go through this and it worked as intended.
I missed the 3rd bullet to "Change TrueNAS HTTP port to 81" and I got the message "Could not bind to port 80: IPv4 & IPv6"....
You have to do that since LetsEncrypt is going to listen on port 80 for the "Dry run" and final command.

thanks @Black_Duck

 

[mrbrdo]

The instructions from @Black_Duck got me in the right direction. In my case I am using Nextcloud from the official charts and you need to select a certificate from a dropdown in the app settings, it's not possible to use a file path directly. The same is true if you want to use this certificate for the web UI.

First of all I recommend a bit updated certbot command:

certbot certonly --standalone --non-interactive --keep-until-expiring --preferred-challenges http-01 --http-01-port=10080 --email my@email -d my.domain --config-dir "/mnt/MY_POOL/certbot/config" --logs-dir "/mnt/MY_POOL/certbot/log" --work-dir "/mnt/MY_POOL/certbot/workdir"

Changes:

• added "--non-interactive --keep-until-expiring" so it will not do any prompts (because we run it as a cronjob) and it will keep a certificate until it is close to expiring, instead of generating a new one every time
• added "--http-01-port=10080", this will make certbot listen on port 10080 instead of 80, so you don't need to change the port of the Web UI. However, you need to expose this port as port 80 on your external IP (e.g. on your router, you need to forward port 10080 from truenas to port 80 external). Because of proving server ownership, ACME will always only check on port 80 and ignore your custom port. In case you want to access Web UI from outside network, you should use SSL anyway and don't need port 80.
• added "--logs-dir" and "--work-dir" to avoid any problems with file permissions

So I also found this script to import the certificate from the filesystem into the TrueNAS Scale Certificates (Credentials - Certificates):

GitHub - danb35/deploy-freenas: Python script to automate deploying TLS certificates to FreeNAS servers

Python script to automate deploying TLS certificates to FreeNAS servers - danb35/deploy-freenas

github.com

Don't mind the name, it works fine with TrueNAS Scale. It has a setting to automatically update the certificate for apps, since the Certificates cannot be modified and it has to create a new one on every import. But that setting only updates the Ingress certificate, and the Nextcloud app in my case just has a Certificate setting, so I updated the deploy-freenas script to also update that:

GitHub - mrbrdo/deploy-freenas: Python script to automate deploying TLS certificates to FreeNAS servers

Python script to automate deploying TLS certificates to FreeNAS servers - mrbrdo/deploy-freenas

github.com

After this I created a shell script to first get the new Letsencrypt certificate, and then import it and update the apps certificate (if enabled in deploy_freenas config).

Code:

#!/bin/bash
certbot certonly --standalone --non-interactive --keep-until-expiring --preferred-challenges http-01 --http-01-port=10080 --email my@email -d my.domain --config-dir "/mnt/MY_POOL/certbot/config" --logs-dir "/mnt/MY_POOL/certbot/log" --work-dir "/mnt/MY_POOL/certbot/workdir"

python /mnt/MY_POOL/certbot/deploy-freenas/deploy_freenas.py

I created a cronjob to call this script weekly:

/bin/bash /mnt/MY_POOL/certbot.sh

Seems to be working great so far, it automatically updates my Letsencrypt certificate, imports it into TrueNAS Certificates list, and then updates it for my apps (and web UI if desired).

 

[NugentS]

How is this different from: https://www.youtube.com/watch?v=TJ5fDiDRcbU? Or is the magic in the danb script?

 

[mrbrdo]

The difference is that in that link he uses a DNS authenticator, with a supported provider like Cloudflare. Which means your DNS must be accessible via API.
The way I did it I use http verification of ownership (Letsencrypt checks for a file on port 80 on the server), so you don't need to have access/modify the DNS for verification.

 

[NugentS]

I rewrote the certbot command to work with cloudflare and an API call. It works - still not sure what the difference is once I have the cert

 

[mrbrdo]

NugentS like I said, I am using http validation and not DNS. There's a big difference, if you use DNS with Cloudflare then you can just use the built-in authenticator that TrueNAS provides and it will take care of everything. But TrueNAS doesn't support http validation out of the box, that's why I need a combination of certbot and importing the cert manually with the script. The cert needs to be updated every time it expires, that's why a cronjob is needed. If you use DNS authenticator then you don't need any of this, just follow the video then. Has nothing to do with this discussion.

 

[NugentS]

Thank you

 

[Black_Duck]

added "--http-01-port=10080", this will make certbot listen on port 10080 instead of 80, so you don't need to change the port of the Web UI. However, you need to expose this port as port 80 on your external IP (e.g. on your router, you need to forward port 10080 from truenas to port 80 external). Because of proving server ownership, ACME will always only check on port 80 and ignore your custom port. In case you want to access Web UI from outside network, you should use SSL anyway and don't need port 80.

Great idea. Works well.

So I also found this script to import the certificate from the filesystem into the TrueNAS Scale Certificates (Credentials - Certificates):

GitHub - danb35/deploy-freenas: Python script to automate deploying TLS certificates to FreeNAS servers

Python script to automate deploying TLS certificates to FreeNAS servers - danb35/deploy-freenas

github.com

Thanks @mrbrdo
Im using your script to now deploy the certificate to TrueNAS and all works well.
However, I’m now wondering if using acme.sh may be better (neater) than certbot, as acme.sh avoids port 80 authentication and can automatically propagate the certificate to TrueNAS without @danb35 script….
Need to think this one through as home-assistant also needs the certificate.
There’s also the new ”shell” DNS Authentication in Scale now, but not sure that helps with home-assistant.

 

[mrbrdo]

@Black_Duck well in case you are hosting your DNS on a supported provider (with API access), then DNS auth is definitely easier and cleaner. In my case I had to use the web authentication, and as far as I know, you _have to_ use port 80 (external), no matter what script you use (even acme.sh). Also, I think TrueNAS only supports the DNS mode of acme.sh at the moment, so in my case it doesn't help me, but maybe it could be possible to use acme.sh via a cronjob, and again manually import the cert.

 

[Black_Duck]

Looks like certbot is no longer in the Truenas base in Cobia....

 

[mrbrdo]

@Black_Duck you might be right, after upgrading I get this error when running certbot

zsh: /usr/bin/certbot: bad interpreter: /dpkg-src/v/bin/python: no such file or directory

Did you find a workaround?

 

[danb35]

Looks like certbot is no longer in the Truenas base in Cobia....

certbot itself is there (heaven only knows why), but it depends on a Python binary that's no longer present.

Did you find a workaround?

I'd say the workaround is to use an ACME client that isn't as dependency-laden as certbot--acme.sh is likely a good choice.

 

[Black_Duck]

Yes, I noticed it was still in /bin directory.
Wondering whether the missing Python library has been deliberately deprecated or if it’s a bug…. If it was deliberate, why leave certbot?
As @danb35 says, acme.sh is looking like the best option - also automates deployment of certificate to TrueNAS. I’ve been playing with it but it’s not a trivial task. You need to set it up to run from your pool; change it to use letsencrypt; copy (modify?) the cronjob over to TrueNAS (so it survives over upgrades); sort out the deployment script to Truenas; and change the host path in your apps.
I guess another possible option may be to run certbot as an app - I do see there is an offical certbot docker container.… Have not looked at this option.

 

[danb35]

You need to set it up to run from your pool;

Automatic if you're using a non-root admin user. Otherwise, acme.sh --install --home /mnt/pool/path --nocron

change it to use letsencrypt

You don't really need to (certs from ZeroSSL are just as valid as those from Let's Encrypt), but /path/to/acme.sh --set-default-ca --server letsencrypt

copy (modify?) the cronjob over to TrueNAS

Set up a cronjob through the UI to run /path/to/acme.sh --cron --home /path/to/.acme.sh" daily, ideally at some time other than midnight.

sort out the deployment script to Truenas

It now has its own (based on my Python script), or I think my script has more flexibility. But to use the built-in one:

• Generate an API key in the TrueNAS web UI
• Obtain the cert--/path/to/acme.sh --issue -d nas.yourdomain.com # add whatever other options, domains, etc. are relevant
• export DEPLOY_TRUENAS_APIKEY="<API_KEY_GENERATED_IN_THE_WEB_UI>"
• /path/to/acme.sh --deploy -d nas.yourdomain.com --deploy-hook truenas

change the host path in your apps.

None of my apps terminate TLS; they all go through a reverse proxy.