Certbot does not run after Scale update (bad interpreter: /dpkg-src/v/bin/python: no such file or directory)

[Roaders]

Hi All
I have a cronjob that updates my certbot certificates every night and it was working fine. Now however my certbot install seems to be broken:

root@ring[/bin]# certbot
zsh: /bin/certbot: bad interpreter: /dpkg-src/v/bin/python: no such file or directory
root@ring[/bin]# where certbot
/bin/certbot
/usr/bin/certbot

I am pretty inexperienced with linux but it looks like dpkg is a package manager but it doesn't seem to be installed on scale. Also no apt-get which is what I am used to from raspbian OS.
I looked at the install steps for certbot and it talks about installing snap. I tried this and it didn't work either.

As I said it definitely used to work. I suspect that when I updated my scale installation it broke (but this is just a guess).

I am currently on:

TrueNAS-SCALE-23.10.0.1

(I think that I was on 22.something before I updated)

 

[bcat]

TrueNAS SCALE is an appliance OS, so expecting system binaries (like a Python interpreter) to be in any particular place is a bit of a crapshoot. It might work in one particular release, but not in another. And, generally, you should assume any modifications you make to the core install that aren't officially supported by SCALE will be lost on upgrade. Basically, the very concept of "installing snap" on TrueNAS is fundamentally unsupported. It uses Debian and has apt under the hood, but that's an implementation detail rather than a feature. :)

If you're just trying to provision a cert for TrueNAS itself, there is built-in ACME support, including support for DNS challenges that seems to work fine (though it only supports a tiny number of DNS providers compared to alternative systems).

If you need something fancier than that, TrueCharts offers Cert-Manager integration, though I've not used this myself and have no idea how to set it up. (Maybe other folks can point at more detailed docs on it.)

Otherwise, if you really want to run certbot "on the NAS" (so it can write the certs directly into a ZFS dataset, for instance), consider the Jailmaker script. It's community supported and still might break on upgrades, but the dev has taken steps to make it minimally invasive to the host system. It uses systemd-nspawn to spin up lightweight Debian containers that can run whatever you want, and that can optionally access the host network (if you choose not to use bridge/macvlan) and filesystem (via bind mounts). I haven't used it for certbot, but I'm trying it out now for Tailscale and Docker (in separate containers), and it seems to work really well.

 

[danb35]

if you really want to run certbot "on the NAS"

The thing is, certbot itself is already on the NAS, and apparently has been for a while. But Cobia broke something that results in being unable to use it now.

 

[bcat]

The thing is, certbot itself is already on the NAS, and apparently has been for a while. But Cobia broke something that results in being unable to use it now.

Oh, that does change things. :) I didn't know certbot was preinstalled, sorry. Then, yes, I'd generally expect it to work....

 

[Roaders]

The thing is, certbot itself is already on the NAS, and apparently has been for a while. But Cobia broke something that results in being unable to use it now.

Thanks for this reply. It seems that this is a known issue but I've not been able to find any discussion about it. Do you have any links? It's the issue likely to be resolved?

 

[Roaders]

TrueNAS SCALE is an appliance OS, so expecting system binaries (like a Python interpreter) to be in any particular place is a bit of a crapshoot. It might work in one particular release, but not in another. And, generally, you should assume any modifications you make to the core install that aren't officially supported by SCALE will be lost on upgrade. Basically, the very concept of "installing snap" on TrueNAS is fundamentally unsupported. It uses Debian and has apt under the hood, but that's an implementation detail rather than a feature. :)

If you're just trying to provision a cert for TrueNAS itself, there is built-in ACME support, including support for DNS challenges that seems to work fine (though it only supports a tiny number of DNS providers compared to alternative systems).

If you need something fancier than that, TrueCharts offers Cert-Manager integration, though I've not used this myself and have no idea how to set it up. (Maybe other folks can point at more detailed docs on it.)

Otherwise, if you really want to run certbot "on the NAS" (so it can write the certs directly into a ZFS dataset, for instance), consider the Jailmaker script. It's community supported and still might break on upgrades, but the dev has taken steps to make it minimally invasive to the host system. It uses systemd-nspawn to spin up lightweight Debian containers that can run whatever you want, and that can optionally access the host network (if you choose not to use bridge/macvlan) and filesystem (via bind mounts). I haven't used it for certbot, but I'm trying it out now for Tailscale and Docker (in separate containers), and it seems to work really well.

Thanks for this detailed reply. I did spend some time trying to setup certbot docker but couldn't get it work. Even if I had got it to work I am not sure how I would get it to restart docker containers when it refreshed a certificate for example. I was pleased to find certbot available and able to run via a simple cron job. Not any more though

 

[Roaders]

Is anyone able to help mw with this please? My certificate has now expired....

 

[danb35]

You've already been given a few suggestions up-thread. To those I'd add using acme.sh and deploying the cert using the TrueNAS API, either using my script (it's in the Resources section) or the script that comes with acme.sh.

Somewhat surprisingly, it doesn't look like anyone's reported a bug on this. Unlikely the devs will do anything to fix it without one.

 

[Roaders]

sorry to ask (I am sure it's obvious to everyone else) but where do I report a bug? I tried to find a form but it looked to me like bug reports were only available to commerical customers.

I do appreciate the replies and the other suggestions (which I might have to resort to) but it took me some time to get this approach working and as it seems like something that should be working isn't I would like to get this working again if I can. One other approach that some people have suggested was to run certbot in a docker instance to update the certificate. I can't see how I would restart my app if the certificate was renewed in that case...

If possible I would like to avoid another learning curve and having to invest more time into looking into how to get these other suggestions to work in my case.

But as I said. Thanks very much for the time you took to reply.

I will look into acme.sh

 

[Roaders]

You've already been given a few suggestions up-thread. To those I'd add using acme.sh and deploying the cert using the TrueNAS API, either using my script (it's in the Resources section) or the script that comes with acme.sh.

Somewhat surprisingly, it doesn't look like anyone's reported a bug on this. Unlikely the devs will do anything to fix it without one.

your post on acme does look very helpful. Thank you for that. When I have time I will try and follow the steps to get that working.

 

[danb35]

where do I report a bug?

The link to "Report a Bug" is at the top of every page in this forum.

 

[Roaders]

Sorry, I had been looking on the IX systems web pages rather than this community.

 

[Roaders]

Coming back to this and trying to implement the script that you mention in your post @danb35 it seems that script is for updating the certificate for truenas.... That's not what I want to do. I want to update my certificate, copy it to a folder and then restart one of my docker containers that uses that certificate...
Looks like I'll have to learn how to write my own acme script to achieve this.
I did raise a bug for certbot and the response was "Yes, we removed certbot from the system in Cobia." I asked if there were any release notes and did not get a response. This doesn't seem entirely accurate though as certbot is obviously still on the system, it just doesn't work... Not blown away by the response to my bug report TBH.