SSL certificate problem: certificate has expired -- the OpenSSL 1.0.2 vs LetsEncrypt issue

[Daisuke]

Because we're discussing FreeNAS

My bad, I thought we are talking the latest TrueNAS release.

Starting in FreeBSD 12.2, a new certificate manager has been added to FreeBSD

For us who use the latest version, I presume we do not have to do anything, right? What jail package we need to make sure is installed, which has the latest LetsEncrypt root certificate? Thank you.

Edit: I believe this will be all fixed into ca_root_nss-3.72. To import the latest LetsEncrypt root certificate into Mono certificate store, use @jgreco's fix and run these commands inside your jail:

Code:

$ iocage console yourjailname
# pkg provides /usr/local/share/certs/ca-root-nss.crt
Name    : ca_root_nss-3.71
Desc    : Root certificate bundle from the Mozilla Project
Repo    : FreeBSD
Filename: usr/local/share/certs/ca-root-nss.crt

# fetch --no-verify-peer https://extranet.www.sol.net/files/misc/ca-root-nss.crt.src
# mv -f ca-root-nss.crt.src /usr/local/share/certs/ca-root-nss.crt
# cert-sync /usr/local/share/certs/ca-root-nss.crt

cert-sync is a binary part of mono6.8-6.8.0.123, note there was an importing bug present into a previous version.

You can always restore the original /usr/local/share/certs/ca-root-nss.crt with:

Code:

# pkg install -f ca_root_nss

Restart the jail, the error will still show when you access the System Status inside Radarr UI. You need to test and save each indexer configuration in Settings > Indexers, then the error will be gone:

You do not need to disable the Certificate Validation in Settings > General:

 

[Trianine]

My bad, I thought we are talking the latest TrueNAS release.

That's on me. I am discussing TrueNAS, but jgreco was clearly mentioning FreeNAS. That said, the problem is clearly affecting us too. I tried looking for a 12.2 release, and only found an update relating to the jails. It didn't help me, however. I guess we just have to wait. :)

In the CLI you can enter "iocage fetch" and select the 12.2 update.

 

[Daisuke]

In the CLI you can enter "iocage fetch" and select the 12.2 update.

I was running 12.2-RELEASE-p10 with indexer errors on Sonarr and Radarr, because the missing LetsEncrypt root certificate. I did an upgrade to p11 and it did not fixed the issue, even if it performs the certificates cleanup during upgrade. Obviously the existing jail certificates are not cleaned.

 

[Trianine]

# pkg provides /usr/local/share/certs/ca-root-nss.crt

This gives an "unknown command" error, any ideas why that would be? I trust it works on your end.

 

[Daisuke]

You need to enable the provides plugin inside your jail. And you don't really need that command, but I use it for reference when I modify a file, to see what package belongs to. In this way, I can check when a new package version is released if the file is updated also.

Code:

# pkg install pkg-provides

Uncomment/update the following lines inside /usr/local/etc/pkg.conf file, the lines below are commented and PLUGINS has weird brackets with no semi-column at end, this is the correct format:

Code:

PKG_PLUGINS_DIR = "/usr/local/lib/pkg/";
PKG_ENABLE_PLUGINS = true;
PLUGINS [ provides ];

Update the database and search for a file:

Code:

# pkg provides -u
# pkg provides /usr/local/share/certs/ca-root-nss.crt

 

[jgreco]

My bad, I thought we are talking the latest TrueNAS release.

For us who use the latest version, I presume we do not have to do anything, right? What jail package we need to make sure is installed, which has the latest LetsEncrypt root certificate? Thank you.

You can presume whatever the heck you'd like, but I'm not dumb enough to make any promises, because there are ALWAYS counterexamples.

SSL is a sorry-ass system that was poorly architected and implemented basically from Day 1, because it gave in to the worst impulses of crypto geeks, developer divas, and corporate interests who had competing visions of what they were trying to create. The certificate authorities, owned by corporations, saw it as a way to print money. Web browsers, such as Mozilla and the ca-root-nss we are discussing, saw it primarily as a way to secure end-to-end communications, but didn't place a particularly high bar to entry on being listed as a root CA. End users, who didn't know better and who probably don't really give a crap anyways, unless they're compromised, blithely ignored certificate validation warnings and added new CA's to their trust stores; the technology failed them in numerous ways. Crypto wonks created increasingly complicated and difficult to implement strategies for certificate issuance, validation, and were generally completely fine with creating a system that had no particular standards for classes of security, or creating practical guidelines for deployment. Operating systems like FreeBSD generally looked at SSL with some puzzlement because there was just a lot of wild west for trust stores, and most of the trust store maintenance was being done by individual web browser developers, who had a vested interest in broadly being able to place a lock symbol next to a website in the URL bar, so they tended to add new CA's that we might otherwise consider dodgy. The whole ball of wax is a mess from numerous angles.

Or, simply: Why are you adding a Mozilla web browser trust store to your FreeBSD OS as its default trust store?

It's a pragmatic but crappy choice. Also, it isn't universally used as a trust store, so any answer I give you is going to be wrong in some cases.

 

[Daisuke]

It's a pragmatic but crappy choice.

I hear you. You said it very well:

End users, who didn't know better and who probably don't really give a crap anyways, unless they're compromised

I'm part of that category. My goal was to fix the issue where the new LetsEncrypt root certificate not being updated, creating issues with Radarr and Sonarr plugins. Once the .NET changes are implemented into new packages, all these issues will be gone.

 

[Trianine]

I'm running into a lot of other roadblocks, but this worked. I'll have to continue my attempts tomorrow. Thank you.

You need to enable the provides plugin inside your jail. And you don't really need that command, but I use it for reference when I modify a file, to see what package belongs to. In this way, I can check when a new package version is released if the file is updated also.

Code:

# pkg install pkg-provides

Uncomment/update the following lines inside /usr/local/etc/pkg.conf file, the lines below are commented and PLUGINS has weird brackets with no semi-column at end, this is the correct format:

Code:

PKG_PLUGINS_DIR = "/usr/local/lib/pkg/";
PKG_ENABLE_PLUGINS = true;
PLUGINS [ provides ];

Update the database and search for a file:

Code:

# pkg provides -u
# pkg provides /usr/local/share/certs/ca-root-nss.crt

 

[Daisuke]

I'm running into a lot of other roadblocks

I updated the previous post with better instructions. If your issues are not related to ca_root_nss, you should not highjack this thread and start a new one.

 

[XamarinMaster]

I am on FreeNAS 11.3U5 and am trying to solve the problem of creating a jail. I have no jails at all and need one for SyncThing. When I try to create a new jail, I get the certificate error. I have followed the initial instructions of this post to delete the expired cert from /usr/local/share/certs/ca-root-nss.crt. The instructions didn't say to restart the FreeNAS but I did anyway. I see no change. I cannot create a jail.

 

[XamarinMaster]

I upgraded to TrueNAS Core and the problem is fixed.

 

[AlphaWolf]

cert-sync /usr/local/share/certs/ca-root-nss.crt fails for me saying it can't import, read-only filesystem.

It's strange that the even log is a cert error, but from the cli of the jail, I can curl the URL in the error log that says it's a cert issue and it returns successfully without an issue.

 

[Trianine]

Alphawolf, welcome to the club! I attempted to make a separate post about the read-only filesystem, but nobody came to the rescue.

 

[raidflex]

Alphawolf, welcome to the club! I attempted to make a separate post about the read-only filesystem, but nobody came to the rescue.

And crickets.... I would imagine others are experiencing the same error, I find it hard to believe that the command magically works for some and others like myself have a read only file system.

Code:

Importing into legacy system store:
I already trust 130, your new list has 129
1 previously trusted certificates were removed.
Error: System.IO.IOException: Read-only file system

 

[Trianine]

Over on the Emby forums, Frogmaster posted their solution, which solved my Cert-sync issue! I appear to have a working system now.

This is explicitly for Emby, and it's already out of date, so I can't tell you if the dependencies are the same if you are using a different jail, and I can tell you to point the release to 4.7.0.18, for now.

Emby unable to get pictures from FanArt.tv on Ubuntu

i have just noticed I'm no longer getting images from fanart.tv too . I'm running on freeNAS/trueNAS Core emby server plugin 4.6.4.0. I'm not sure when this started tho

emby.media

https://github.com/MediaBrowser/Emby.Releases/releases/download/4.7.0.18/emby-server-freebsd12_4.7.0.18_amd64.pkg

 

[raidflex]

Over on the Emby forums, Frogmaster posted their solution, which solved my Cert-sync issue! I appear to have a working system now.

This is explicitly for Emby, and it's already out of date, so I can't tell you if the dependencies are the same if you are using a different jail, and I can tell you to point the release to 4.7.0.18, for now.

Emby unable to get pictures from FanArt.tv on Ubuntu

i have just noticed I'm no longer getting images from fanart.tv too . I'm running on freeNAS/trueNAS Core emby server plugin 4.6.4.0. I'm not sure when this started tho

emby.media

https://github.com/MediaBrowser/Emby.Releases/releases/download/4.7.0.18/emby-server-freebsd12_4.7.0.18_amd64.pkg

Yeah I noticed the issue with Emby, currently it seems to be working with 4.6.7, so at least you don't need to run a beta. There is still something going on here because I have Radar, Sonarr and Lidarr in a jail and although Sonarr/Radarr seem to be working, Lidarr still has indexer errors. At this point most of the apps seem to be working for me but that read only error I would imagine is preventing the command from completing properly. I do not have enough experience with SSL certs and Freebsd so I was hoping someone with more knowledge could chime in here.

 

[mortis2600]

I am so frustrated. I had some hardware failures that took down my server. I fixed those issues and took the opportunity to update my system and do some fresh installs of my plugins and reorganize things a little. Installed TrueNAS 12 U6 and am getting the error above and cannot add any indexers to my clients.

I've deleted the offending cert as described, but I'm still unable to add any indexers to Sonarr or Radarr. I'm not sure what to do next? Help please?

 

[raidflex]

I am so frustrated. I had some hardware failures that took down my server. I fixed those issues and took the opportunity to update my system and do some fresh installs of my plugins and reorganize things a little. Installed TrueNAS 12 U6 and am getting the error above and cannot add any indexers to my clients.

I've deleted the offending cert as described, but I'm still unable to add any indexers to Sonarr or Radarr. I'm not sure what to do next? Help please?

It is currently working for me with Radarr, Sonarr, Lidarr and Emby after importing the latest Cert with mono 6.8. I have all my apps running in custom jails, I do not use the plugins. I would try setting up a jail and manually installing an app you are having issues with and then updating the cert. I believe this issue should be fixed in 12.0U7.1.

I would also update the pkg repository to the latest to receive the latest app versions. You just need to change "quarterly" to "latest" under the /etc/pkg/FDreeBSD.conf config file in your jail.

Code:

url: "pkg+http://pkg.FreeBSD.org/${ABI}/quarterly"

 

[marcevan]

I'm running a few plugins, and ONLY sonarr is failing the basic sniff test of "does it work?". It cannot even add a new series or one on disk. All indexers are toast.

I've read it's a mono issue, but on the Sonarr forums they arrogantly proclaim it's not a Sonarr issue at all.

So....what is equiv of Sonarr that actually works?

 

[hmvh1]

I'm running a few plugins, and ONLY sonarr is failing the basic sniff test of "does it work?". It cannot even add a new series or one on disk. All indexers are toast.

I've read it's a mono issue, but on the Sonarr forums they arrogantly proclaim it's not a Sonarr issue at all.

So....what is equiv of Sonarr that actually works?

Try to set the Certificate Validation to disable under general settings. Then test each indexer to see if it connects.

If that works, I posted a workaround to fix the certificate validation in this thread on Oct 25, 2021