Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.

SSL certificate problem: certificate has expired -- the OpenSSL 1.0.2 vs LetsEncrypt issue

How helpful was this to resolving the certificate is expired issue?

  • Not technical enough

    Votes: 0 0.0%
  • You suck, Grinch!

    Votes: 0 0.0%

  • Total voters
    7
  • This poll will close: .

jgreco

Resident Grinch
Moderator
Joined
May 29, 2011
Messages
14,179
Posted in response to a staff request, this is intended to help answer the "certificate is expired" issues.

SSL underpins most network session security on the Internet. Historically, this was implemented with self-signed certificates (a bad idea), or through commercial certificate authorities (a pricey proposition). In a bid to see the Internet default to securing everything (which is a bad idea of a different sort), several industry players cobbled together a free, automatic certificate authority called LetsEncrypt, and released software to make it easy to get valid SSL certificates for your website (generally a good idea).

To get this bootstrapped, they initially signed this with the IdenTrust DST Root X3 certificate, which expired at the end of September 2021. LetsEncrypt has experienced wild success, and is now signing with their own ISRG Root X1 certificate, now trusted in most browsers and SSL trust stores. I'm not making a full accounting or description of the bootstrap and cross-signing processes used to bring this online. You can Google all that.

However, OpenSSL 1.0.2 has a "bug" that causes it to prefer the expired certificate. Without going into a lot of detail and contentious debate over how this kind of thing should work, the fact of the matter is that you're probably reading this because you've encountered it.

FreeBSD (and therefore FreeNAS) 11 and before all use OpenSSL 1.0.2, and FreeNAS uses the Mozilla NSS port (ports/ca_root_nss) to define its default trust list. This list includes the expired DST Root X3 certificate, so after September 30th, FreeNAS clients attempting to connect to a LetsEncrypt-based web site are likely to hit the "certificate is expired" error.

This does not mean the remote server's certificate. It is, instead, an expired certificate in your own trust store.

To resolve it, you need to locate and eliminate the old DST Root X3 certificate. For the base FreeBSD/FreeNAS system, this is the Netscape NSS list of CA's, which is located in /usr/local/share/certs/ca-root-nss.crt as a single file. Jails and other applications may have their own trust stores, and may need custom adjustment of various types.

You may remove the offending certificate from /usr/local/share/certs/ca-root-nss.crt with an editor.

Note that the decoded (human-readable) text version of the certificate comes BEFORE the encoded (machine-readable) certificate, so you would want to make a backup of the file, then try deleting the text version starting at

Code:
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            44:af:b0:80:d6:a3:27:ba:89:30:39:86:2e:f8:40:6b
    Signature Algorithm: sha1WithRSAEncryption
        Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3
        Validity
            Not Before: Sep 30 21:12:19 2000 GMT
            Not After : Sep 30 14:01:15 2021 GMT
where you will eventually see a BEGIN CERTIFICATE and a bunch of ASCII encoded binary garbage. Delete all of this, all the way through the following
Code:
-----END CERTIFICATE-----


The total deleted section should be 81 lines.

The ca-root-nss on older FreeNAS systems is probably a bit out of date, and the file that we're using for our installs here is nss-3.71, which has been manually edited to remove the "bad" DST certificate. You can get a copy at

Code:
# fetch --no-verify-peer https://extranet.www.sol.net/files/misc/ca-root-nss.crt.src


if you wish. I do not recommend doing this. SSL is the foundation of trust on the Internet, and you SHOULD NOT blindly accept my assurances that this is nss-3.71 with a single certificate deleted. You are better off acquiring nss-3.71 yourself and removing the certificate in question yourself. Because this is certainly newer than any shipped FreeNAS ca-root-nss file, it will have a number of differences, and this represents some risk in that you have effectively allowed some rando Internet crank define your SSL trust. On the other hand, I'm not actually a rando Internet crank, I do infrastructure engineering for NTP.ORG and if you've got your NTP servers set to pool.ntp.org, then you are already relying on services I've engineered. ;-)

I am absolutely an Internet crank, just not a rando one.

Note that you would need to move the downloaded file into place once you've inspected and approved of it:

Code:
# cp  /usr/local/share/certs/ca-root-nss.crt  /usr/local/share/certs/ca-root-nss.crt.old
# cp  ca-root-nss.crt.src  /usr/local/share/certs/ca-root-nss.crt


You are strongly encouraged to instead download a fresh copy of FreeBSD, update your ports, install the security/ca_root_nss port, and then copy that file to your FreeNAS after editing out the DST Root X3 certificate.

This hasn't solved your problem? Need to update some other trust store? There's already some hints in the thread over here, or feel free to ask below. I fully understand how frustrating this is. As someone who is experienced with this stuff, it's exasperating to me too. Let me know how and where this topic needs to be improved. I write these things for you, and the community exists to help each other through these challenging bits.
 

djh_au

Neophyte
Joined
Aug 11, 2021
Messages
4
ok .. I have deleted this entry from the suggested file locations in the plugin that I have install (Sonarr).
I have restarted Sonarr and am stuck in exactly the same broken spot, so where do the jails possibly store this certificate?
 

jgreco

Resident Grinch
Moderator
Joined
May 29, 2011
Messages
14,179
That's the wrong way to look at it. Jails do not store the certificate, and neither does a default FreeBSD host.

The base behavior on a newly installed FreeBSD host is that there aren't any SSL certificates, and because a jail is often just an untarred FreeBSD install, that's also a jail's default behavior.

The FreeBSD system's OpenSSL looks in /etc/ssl/certs or /etc/ssl/cert.pem, depending on whether it is in directory or jumbo amalgamated file mode, and typically on a FreeBSD system, /etc/ssl/cert.pem is pointed at /usr/local/share/certs/ca-root-nss.pem when the Mozilla NSS package is installed. But this is relatively arbitrary. If you've also installed the OpenSSL port, you may find /usr/local/etc/ssl/certs or /usr/local/etc/ssl/cert.pem populated, and some of us purists install a full table at /etc/ssl/certs/* along with the proper hashing to "make it go fast."

More importantly, since any app is allowed to specify its own list of acceptable root CA's, this is really only a starting point for the joyous possibilities of where to track this all down. For example, Java has its own trust store embedded, Firefox has its own trust store embedded, etc.

So the correct answer to your question of "so where do jails store it" is "could literally be anywhere, depending on who wrote the jail and what they did".

There are two things you might try that could potentially do a good job of locating this. One is to look for "DST Root X3":

Code:
# cd /mnt/mypool/myjailroot
# find . -type f -print0 | xargs -0 grep "DST Root X3"


will search EVERY FILE in a jail for the human-readable decoded bit of the certificate. This is only a substring of course, and there's also no guarantee that a human-readable decoded bit was included in a certificate list.

You could also try this with data from the certificate itself. The second line contains only alnum data, so:

Code:
# cd /mnt/mypool/myjailroot
# find . -type -print0 | xargs -0 grep "MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT"


This is still not a guarantee of finding it, since it is POSSIBLE, unusual, but possible, for certificates to be stored in other formats.
 

hmvh1

Neophyte
Joined
Aug 26, 2021
Messages
6
A temporary fix in sonarr or radarr is to disable the Certificate Validation under general settings.
 

djh_au

Neophyte
Joined
Aug 11, 2021
Messages
4
This is still not a guarantee of finding it, since it is POSSIBLE, unusual, but possible, for certificates to be stored in other formats.
I had a crack and came to this conclusion .. it's stored in a way I can't find it.
I have gone the hack path of stopping validation as @hmvh1 suggested. I'll keep an eye on updates on hope they put one out soon.
 

SlackerDude

Member
Joined
Feb 1, 2014
Messages
47
A temporary fix in sonarr or radarr is to disable the Certificate Validation under general settings.
I tried this in Sonarr and Radarr, but the issue remains, unfortunately.
 

catnas

Member
Joined
Dec 12, 2015
Messages
56
I am experiencing a similar issue when attempting to create a new jail under 11.3-U5. The fix mentioned here corrected the issue in the web GUI for updating FreeNAS itself, but it didn't correct the jail-creation issue. Any thoughts?

 

sovcali

Neophyte
Joined
Oct 16, 2021
Messages
4
I am experiencing a similar issue when attempting to create a new jail under 11.3-U5. The fix mentioned here corrected the issue in the web GUI for updating FreeNAS itself, but it didn't correct the jail-creation issue. Any thoughts?

I'm currently having this exact same error, no solutions over internet, has anyone been able to fix this?
 

jgreco

Resident Grinch
Moderator
Joined
May 29, 2011
Messages
14,179
I don't use FreeNAS jails and I don't have any FreeNAS hosts in roles where they can even talk to the Internet, so I'm not likely to be able to duplicate this easily.

However, it would still be very useful if you could describe exactly what you've done to get to the posted error. I might be more inclined to set up a testing VM that can reach the Internet if I had a solid set of steps such as "I went to Jails, used these configuration things, tried to do this other thing, did steps A B and C, and got this error." Pretend like I've never used the FreeNAS GUI for its jail system, but I know FreeNAS basics very well.

Please remember that most of the people here aren't iX staffers but like to be helpful if reasonably possible, me included.
 

sovcali

Neophyte
Joined
Oct 16, 2021
Messages
4
I don't use FreeNAS jails and I don't have any FreeNAS hosts in roles where they can even talk to the Internet, so I'm not likely to be able to duplicate this easily.

However, it would still be very useful if you could describe exactly what you've done to get to the posted error. I might be more inclined to set up a testing VM that can reach the Internet if I had a solid set of steps such as "I went to Jails, used these configuration things, tried to do this other thing, did steps A B and C, and got this error." Pretend like I've never used the FreeNAS GUI for its jail system, but I know FreeNAS basics very well.

Please remember that most of the people here aren't iX staffers but like to be helpful if reasonably possible, me included.
Thanks for your help, in my case I upgraded from 11.1 to 11.2-U8 and the first thing I notice was the "update" section of FreeNAS didn't show updates and also some errors related to DST Root X3 but that got solved when followed steps of this thread. Next thing I did was to try creating jails with iocage, here's when the error described by @catnas showed. I'm pretty sure this is not a bad network configuration because I can ping everything related, but not sure what else to do.
 

nakosenk

Newbie
Joined
Oct 16, 2021
Messages
1
The issue is that iocage is using Python, which uses its own cacert.pem file located in /usr/local/lib/python3.7/site-packages/certifi/cacert.pem. I was able to fix the problem by symlinking the updated ca-root-nss.crt file like so:

Code:
# mv /usr/local/lib/python3.7/site-packages/certifi/cacert.pem /usr/local/lib/python3.7/site-packages/certifi/cacert.pem.old
# ln -s /usr/local/share/certs/ca-root-nss.crt /usr/local/lib/python3.7/site-packages/certifi/cacert.pem
 

sovcali

Neophyte
Joined
Oct 16, 2021
Messages
4
The issue is that iocage is using Python, which uses its own cacert.pem file located in /usr/local/lib/python3.7/site-packages/certifi/cacert.pem. I was able to fix the problem by symlinking the updated ca-root-nss.crt file like so:

Code:
# mv /usr/local/lib/python3.7/site-packages/certifi/cacert.pem /usr/local/lib/python3.7/site-packages/certifi/cacert.pem.old
# ln -s /usr/local/share/certs/ca-root-nss.crt /usr/local/lib/python3.7/site-packages/certifi/cacert.pem
This is the answer I was looking for, thank you very much!

In my case I just changed python3.7 to python3.6 and that resolved the problem.
 

hmvh1

Neophyte
Joined
Aug 26, 2021
Messages
6
I have had some time today to try and find a solution in sonar. I updated the cacert.pem, cert.pem and ca-root-nss.crt in the jail, and still get the same error.

Logs display a mono error - Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
I have tried to update the mono certificates store with 'cert-sync --user /usr/local/share/certs/ca-root-nss.crt'
I see the certificates updated in mono, but I still can't get sonar to connect with an indexer when the Certificate Validation is set to Enable.

Anyone else figure out the solution?

Code:
###SONARR ERROR MESSAGE####
#######REMOVED TOKENS#######

2021-10-17 12:13:56.0|Error|X509CertificateValidationService|Certificate validation for https://api.dognzb.cr/api?t=caps&apikey=(removed) failed. RemoteCertificateChainErrors
2021-10-17 12:13:57.6|Error|X509CertificateValidationService|Certificate validation for https://api.dognzb.cr/api?t=caps&apikey=(removed) failed. RemoteCertificateChainErrors
2021-10-17 12:13:57.6|Warn|Newznab|Unable to connect to indexer

[v3.0.6.1342] System.Net.WebException: Error: TrustFailure (Authentication failed, see inner exception.): 'https://api.dognzb.cr/api?t=caps&apikey=(removed) ---> System.Net.WebException: Error: TrustFailure (Authentication failed, see inner exception.) ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. ---> Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
  at /wrkdirs/usr/ports/lang/mono6.8/work/mono-6.8.0.123/external/boringssl/ssl/handshake_client.c:1132
  at Mono.Btls.MonoBtlsContext.ProcessHandshake () [0x00048] in <>:0
  at Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake (Mono.Net.Security.AsyncOperationStatus status, System.Boolean renegotiate) [0x000da] in <>:0
  at (wrapper remoting-invoke-with-check) Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake(Mono.Net.Security.AsyncOperationStatus,bool)
  at Mono.Net.Security.AsyncHandshakeRequest.Run (Mono.Net.Security.AsyncOperationStatus status) [0x00006] in <>:0
  at Mono.Net.Security.AsyncProtocolRequest.ProcessOperation (System.Threading.CancellationToken cancellationToken) [0x000fc] in <>:0
   --- End of inner exception stack trace ---
  at Mono.Net.Security.MobileAuthenticatedStream.ProcessAuthentication (System.Boolean runSynchronously, Mono.Net.Security.MonoSslAuthenticationOptions options, System.Threading.CancellationToken cancellationToken) [0x00262] in <>:0
  at Mono.Net.Security.MonoTlsStream.CreateStream (System.Net.WebConnectionTunnel tunnel, System.Threading.CancellationToken cancellationToken) [0x0016a] in <>:0
  at System.Net.WebConnection.CreateStream (System.Net.WebOperation operation, System.Boolean reused, System.Threading.CancellationToken cancellationToken) [0x001ba] in <>:0
   --- End of inner exception stack trace ---
  at System.Net.WebConnection.CreateStream (System.Net.WebOperation operation, System.Boolean reused, System.Threading.CancellationToken cancellationToken) [0x0021a] in <>:0
  at System.Net.WebConnection.InitConnection (System.Net.WebOperation operation, System.Threading.CancellationToken cancellationToken) [0x00141] in <>:0
  at System.Net.WebOperation.Run () [0x0009a] in <>:0
  at System.Net.WebCompletionSource`1[T].WaitForCompletion () [0x00094] in <>:0
  at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <>:0
  at System.Net.HttpWebRequest.GetResponse () [0x00016] in <>:0
  at NzbDrone.Common.Http.Dispatchers.ManagedHttpDispatcher.GetResponse (NzbDrone.Common.Http.HttpRequest request, System.Net.CookieContainer cookies) [0x00123] in <>:0
   --- End of inner exception stack trace ---
  at NzbDrone.Common.Http.Dispatchers.ManagedHttpDispatcher.GetResponse (NzbDrone.Common.Http.HttpRequest request, System.Net.CookieContainer cookies) [0x001ec] in <>:0
  at NzbDrone.Common.Http.HttpClient.ExecuteRequest (NzbDrone.Common.Http.HttpRequest request, System.Net.CookieContainer cookieContainer) [0x00086] in <>:0
  at NzbDrone.Common.Http.HttpClient.Execute (NzbDrone.Common.Http.HttpRequest request) [0x00008] in <a1cd7a1bf9df40cfa6971ab77142a163>:0
  at NzbDrone.Common.Http.HttpClient.Get (NzbDrone.Common.Http.HttpRequest request) [0x00007] in <a>:0
  at NzbDrone.Core.Indexers.Newznab.NewznabCapabilitiesProvider.FetchCapabilities (NzbDrone.Core.Indexers.Newznab.NewznabSettings indexerSettings) [0x000a1] in <>:0
  at NzbDrone.Core.Indexers.Newznab.NewznabCapabilitiesProvider+<>c__DisplayClass4_0.<GetCapabilities>b__0 () [0x00000] in <d>:0
  at NzbDrone.Common.Cache.Cached`1[T].Get (System.String key, System.Func`1[TResult] function, System.Nullable`1[T] lifeTime) [0x000b1] in <>:0
  at NzbDrone.Core.Indexers.Newznab.NewznabCapabilitiesProvider.GetCapabilities (NzbDrone.Core.Indexers.Newznab.NewznabSettings indexerSettings) [0x00046] in <>:0
  at NzbDrone.Core.Indexers.Newznab.Newznab.get_PageSize () [0x0000c] in <>:0
  at NzbDrone.Core.Indexers.Newznab.Newznab.GetRequestGenerator () [0x0000b] in <2>:0
  at NzbDrone.Core.Indexers.HttpIndexerBase`1[TSettings].TestConnection () [0x00007] in <>:0

2021-10-17 12:13:57.6|Warn|SonarrErrorPipeline|Invalid request Validation failed:
-- : Unable to connect to indexer, check the log for more details
 

alexr

Member
Joined
Apr 14, 2016
Messages
49
(migrated here from another thread)

"distributions" or "distros" are a Linux community term that refers to either a flavor of Linux ("Debian", "Red Hat", etc) or, confusingly, sometimes, a specific release of a flavor.

FreeBSD has "releases". Newer releases of FreeBSD have also moved to OpenSSL 1.1.1, which doesn't have the LetsEnc issue.

FreeNAS is based on FreeBSD 11.(n)R or earlier, and definitely does not have anything called "certctl".
Thanks for explaining. (I didn't think using "distributions" would sound so wrong. FWIW, I'm the guy who got PlayStation onto FreeBSD, so I'm not totally a bozo. I swear.)

It's pretty clear that this is a general shortcoming of OpenSSL on many platforms, as trust store management has generally been highly inconsistent (often nonexistent) between platforms. As noted a few messages above, there are several places on a default FreeBSD system with ca-root-nss installed that have the CA file, and nothing that even breaks up the file into the more efficient hashed version. If I get time later, maybe I'll go see if I can figure out wtf "certctl" is.
https://github.com/freebsd/freebsd-src/tree/main/usr.sbin/certctl Looks like the directory has since been renamed 'untrusted'. However, I haven't quite found where in the OpenSSL source it looks at anything past cert.pem.
 

Meya

Neophyte
Joined
Dec 5, 2018
Messages
9
You may remove the offending certificate from /usr/local/share/certs/ca-root-nss.crt with an editor.
I was having issues when trying to update from 11.3 to 12.0 with a 'base-os' error:
1634684837923.png


After browsing 151 certificates and deleting the expired ones, I was able to finally upgrade to TrueNAS 12.0! This post helped although it was a tad too technical for me :)

Thank you!
 
Top