SSL certificate problem: certificate has expired -- the OpenSSL 1.0.2 vs LetsEncrypt issue

How helpful was this to resolving the certificate is expired issue?

  • Not technical enough

    Votes: 0 0.0%
  • You suck, Grinch!

    Votes: 0 0.0%

  • Total voters
    8
  • Poll closed .

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Posted in response to a staff request, this is intended to help answer the "certificate is expired" issues.

SSL underpins most network session security on the Internet. Historically, this was implemented with self-signed certificates (a bad idea), or through commercial certificate authorities (a pricey proposition). In a bid to see the Internet default to securing everything (which is a bad idea of a different sort), several industry players cobbled together a free, automatic certificate authority called LetsEncrypt, and released software to make it easy to get valid SSL certificates for your website (generally a good idea).

To get this bootstrapped, they initially signed this with the IdenTrust DST Root X3 certificate, which expired at the end of September 2021. LetsEncrypt has experienced wild success, and is now signing with their own ISRG Root X1 certificate, now trusted in most browsers and SSL trust stores. I'm not making a full accounting or description of the bootstrap and cross-signing processes used to bring this online. You can Google all that.

However, OpenSSL 1.0.2 has a "bug" that causes it to prefer the expired certificate. Without going into a lot of detail and contentious debate over how this kind of thing should work, the fact of the matter is that you're probably reading this because you've encountered it.

FreeBSD (and therefore FreeNAS) 11 and before all use OpenSSL 1.0.2, and FreeNAS uses the Mozilla NSS port (ports/ca_root_nss) to define its default trust list. This list includes the expired DST Root X3 certificate, so after September 30th, FreeNAS clients attempting to connect to a LetsEncrypt-based web site are likely to hit the "certificate is expired" error.

This does not mean the remote server's certificate. It is, instead, an expired certificate in your own trust store.

To resolve it, you need to locate and eliminate the old DST Root X3 certificate. For the base FreeBSD/FreeNAS system, this is the Netscape NSS list of CA's, which is located in /usr/local/share/certs/ca-root-nss.crt as a single file. Jails and other applications may have their own trust stores, and may need custom adjustment of various types.

You may remove the offending certificate from /usr/local/share/certs/ca-root-nss.crt with an editor.

Note that the decoded (human-readable) text version of the certificate comes BEFORE the encoded (machine-readable) certificate, so you would want to make a backup of the file, then try deleting the text version starting at

Code:
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            44:af:b0:80:d6:a3:27:ba:89:30:39:86:2e:f8:40:6b
    Signature Algorithm: sha1WithRSAEncryption
        Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3
        Validity
            Not Before: Sep 30 21:12:19 2000 GMT
            Not After : Sep 30 14:01:15 2021 GMT
where you will eventually see a BEGIN CERTIFICATE and a bunch of ASCII encoded binary garbage. Delete all of this, all the way through the following
Code:
-----END CERTIFICATE-----


The total deleted section should be 81 lines.

The ca-root-nss on older FreeNAS systems is probably a bit out of date, and the file that we're using for our installs here is nss-3.71, which has been manually edited to remove the "bad" DST certificate. You can get a copy at

Code:
# fetch --no-verify-peer https://extranet.www.sol.net/files/misc/ca-root-nss.crt.src


if you wish. I do not recommend doing this. SSL is the foundation of trust on the Internet, and you SHOULD NOT blindly accept my assurances that this is nss-3.71 with a single certificate deleted. You are better off acquiring nss-3.71 yourself and removing the certificate in question yourself. Because this is certainly newer than any shipped FreeNAS ca-root-nss file, it will have a number of differences, and this represents some risk in that you have effectively allowed some rando Internet crank define your SSL trust. On the other hand, I'm not actually a rando Internet crank, I do infrastructure engineering for NTP.ORG and if you've got your NTP servers set to pool.ntp.org, then you are already relying on services I've engineered. ;-)

I am absolutely an Internet crank, just not a rando one.

Note that you would need to move the downloaded file into place once you've inspected and approved of it:

Code:
# cp  /usr/local/share/certs/ca-root-nss.crt  /usr/local/share/certs/ca-root-nss.crt.old
# cp  ca-root-nss.crt.src  /usr/local/share/certs/ca-root-nss.crt


You are strongly encouraged to instead download a fresh copy of FreeBSD, update your ports, install the security/ca_root_nss port, and then copy that file to your FreeNAS after editing out the DST Root X3 certificate.

This hasn't solved your problem? Need to update some other trust store? There's already some hints in the thread over here, or feel free to ask below. I fully understand how frustrating this is. As someone who is experienced with this stuff, it's exasperating to me too. Let me know how and where this topic needs to be improved. I write these things for you, and the community exists to help each other through these challenging bits.
 

djh_au

Cadet
Joined
Aug 11, 2021
Messages
4
ok .. I have deleted this entry from the suggested file locations in the plugin that I have install (Sonarr).
I have restarted Sonarr and am stuck in exactly the same broken spot, so where do the jails possibly store this certificate?
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
That's the wrong way to look at it. Jails do not store the certificate, and neither does a default FreeBSD host.

The base behavior on a newly installed FreeBSD host is that there aren't any SSL certificates, and because a jail is often just an untarred FreeBSD install, that's also a jail's default behavior.

The FreeBSD system's OpenSSL looks in /etc/ssl/certs or /etc/ssl/cert.pem, depending on whether it is in directory or jumbo amalgamated file mode, and typically on a FreeBSD system, /etc/ssl/cert.pem is pointed at /usr/local/share/certs/ca-root-nss.pem when the Mozilla NSS package is installed. But this is relatively arbitrary. If you've also installed the OpenSSL port, you may find /usr/local/etc/ssl/certs or /usr/local/etc/ssl/cert.pem populated, and some of us purists install a full table at /etc/ssl/certs/* along with the proper hashing to "make it go fast."

More importantly, since any app is allowed to specify its own list of acceptable root CA's, this is really only a starting point for the joyous possibilities of where to track this all down. For example, Java has its own trust store embedded, Firefox has its own trust store embedded, etc.

So the correct answer to your question of "so where do jails store it" is "could literally be anywhere, depending on who wrote the jail and what they did".

There are two things you might try that could potentially do a good job of locating this. One is to look for "DST Root X3":

Code:
# cd /mnt/mypool/myjailroot
# find . -type f -print0 | xargs -0 grep "DST Root X3"


will search EVERY FILE in a jail for the human-readable decoded bit of the certificate. This is only a substring of course, and there's also no guarantee that a human-readable decoded bit was included in a certificate list.

You could also try this with data from the certificate itself. The second line contains only alnum data, so:

Code:
# cd /mnt/mypool/myjailroot
# find . -type -print0 | xargs -0 grep "MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT"


This is still not a guarantee of finding it, since it is POSSIBLE, unusual, but possible, for certificates to be stored in other formats.
 

hmvh1

Dabbler
Joined
Aug 26, 2021
Messages
15
A temporary fix in sonarr or radarr is to disable the Certificate Validation under general settings.
 

djh_au

Cadet
Joined
Aug 11, 2021
Messages
4
This is still not a guarantee of finding it, since it is POSSIBLE, unusual, but possible, for certificates to be stored in other formats.

I had a crack and came to this conclusion .. it's stored in a way I can't find it.
I have gone the hack path of stopping validation as @hmvh1 suggested. I'll keep an eye on updates on hope they put one out soon.
 

SlackerDude

Explorer
Joined
Feb 1, 2014
Messages
76
A temporary fix in sonarr or radarr is to disable the Certificate Validation under general settings.
I tried this in Sonarr and Radarr, but the issue remains, unfortunately.
 

catnas

Explorer
Joined
Dec 12, 2015
Messages
57
I am experiencing a similar issue when attempting to create a new jail under 11.3-U5. The fix mentioned here corrected the issue in the web GUI for updating FreeNAS itself, but it didn't correct the jail-creation issue. Any thoughts?

 

sovcali

Cadet
Joined
Oct 16, 2021
Messages
4
I am experiencing a similar issue when attempting to create a new jail under 11.3-U5. The fix mentioned here corrected the issue in the web GUI for updating FreeNAS itself, but it didn't correct the jail-creation issue. Any thoughts?

I'm currently having this exact same error, no solutions over internet, has anyone been able to fix this?
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
I don't use FreeNAS jails and I don't have any FreeNAS hosts in roles where they can even talk to the Internet, so I'm not likely to be able to duplicate this easily.

However, it would still be very useful if you could describe exactly what you've done to get to the posted error. I might be more inclined to set up a testing VM that can reach the Internet if I had a solid set of steps such as "I went to Jails, used these configuration things, tried to do this other thing, did steps A B and C, and got this error." Pretend like I've never used the FreeNAS GUI for its jail system, but I know FreeNAS basics very well.

Please remember that most of the people here aren't iX staffers but like to be helpful if reasonably possible, me included.
 

sovcali

Cadet
Joined
Oct 16, 2021
Messages
4
I don't use FreeNAS jails and I don't have any FreeNAS hosts in roles where they can even talk to the Internet, so I'm not likely to be able to duplicate this easily.

However, it would still be very useful if you could describe exactly what you've done to get to the posted error. I might be more inclined to set up a testing VM that can reach the Internet if I had a solid set of steps such as "I went to Jails, used these configuration things, tried to do this other thing, did steps A B and C, and got this error." Pretend like I've never used the FreeNAS GUI for its jail system, but I know FreeNAS basics very well.

Please remember that most of the people here aren't iX staffers but like to be helpful if reasonably possible, me included.
Thanks for your help, in my case I upgraded from 11.1 to 11.2-U8 and the first thing I notice was the "update" section of FreeNAS didn't show updates and also some errors related to DST Root X3 but that got solved when followed steps of this thread. Next thing I did was to try creating jails with iocage, here's when the error described by @catnas showed. I'm pretty sure this is not a bad network configuration because I can ping everything related, but not sure what else to do.
 

nakosenk

Cadet
Joined
Oct 16, 2021
Messages
1
The issue is that iocage is using Python, which uses its own cacert.pem file located in /usr/local/lib/python3.7/site-packages/certifi/cacert.pem. I was able to fix the problem by symlinking the updated ca-root-nss.crt file like so:

Code:
# mv /usr/local/lib/python3.7/site-packages/certifi/cacert.pem /usr/local/lib/python3.7/site-packages/certifi/cacert.pem.old
# ln -s /usr/local/share/certs/ca-root-nss.crt /usr/local/lib/python3.7/site-packages/certifi/cacert.pem
 

sovcali

Cadet
Joined
Oct 16, 2021
Messages
4
The issue is that iocage is using Python, which uses its own cacert.pem file located in /usr/local/lib/python3.7/site-packages/certifi/cacert.pem. I was able to fix the problem by symlinking the updated ca-root-nss.crt file like so:

Code:
# mv /usr/local/lib/python3.7/site-packages/certifi/cacert.pem /usr/local/lib/python3.7/site-packages/certifi/cacert.pem.old
# ln -s /usr/local/share/certs/ca-root-nss.crt /usr/local/lib/python3.7/site-packages/certifi/cacert.pem
This is the answer I was looking for, thank you very much!

In my case I just changed python3.7 to python3.6 and that resolved the problem.
 

hmvh1

Dabbler
Joined
Aug 26, 2021
Messages
15
I have had some time today to try and find a solution in sonar. I updated the cacert.pem, cert.pem and ca-root-nss.crt in the jail, and still get the same error.

Logs display a mono error - Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
I have tried to update the mono certificates store with 'cert-sync --user /usr/local/share/certs/ca-root-nss.crt'
I see the certificates updated in mono, but I still can't get sonar to connect with an indexer when the Certificate Validation is set to Enable.

Anyone else figure out the solution?

Code:
###SONARR ERROR MESSAGE####
#######REMOVED TOKENS#######

2021-10-17 12:13:56.0|Error|X509CertificateValidationService|Certificate validation for https://api.dognzb.cr/api?t=caps&apikey=(removed) failed. RemoteCertificateChainErrors
2021-10-17 12:13:57.6|Error|X509CertificateValidationService|Certificate validation for https://api.dognzb.cr/api?t=caps&apikey=(removed) failed. RemoteCertificateChainErrors
2021-10-17 12:13:57.6|Warn|Newznab|Unable to connect to indexer

[v3.0.6.1342] System.Net.WebException: Error: TrustFailure (Authentication failed, see inner exception.): 'https://api.dognzb.cr/api?t=caps&apikey=(removed) ---> System.Net.WebException: Error: TrustFailure (Authentication failed, see inner exception.) ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. ---> Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
  at /wrkdirs/usr/ports/lang/mono6.8/work/mono-6.8.0.123/external/boringssl/ssl/handshake_client.c:1132
  at Mono.Btls.MonoBtlsContext.ProcessHandshake () [0x00048] in <>:0
  at Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake (Mono.Net.Security.AsyncOperationStatus status, System.Boolean renegotiate) [0x000da] in <>:0
  at (wrapper remoting-invoke-with-check) Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake(Mono.Net.Security.AsyncOperationStatus,bool)
  at Mono.Net.Security.AsyncHandshakeRequest.Run (Mono.Net.Security.AsyncOperationStatus status) [0x00006] in <>:0
  at Mono.Net.Security.AsyncProtocolRequest.ProcessOperation (System.Threading.CancellationToken cancellationToken) [0x000fc] in <>:0
   --- End of inner exception stack trace ---
  at Mono.Net.Security.MobileAuthenticatedStream.ProcessAuthentication (System.Boolean runSynchronously, Mono.Net.Security.MonoSslAuthenticationOptions options, System.Threading.CancellationToken cancellationToken) [0x00262] in <>:0
  at Mono.Net.Security.MonoTlsStream.CreateStream (System.Net.WebConnectionTunnel tunnel, System.Threading.CancellationToken cancellationToken) [0x0016a] in <>:0
  at System.Net.WebConnection.CreateStream (System.Net.WebOperation operation, System.Boolean reused, System.Threading.CancellationToken cancellationToken) [0x001ba] in <>:0
   --- End of inner exception stack trace ---
  at System.Net.WebConnection.CreateStream (System.Net.WebOperation operation, System.Boolean reused, System.Threading.CancellationToken cancellationToken) [0x0021a] in <>:0
  at System.Net.WebConnection.InitConnection (System.Net.WebOperation operation, System.Threading.CancellationToken cancellationToken) [0x00141] in <>:0
  at System.Net.WebOperation.Run () [0x0009a] in <>:0
  at System.Net.WebCompletionSource`1[T].WaitForCompletion () [0x00094] in <>:0
  at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <>:0
  at System.Net.HttpWebRequest.GetResponse () [0x00016] in <>:0
  at NzbDrone.Common.Http.Dispatchers.ManagedHttpDispatcher.GetResponse (NzbDrone.Common.Http.HttpRequest request, System.Net.CookieContainer cookies) [0x00123] in <>:0
   --- End of inner exception stack trace ---
  at NzbDrone.Common.Http.Dispatchers.ManagedHttpDispatcher.GetResponse (NzbDrone.Common.Http.HttpRequest request, System.Net.CookieContainer cookies) [0x001ec] in <>:0
  at NzbDrone.Common.Http.HttpClient.ExecuteRequest (NzbDrone.Common.Http.HttpRequest request, System.Net.CookieContainer cookieContainer) [0x00086] in <>:0
  at NzbDrone.Common.Http.HttpClient.Execute (NzbDrone.Common.Http.HttpRequest request) [0x00008] in <a1cd7a1bf9df40cfa6971ab77142a163>:0
  at NzbDrone.Common.Http.HttpClient.Get (NzbDrone.Common.Http.HttpRequest request) [0x00007] in <a>:0
  at NzbDrone.Core.Indexers.Newznab.NewznabCapabilitiesProvider.FetchCapabilities (NzbDrone.Core.Indexers.Newznab.NewznabSettings indexerSettings) [0x000a1] in <>:0
  at NzbDrone.Core.Indexers.Newznab.NewznabCapabilitiesProvider+<>c__DisplayClass4_0.<GetCapabilities>b__0 () [0x00000] in <d>:0
  at NzbDrone.Common.Cache.Cached`1[T].Get (System.String key, System.Func`1[TResult] function, System.Nullable`1[T] lifeTime) [0x000b1] in <>:0
  at NzbDrone.Core.Indexers.Newznab.NewznabCapabilitiesProvider.GetCapabilities (NzbDrone.Core.Indexers.Newznab.NewznabSettings indexerSettings) [0x00046] in <>:0
  at NzbDrone.Core.Indexers.Newznab.Newznab.get_PageSize () [0x0000c] in <>:0
  at NzbDrone.Core.Indexers.Newznab.Newznab.GetRequestGenerator () [0x0000b] in <2>:0
  at NzbDrone.Core.Indexers.HttpIndexerBase`1[TSettings].TestConnection () [0x00007] in <>:0

2021-10-17 12:13:57.6|Warn|SonarrErrorPipeline|Invalid request Validation failed:
-- : Unable to connect to indexer, check the log for more details
 

alexr

Explorer
Joined
Apr 14, 2016
Messages
59
(migrated here from another thread)

"distributions" or "distros" are a Linux community term that refers to either a flavor of Linux ("Debian", "Red Hat", etc) or, confusingly, sometimes, a specific release of a flavor.

FreeBSD has "releases". Newer releases of FreeBSD have also moved to OpenSSL 1.1.1, which doesn't have the LetsEnc issue.

FreeNAS is based on FreeBSD 11.(n)R or earlier, and definitely does not have anything called "certctl".

Thanks for explaining. (I didn't think using "distributions" would sound so wrong. FWIW, I'm the guy who got PlayStation onto FreeBSD, so I'm not totally a bozo. I swear.)

It's pretty clear that this is a general shortcoming of OpenSSL on many platforms, as trust store management has generally been highly inconsistent (often nonexistent) between platforms. As noted a few messages above, there are several places on a default FreeBSD system with ca-root-nss installed that have the CA file, and nothing that even breaks up the file into the more efficient hashed version. If I get time later, maybe I'll go see if I can figure out wtf "certctl" is.
https://github.com/freebsd/freebsd-src/tree/main/usr.sbin/certctl Looks like the directory has since been renamed 'untrusted'. However, I haven't quite found where in the OpenSSL source it looks at anything past cert.pem.
 

Meya

Dabbler
Joined
Dec 5, 2018
Messages
39
You may remove the offending certificate from /usr/local/share/certs/ca-root-nss.crt with an editor.
I was having issues when trying to update from 11.3 to 12.0 with a 'base-os' error:
1634684837923.png


After browsing 151 certificates and deleting the expired ones, I was able to finally upgrade to TrueNAS 12.0! This post helped although it was a tad too technical for me :)

Thank you!
 

hmvh1

Dabbler
Joined
Aug 26, 2021
Messages
15
I have had some time today to try and find a solution in sonar. I updated the cacert.pem, cert.pem and ca-root-nss.crt in the jail, and still get the same error.

Logs display a mono error - Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
I have tried to update the mono certificates store with 'cert-sync --user /usr/local/share/certs/ca-root-nss.crt'
I see the certificates updated in mono, but I still can't get sonar to connect with an indexer when the Certificate Validation is set to Enable.

Anyone else figure out the solution?

Code:
###SONARR ERROR MESSAGE####
#######REMOVED TOKENS#######

2021-10-17 12:13:56.0|Error|X509CertificateValidationService|Certificate validation for https://api.dognzb.cr/api?t=caps&apikey=(removed) failed. RemoteCertificateChainErrors
2021-10-17 12:13:57.6|Error|X509CertificateValidationService|Certificate validation for https://api.dognzb.cr/api?t=caps&apikey=(removed) failed. RemoteCertificateChainErrors
2021-10-17 12:13:57.6|Warn|Newznab|Unable to connect to indexer

[v3.0.6.1342] System.Net.WebException: Error: TrustFailure (Authentication failed, see inner exception.): 'https://api.dognzb.cr/api?t=caps&apikey=(removed) ---> System.Net.WebException: Error: TrustFailure (Authentication failed, see inner exception.) ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. ---> Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
  at /wrkdirs/usr/ports/lang/mono6.8/work/mono-6.8.0.123/external/boringssl/ssl/handshake_client.c:1132
  at Mono.Btls.MonoBtlsContext.ProcessHandshake () [0x00048] in <>:0
  at Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake (Mono.Net.Security.AsyncOperationStatus status, System.Boolean renegotiate) [0x000da] in <>:0
  at (wrapper remoting-invoke-with-check) Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake(Mono.Net.Security.AsyncOperationStatus,bool)
  at Mono.Net.Security.AsyncHandshakeRequest.Run (Mono.Net.Security.AsyncOperationStatus status) [0x00006] in <>:0
  at Mono.Net.Security.AsyncProtocolRequest.ProcessOperation (System.Threading.CancellationToken cancellationToken) [0x000fc] in <>:0
   --- End of inner exception stack trace ---
  at Mono.Net.Security.MobileAuthenticatedStream.ProcessAuthentication (System.Boolean runSynchronously, Mono.Net.Security.MonoSslAuthenticationOptions options, System.Threading.CancellationToken cancellationToken) [0x00262] in <>:0
  at Mono.Net.Security.MonoTlsStream.CreateStream (System.Net.WebConnectionTunnel tunnel, System.Threading.CancellationToken cancellationToken) [0x0016a] in <>:0
  at System.Net.WebConnection.CreateStream (System.Net.WebOperation operation, System.Boolean reused, System.Threading.CancellationToken cancellationToken) [0x001ba] in <>:0
   --- End of inner exception stack trace ---
  at System.Net.WebConnection.CreateStream (System.Net.WebOperation operation, System.Boolean reused, System.Threading.CancellationToken cancellationToken) [0x0021a] in <>:0
  at System.Net.WebConnection.InitConnection (System.Net.WebOperation operation, System.Threading.CancellationToken cancellationToken) [0x00141] in <>:0
  at System.Net.WebOperation.Run () [0x0009a] in <>:0
  at System.Net.WebCompletionSource`1[T].WaitForCompletion () [0x00094] in <>:0
  at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <>:0
  at System.Net.HttpWebRequest.GetResponse () [0x00016] in <>:0
  at NzbDrone.Common.Http.Dispatchers.ManagedHttpDispatcher.GetResponse (NzbDrone.Common.Http.HttpRequest request, System.Net.CookieContainer cookies) [0x00123] in <>:0
   --- End of inner exception stack trace ---
  at NzbDrone.Common.Http.Dispatchers.ManagedHttpDispatcher.GetResponse (NzbDrone.Common.Http.HttpRequest request, System.Net.CookieContainer cookies) [0x001ec] in <>:0
  at NzbDrone.Common.Http.HttpClient.ExecuteRequest (NzbDrone.Common.Http.HttpRequest request, System.Net.CookieContainer cookieContainer) [0x00086] in <>:0
  at NzbDrone.Common.Http.HttpClient.Execute (NzbDrone.Common.Http.HttpRequest request) [0x00008] in <a1cd7a1bf9df40cfa6971ab77142a163>:0
  at NzbDrone.Common.Http.HttpClient.Get (NzbDrone.Common.Http.HttpRequest request) [0x00007] in <a>:0
  at NzbDrone.Core.Indexers.Newznab.NewznabCapabilitiesProvider.FetchCapabilities (NzbDrone.Core.Indexers.Newznab.NewznabSettings indexerSettings) [0x000a1] in <>:0
  at NzbDrone.Core.Indexers.Newznab.NewznabCapabilitiesProvider+<>c__DisplayClass4_0.<GetCapabilities>b__0 () [0x00000] in <d>:0
  at NzbDrone.Common.Cache.Cached`1[T].Get (System.String key, System.Func`1[TResult] function, System.Nullable`1[T] lifeTime) [0x000b1] in <>:0
  at NzbDrone.Core.Indexers.Newznab.NewznabCapabilitiesProvider.GetCapabilities (NzbDrone.Core.Indexers.Newznab.NewznabSettings indexerSettings) [0x00046] in <>:0
  at NzbDrone.Core.Indexers.Newznab.Newznab.get_PageSize () [0x0000c] in <>:0
  at NzbDrone.Core.Indexers.Newznab.Newznab.GetRequestGenerator () [0x0000b] in <2>:0
  at NzbDrone.Core.Indexers.HttpIndexerBase`1[TSettings].TestConnection () [0x00007] in <>:0

2021-10-17 12:13:57.6|Warn|SonarrErrorPipeline|Invalid request Validation failed:
-- : Unable to connect to indexer, check the log for more details

I was able to fix my sonarr issue by copying the .mono folder in /root/root/.config/.mono (sonarr jail) to the iocage release folder (/iocage/releases/12.2-RELEASE/root/usr/share/.mono) and restarting the jail. The /user/share/.mono folder should now exits in sonarr. If you don't have the .mono folder in sonarr you need to shell into the jail and run 'cert-sync --user /usr/local/share/certs/ca-root-nss.crt' make sure your ca-root-nss.crt is updated. I used the version linked in the first post of this thread. In Radarr you should see the .mono folder in /user/share/.mono and you should be able to connect to your indexers without errors.
 
Last edited:

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
https://github.com/freebsd/freebsd-src/tree/main/usr.sbin/certctl Looks like the directory has since been renamed 'untrusted'. However, I haven't quite found where in the OpenSSL source it looks at anything past cert.pem.

It doesn't. I finally happened to notice that this was introduced in FreeBSD 12.2R. certctl is basically a shell script to manage hashing the trust store and has replaced(?) the previous ca-merge proposal, I guess...? Which is unfortunate since that was supposed to handle Mono and Java. Which, as people have noted in this thread, is an issue. certctl is mentioned in /usr/sbin/etcupdate, /usr/sbin/freebsd-update, and /usr/sbin/mergemaster but I don't see how it is supposed to work overall. Mmm. On one hand, I'm glad someone finally addressed this...

Anyways, you can't just put a certificate in "untrusted" (or "blacklisted" or "trusted" as mentioned elsewhere) and have that magically work. These directories are only data directories for certctl to use to generate the hashed files in /etc/ssl/certs/ that OpenSSL **ACTUALLY** uses. I generally approve of this since some of us have written our own tools for the job, and that sucks too. But it does look like you are supposed to use certctl to add the DST certificate to the blacklist and then certctl may remove it from OpenSSL's certs/ directory.

I am a little unclear on certctl's exact run points and data flow in the lifecycle of a normal FreeBSD system and will have to look at this more closely later, because I would prefer to have a "standardized" way to do this. Specifically, I don't see how it runs the first time on a new system (maybe it doesn't), and I don't have time to chase this down more right now. But I thought people might want to be brought up to speed.
 

Trianine

Cadet
Joined
Nov 8, 2021
Messages
6
jgreco, First of all, thank you for your work and advice. I get the impression that this is a non-trivial matter without an easy solution, especially for people who have never had to touch certificates or tear apart their installation for the first time. To that point, I'm having a difficult time even approaching you with questions or requests for guidance.

As one example, when you say "Download a fresh copy of FreeBSD," how does one choose between the hundreds available on the FreeBSD website? I don't see any info about this on my TrueNAS installation to make an informed decision. My hardware may differ from others, making the solution very personal. And lastly, don't I need FreeBSD to operate this machine? How do I remove it to install a new one?

Is there any word on this "bug" being repaired? Because currently, I fear that this will end my relationship with TrueNAS.
 

Daisuke

Contributor
Joined
Jun 23, 2011
Messages
1,038
You are strongly encouraged to instead download a fresh copy of FreeBSD, update your ports, install the security/ca_root_nss port, and then copy that file to your FreeNAS after editing out the DST Root X3 certificate.
I don't understand why the end-user needs to do this manually, when this should be a provided TrueNAS update/fix.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
I don't understand why the end-user needs to do this manually, when this should be a provided TrueNAS update/fix.

Because we're discussing FreeNAS, which a lot of people still run, but is no longer officially supported.

Starting in FreeBSD 12.2, a new certificate manager has been added to FreeBSD which might "fix" this situation, but it doesn't backport without some changes, and since FreeBSD 12.2 also uses an unaffected version of OpenSSL, this doesn't pose a real problem for TrueNAS.

However, the userland is also problematic, because of jails, and apps/languages/etc with their own private trust stores. This isn't really a FreeNAS/TrueNAS bug, it's more of a "the Internet community failed to be aware that the gung-ho folks who wanted to deploy SSL everywhere did so with a casual disregard to building a proper long-term plan for their service, and the OpenSSL developers fudged this up too, and several OS's including FreeBSD dragged their heels moving off of OpenSSL 1.0.2 (even if for relatively good reasons) unaware of the eventual issue".
 
Top