TrueNAS-13.0-U3.1 certificate verify failed: certificate has expired?

[M-System]

Hi, i upgraded my system to the latest TrueNAS-13.0-U3.1 yesterday and ever since ive been getting

"Cannot connect to host update.freenas.org:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1134)')]: Automatic update check failed. Please check system network settings"

My network settings have not been changed, everything was running fine before the update

 

[jgreco]

Check your time and date. It's probably off.

 

[Samuel Tai]

This is because the certificate for update.freenas.org expired, as the error message said. Other iX sites with expired certs are icons.freenas.org and update-master.ixsystems.com.

 

[jgreco]

This is because the certificate for update.freenas.org expired, as the error message said. Other iX sites with expired certs are icons.freenas.org and update-master.ixsystems.com.

 

[M-System]

I can confirm the time and date are as they should be.

The default certificate is still in date

freenas_default
external
/C=US/O=iXsystems/CN=localhost/emailAddress=info@ixsystems.com/ST=Tennessee/L=Maryville/subjectAltName=DNS:localhost
20 November, 2022 13:13:16
22 February, 2025 13:13:16

 

[WN1X]

This is because the certificate for update.freenas.org expired, as the error message said. Other iX sites with expired certs are icons.freenas.org and update-master.ixsystems.com.

 

[33_viper_33]

Thanks for the response, but please try to explain in a different way next time. I didn't understand your meaning either at first.

The problem is not on your side. The server you are trying to update from has a certificate out of date. I just confirmed on two separate systems. I believe we are all unable to update.

Is there another server we can direct our system to for updates? How long does it usually take for TrueNAS to update their certificates?

 

[jgreco]

Thanks for the response, but please try to explain in a different way next time.

Perhaps you could outline what sort of explanation would have suited you, if the technically accurate explanation was inappropriate.

The problem is not on your side. The server you are trying to update from has a certificate out of date. I just confirmed on two separate systems. I believe we are all unable to update.

Well, at least, you and many people are unable to update.

Is there another server we can direct our system to for updates?

Yes. You can set up your own server and intercept. Or use a proxy and tell it to ignore certificate errors.

How long does it usually take for TrueNAS to update their certificates?

Well, now, that's the thing. The whole idea behind Let's Encrypt is to automate the renewal of SSL certificates. Unfortunately, the technology underpinning LE has morphed several times, often breaking in the process, making the whole LE experience much less "hands off" than what it should really be. This is what the facepalm thing above is all about. You really need to account for the fallibility of LetsEncrypt; we use the following test (or variations of) over here:

Code:

                                if [ -s /${type}/conf/apphook.runrenewals.precheck ]; then
                                        # shellcheck source=/dev/null
                                        . /${type}/conf/apphook.runrenewals.precheck
                                fi

                                if openssl x509 -checkend 2419200 -noout -in "${cer}" > /dev/null; then
                                        # Certificate good for at least 28 days
                                        if [ -s /${type}/conf/apphook.runrenewals.checkok ]; then
                                                # shellcheck source=/dev/null
                                                . /${type}/conf/apphook.runrenewals.checkok
                                        fi
                                else
                                        if [ -s /${type}/conf/apphook.runrenewals.needrenew ]; then
                                                # shellcheck source=/dev/null
                                                . /${type}/conf/apphook.runrenewals.needrenew
                                        fi

                                        # shellcheck disable=SC2086
                                        ${injail} /${type}/bin/acmesh3 --renew -d "${certdomain}" ${_acme_home} ${_acme_logopts} > "${errorfile}" 2>&1 || ( (echo ""; cat "${errorfile}") | reporterror "acmesh3 renew failed for ${certdomain}")
                                        if ${isatty}; then
                                                cat "${errorfile}"
                                        fi
                                        updated=true

                                        if [ -s /${type}/conf/apphook.runrenewals.renewed ]; then
                                                # shellcheck source=/dev/null
                                                . /${type}/conf/apphook.runrenewals.renewed
                                        fi
                                fi
                                if openssl x509 -checkend 1209600 -noout -in "${cer}" > /dev/null; then
                                        # Certificate good for at least 14 days
                                        auditlog "${certdomain} valid for at least 14 more days"
                                else
                                        echo "letsencrypt acme.sh renewal is failing for ${certdomain}" | reporterror "acmesh3 repeatedly failing to renew certificate for ${certdomain}"
                                fi

which starts trying to renew if a cert has less than twenty-eight days and starts crying for help if there's less than fourteen days remaining.

My impression is that very few folks actually try this hard to make sure their LE renewals are not failing.

 

[NomasTomas]

root@freenas:~ # curl -v -k https://update.freenas.org:443
* Trying 68.70.205.2:443...
* Connected to update.freenas.org (68.70.205.2) port 443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
* subject: CN=update-master.ixsystems.com
* start date: Oct 19 18:45:47 2021 GMT
* expire date: Nov 19 20:46:12 2022 GMT
* issuer: C=US; ST=Arizona; L=Scottsdale; O=GoDaddy.com, Inc.; OU=http://certs.godaddy.com/repository/; CN=Go Daddy Secure Certificate Authority - G2
* SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.

-shrug-

 

[danb35]

My impression is that very few folks actually try this hard to make sure their LE renewals are not failing.

It takes very little effort; LE will notify you 20 days out that the cert is about to expire, then 10, then 1--if you give them your email address. This is a pretty big fail on iX' part.

 

[jgreco]

It takes very little effort; LE will notify you 20 days out that the cert is about to expire, then 10, then 1--if you give them your email address. This is a pretty big fail on iX' part.

Yet I see it happen elsewhere too.

Since I'm a bit more familiar with this than your average webmaster, it was bugging me for a bit that you said this, because it turns out that it looks like it is entirely possible to get a LE cert without having an e-mail address attached to it. This occurred to me because in the environment here, there would need to be a way to seed the email address, and I didn't remember writing one. Hm.

 

[danb35]

it was bugging me for a bit that you said this, because it turns out that it looks like it is entirely possible to get a LE cert without having an e-mail address attached to it.

It's entirely possible. certbot will ask for, but not require, an email address. I think most other clients operate similarly, but certainly not all of them. And, of course, if you don't provide one, you're on your own to monitor your certs. If your client configuration is sensible (as the one in TrueNAS isn't), you should never hear from them except on changes to the subscriber agreement.

 

[jgreco]

certbot blows. That was the first big mistake in the LE system.

 

[danb35]

Yeah, I'm not a fan of that one myself. It works well enough, but there are lot of what seem to be questionable design choices.

 

[trevaaar]

They're not using Let's Encrypt for update-master.ixsystems.com or update.freenas.org any more, the certificate for both is signed by GoDaddy. I guess they must not have set up auto-renewal when they changed.

Based on the validity start date of 19 Oct 2021, I'll bet they changed CAs for compatibility. FreeNAS 11, which uses OpenSSL 1.0.2, has a problem with Let's Encrypt certs since the DST X3 root CA expired at the end of September 2021. There were several posts here and on /r/truenas in October 2021 from people having trouble updating.

 

[jgreco]

the certificate for both is signed by GoDaddy. I guess they must not have set up auto-renewal when they changed.

We need a triple facepalm here.

 

[awasb]

Your wish is my command ...

 

[Samuel Tai]

iX IT support informed me this morning this cert should now be renewed. I was able to access https://update-master.ixsystems.com/TrueNAS/trains.txt without error.

 

[trevaaar]

It's fixed on IPv4, still broken on IPv6.

 

[jgreco]

And here's @awasb 's chance for the quadruple.