SOLVED System Update Not Working

[RDM]

Hello, I was going to make the jump to TrueNas yesterday by switching trains but then discovered that my System Update is not working.

As I have not made any changes to FreeNas (probably a year or more) I am perplexed as to what could have happened.

I can ping the update site from the shell but still get the can't connect. Tried looking on web but only thing I could find was something about the date not being correct but it is.

Anyone have aby ideas?

TIA,

RDM

 

[Tigersharke]

Others will certainly give more exact information or advice but I suspect that since FreeNAS is effectively a legacy product now and may no longer have the same mechanisms in place, the upgrade you are attempting may no longer be possible. There should be an upgrade path or upgrade technique to transition from FreeNAS to some variety of TrueNAS. Someone here may be able to direct you to it easily, or it may be discoverable on these forums, perhaps mentioned in this forum space (a sticky post?) or in the resources section.

I wish you success in your efforts.

 

[RDM]

OK, thanks I appreciate it. I wish I would have done it sooner to avoid this exact situation. I'll see what I can find.

 

[c77dk]

It could be an issue with expired certificate - that your system haven't got the new CA certificates installed and thus can't verify the site. Just a guess, but the timing makes me think in those terms (there were some expirations of letsencrypt certificates the other day)

 

[RDM]

Oh, thats interesting. Is there any way I can get the certs manually?

I was looking for an update file for manual update but cannot seem to locate any.

Thanks...

 

[Toiffel]

I had the same exact issue, ended up downloading Truenas from front page and ran update from usb, worked like a charm....

 

[RDM]

Thanks Toiffel, i found this file TrueNAS-12.0-RELEASE-manual-update.tar, I am going to do a manual from update page.

Is this right? I'm wondering how you did it from the USB?

 

[Toiffel]

Thanks Toiffel, i found this file TrueNAS-12.0-RELEASE-manual-update.tar, I am going to do a manual from update page. Is this right? I'm wondering how you did it from the USB?

I did it like a new install, but used the upgrade option from the main page...hope that makes sense....

 

[RDM]

OK fyi, that did work. And I can get to updates know. However, I still seem to have an issue when I go to apply the new updates. I get a BASE-OS error. I guess I will take a look for that and see what the issue is.

 

[Darknight_DS]

I've also run into the same issue from latest 11.3 to the 12 train. The error is with the SSL cert. Here's what I get when I curl the failing site for the upgrades:

curl https://update-master.ixsystems.com/FreeNAS/trains.txt
curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

No clue how to fix this. So any help would be appreciated. I tried upgrading to the 12 train a week ago but it wouldn't startup properly. I saw the latest 12.0-u6 was available for my other server already on 12.0-U5, so I wanted to try it on this box.

 

[jgreco]

The underlying issue is that OpenSSL packaged with FreeBSD 11 is OpenSSL 1.0.2somethingorother.

The OpenSSL developers discuss this in their LE blog post.

Basically what it comes down to is that client side stuff using 1.0.2 will see the old DST Root X3 certificate and errors out on it, despite modern LE certs also being signed by the still-valid ISRG Root X1 certificate. The workaround seems to be to eradicate the DST Root X3 from your certificate store, since it's expired anyways.

 

[Darknight_DS]

So, how do we fix this on freenas 11.3 then? Is the cert not tied to the OS layer that can't be changed or is it in a folder that allows you to permanently delete it?

 

[captain118]

The underlying issue is that OpenSSL packaged with FreeBSD 11 is OpenSSL 1.0.2somethingorother.

The OpenSSL developers discuss this in their LE blog post.

Basically what it comes down to is that client side stuff using 1.0.2 will see the old DST Root X3 certificate and errors out on it, despite modern LE certs also being signed by the still-valid ISRG Root X1 certificate. The workaround seems to be to eradicate the DST Root X3 from your certificate store, since it's expired anyways.

How do you do this for freenas though. the pki folder doesnt exist.

 

[jgreco]

How do you do this for freenas though. the pki folder doesnt exist.

I don't think I have ever seen a "pki folder". I have designed and implemented several enterprise root CA's and a bunch of other random SSL stuff under my belt, along with automated systems to integrate enterprise CA's into system default lists alongside stuff like the Netscape ca-root-nss.crt list, fully hashed even, and I'll go so far as to say that in my opinion there is not a uniform standard as to whether these even go into a single file, a hashed directory, or where or what specifically that may be, depending on OpenSSL's installation directory and OS defaults.

On FreeNAS, it seems like they're borrowing the ports mechanism, and have what appears to be the ports Netscape NSS list in /usr/local/share/certs/ca-root-nss.crt as a single file. You could try plucking out the offending certificate from there to see if it fixes it. Note that the decoded certificate comes BEFORE the encoded certificate, so you would want to make a backup of the file, then try deleting the decoded bits starting at

Code:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            44:af:b0:80:d6:a3:27:ba:89:30:39:86:2e:f8:40:6b
    Signature Algorithm: sha1WithRSAEncryption
        Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3
        Validity
            Not Before: Sep 30 21:12:19 2000 GMT
            Not After : Sep 30 14:01:15 2021 GMT

all the way through the following

Code:

-----END CERTIFICATE-----

which follows the decoded bits and the machine-readable certificate (about 78 lines). This is not a guarantee that this will work, it's just the first thing I'd try.

 

[fireheadman]

found this thread... having same issue with the certs expired.
its been a while since I have updated my system, it is still on 11.3 U5. I didn't realize FreeNas was retired (or retiring)....

Do we need to update to TrueNas 12.x sooner than later?
If anyone comes up with a solution how to resolve the certs issue, I am all ears!

Thanks

 

[captain118]

I don't think I have ever seen a "pki folder". I have designed and implemented several enterprise root CA's and a bunch of other random SSL stuff under my belt, along with automated systems to integrate enterprise CA's into system default lists alongside stuff like the Netscape ca-root-nss.crt list, fully hashed even, and I'll go so far as to say that in my opinion there is not a uniform standard as to whether these even go into a single file, a hashed directory, or where or what specifically that may be, depending on OpenSSL's installation directory and OS defaults.

On FreeNAS, it seems like they're borrowing the ports mechanism, and have what appears to be the ports Netscape NSS list in /usr/local/share/certs/ca-root-nss.crt as a single file. You could try plucking out the offending certificate from there to see if it fixes it. Note that the decoded certificate comes BEFORE the encoded certificate, so you would want to make a backup of the file, then try deleting the decoded bits starting at

Code:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            44:af:b0:80:d6:a3:27:ba:89:30:39:86:2e:f8:40:6b
    Signature Algorithm: sha1WithRSAEncryption
        Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3
        Validity
            Not Before: Sep 30 21:12:19 2000 GMT
            Not After : Sep 30 14:01:15 2021 GMT

all the way through the following

Code:

-----END CERTIFICATE-----

which follows the decoded bits and the machine-readable certificate (about 78 lines). This is not a guarantee that this will work, it's just the first thing I'd try.

That totally worked. Thank you!

 

[jgreco]

Do we need to update to TrueNas 12.x sooner than later?
If anyone comes up with a solution how to resolve the certs issue, I am all ears!

FreeNAS, the name, is being deprecated in favor of TrueNAS Core (the FreeBSD version) and TrueNAS Scale ("experimental" Linux version). There are some things that can be done on Linux that aren't really available or possible on FreeBSD, alas.

Whether you decide to move forward to TrueNAS right now depends on what you need to accomplish. Some of us are likely to avoid the Linux stuff for awhile.

The edit I quoted above should be a reasonable solution to the LetsEncrypt problem. The ca-root-nss on older FreeNAS systems is probably a bit out of date, and the file that we're using for our installs here is nss-3.63, which has been manually edited to remove the "bad" DST certificate. You can get a copy at

fetch --no-verify-peer https://extranet.www.sol.net/files/misc/ca-root-nss.crt.src

which you then need to compare to your existing certificate list, and, if you approve of the differences, you can then install as /usr/local/share/certs/ca-root-nss.crt on your FreeNAS host. Be aware that SSL is the foundation of trust on the Internet, and that you SHOULD NOT blindly accept my assurances that this is nss-3.63 with a single certificate deleted. You are better off acquiring nss-3.63 yourself and removing the certificate in question yourself.

 

[Diego Queiroz]

If you're here because of the certificate error (as I was), this line of code should solve your problem:

Code:

fetch --no-verify-peer https://extranet.www.sol.net/files/misc/ca-root-nss.crt.src -o /usr/local/share/certs/ca-root-nss.crt

You should follow @jgreco advice of not running this code before assuring you trust on it (you're basically trusting that the file in extranet.www.sol.net domain does not contain fake CAs that will make your box trust in a certificate it should not).

Yes, we're saying you should not trust in us.

But if you don't have an idea what is a CA ROOT file, or if you think this is just something odd that is preventing you from updating from the cool user interface, you're probably ok running this code (even if there's a fake CA in the file, I would say the risk is low, and when you update the box, the file will probably be overwritten). If you're still unsure, run cp /usr/local/share/certs/ca-root-nss.crt ./ca-root-nss.crt.old to make a backup of your current file before running the above command. If it doesn't work, revert changes.

 

[Darknight_DS]

Thanks jgreco! This worked for me as well! Now I need to figure out why the jump from 11.3 stable to 12 stable doesn't work on my server!

 

[jgreco]

Code:

fetch --no-verify-peer https://extranet.www.sol.net/files/misc/ca-root-nss.crt.src -o /usr/local/share/certs/ca-root-nss.crt

You should follow @jgreco advice of not running this code before assuring you trust on it (you're basically trusting that the file in extranet.www.sol.net domain does not contain fake CAs that will make your box trust in a certificate it should not).

Yes, we're saying you should not trust in us.

Which is the reason I wrote what I did the way that I did.... but I thought about talking about diff'ing it, and the problem is that there's some drift between the versions of nss-3.63 and the older ones on the FreeBSD platform. Apparently newer ones are available; nss-3.71 appears to be just a few days old. Unfortunately that's like an 80MB package and does not include the processed file as output. You can download https://ftp.mozilla.org/pub/security/nss/releases/NSS_3_71_RTM/src/nss-3.71.tar.gz and the file tree is at nss-3.71/nss/lib/ckfw/builtins/certdata.txt which needs to be processed through /usr/ports/security/ca_root_nss/files/MAca-bundle.pl.in along with some variable assignments to result in a new file; unfortunately, this still seems to list the DST certificate at the heart of this problem.