SOLVED Calling on Go developers...

[Basil Hendroff]

Up until quite recently, the FreeBSD trust store has been in disarray. @jgreco is facilitating a discussion on the impact of this in the thread SSL certificate problem: certificate has expired -- the OpenSSL 1.0.2 vs LetsEncrypt issue. The FreeBSD bug report #160387 - security/ca_root_nss: Allow user to trust extra local certificates shows how pervasive the issue is. That report was open for nine years and the outcome was unsatisfactory.

A consequence of the lack of a unified trust store is that application developers who rely on the trust store have shied away from fully embracing the FreeBSD platform for their application, that is until now. Under FreeBSD 12.2, certctl is the new certificate manager for FreeBSD. It makes it straightforward to add additional root certificates to a unified trust store.

Web public key infrastructure (PKI) is well understood; internal PKI is gaining traction. In the latter case, adding the root certificate of a local certificate authority (CA) to the trust store is an essential requirement. Solutions to date use a platform other than FreeBSD to work around the problem. For example, @danb35 has built a local CA using a Raspberry Pi running the Debian-based Pi OS. Refer to the thread TLS certificates from a local certificate authority for details.

I've been liaising with Caddy and SmallStep developers over the last six months or so on the FreeBSD trust store issue. They are now at a point they can do something about it, at least for FreeBSD 12.2 and newer versions (read TrueNAS, not FreeNAS). I've come up with the high-level pseudocode that's required, however, being more of a troubleshooter than a developer, I'm unable to go the last mile with them. Are there any Go developers with FreeBSD experience in the TrueNAS community or within iXsystems who can contribute some of their time to help wrap this up? If you can, please head over to the SmallStep issue thread Not compatible with FreeBSD and use your expertise to bring some closure to this issue. Caddy utilises SmallStep libraries so the benefits of a solution here will eventually flow on to Caddy.

 

[jgreco]

Its worse than all that.

certctl sucks because it wasn't written to be portable; it's dependent on (at least) a sysctl to determine the path, which was only added in FreeBSD 12.2R.

More frightening is that we have apparently raised a generation of webtards who think that the web browser is the only use for a trust store, and that every use of SSL must come from Google's, Mozilla's, etc., preferred list of trust. Consider the idiots who wrote the paper referred to by

https://www.theregister.com/2021/11/19/web_trust_certificates/

which pays mild lip service to valid uses of private PKI, but generally tries to paint a terrifying picture of "Rusted Anchors", which is just scaremongering sensationalist titling, or that "41.4 per cent of hidden root CAs owned by government agencies and enterprises are used for direct signing of certificates in the chain", which indicates profound ignorance of how deep inspection web filtering works, and a whole bunch of other idiotic claptrap, mixed in with some truly valid concerns. This is not helpful to the community. Ignorant people should not be authoring papers like this.

SSL is necessary for security on the Web, and it is important that we not have every end user needing to figure out what root CA's are credible, or, worse, which website certificates are valid. It absolutely makes sense to have this information seeded into web browsers. However, this is not a high bar for trust. By allowing dozens of CA companies to define what we believe to be "trusted" sites, any one of them serves as a potential vulnerability, and certificates have been misissued in the past by a bunch of them. This probably isn't catastrophic in most cases, but it is worth remembering that SSL security isn't just the difficulty of breaking 4096-bit RSA, but also whether or not the scaffolding can be subverted. And we know the scaffolding is vulnerable, especially with so many players, and with many governments around the world having interest in this.

However, even within Web security, private roots are fine. They are commonly used for content inspection, which is controversial to some people, but enterprise and government sites often rely on it as part of a defense in depth strategy. Some of us also use private CA's for our own internal sites, because it's not that hard to do, and sometimes we have needs beyond just protecting the website connection, such as client certificates.

Moving on, there's the need for things like VPN certificates, where you ABSOLUTELY DO NOT WANT any other CA but the VPN's to be valid, or other authentication services such as LDAP, or data services such as SQL, which should probably never be issued public CA SSL certificates. A proper design for these services includes both client- and server-side certificates, and when deployed properly, with a private CA, results in a substantially greater level of security than could be obtained with public SSL. Private roots are much better at providing SSL security, but only when you are using them on both endpoints, and when properly distributed and installed.

So, after that monster tangent, we come around to my point, which is that I'm not really certain of the wisdom of trying to maintain a systemwide trust store, because it feels very much to me like there should really be carefully scoped trust stores for many applications. The Web CA's do provide a baseline, but are mostly protecting data in transit (especially now with LetsEncrypt) rather than validating the identity of remote endpoints.

Solutions to date use a platform other than FreeBSD to work around the problem.

That's not really true. All you really need is OpenSSL and some platform-agnostic scripting to manually generate certificates, and this is commonly done for five or ten year certificates, which reduces some of the SSL drama with installing certificates on infrastructure devices like switches or IPMI interfaces. There are also engines that can implement the ACME protocol for private CA's, which is handy if you're only dealing with UNIX boxes/websites/other LE'able stuff.

 

[Basil Hendroff]

@jgreco no doubt you're right and I need to qualify.

I'm not really certain of the wisdom of trying to maintain a systemwide trust store, because it feels very much to me like there should really be carefully scoped trust stores for many applications.

Some applications such a Caddy depend on the root cert being in the local trust store. That's optional for other programs and applications such as curl and SmallStep, which allow you to explicitly specify the location of the root cert using an option e.g. curl --cacert if the root cert is not already in the local trust store. From a purely pragmatic point of view, having the root cert in the local trust store levels the playing field. The beauty of running applications such as Caddy and SmallStep in FreeBSD jails is that the trust arrangement is further limited to the local trust store within the jail and not the TrueNAS systemwide trust store.

That's not really true. All you really need is OpenSSL and some platform-agnostic scripting to manually generate certificates, and this is commonly done for five or ten year certificates, which reduces some of the SSL drama with installing certificates on infrastructure devices like switches or IPMI interfaces. There are also engines that can implement the ACME protocol for private CA's, which is handy if you're only dealing with UNIX boxes/websites/other LE'able stuff.

Yes, you're correct. I should have qualified by stating that I'm looking at this from the position of certificate lifecycle management. OpenSSL is super powerful, open to abuse in this respect, but still necessary in some edge cases as you've pointed out. Certificate management in the enterprise is complicated and messy. The ACME protocol having taken web PKI by storm is sorely needed on private PKI, but is very much a work-in-progress for now.

 

[Basil Hendroff]

A HUGE thank you to @qiu3344 from the FreeBSD community for responding to the plea for a Go developer to help resolve the FreeBSD trust store issue within SmallStep. @qiu3344 stepped up to the plate and unselfishly contributed their expertise and time to bring closure to this issue, Around 100 lines of Go code were added in a pull request. The fix should find its way in a future release of SmallStep and Caddy. Anybody who uses these products on FreeBSD, and its derivatives such as TrueNAS, for private PKI certificate lifecycle management, owes @qiu3344 a debt of gratitude for their contribution in making these products fully functional on more recent FreeBSD versions.