Notes on setting up SSH certificates for hosts and users

[danb35]

I don't recall who it was here who initially posted the link, but someone, a while back (edit: it was @rvassar, here: https://www.truenas.com/community/threads/handling-self-signed-certs-in-lan.90823/post-630195), linked to a post on the smallstep.com blog about building your own local CA:

Build a Tiny Certificate Authority For Your Homelab

Let's make a tiny, standalone CA! We'll use a Raspberry Pi 4, YubiKey 5 NFC, and Infinite Noise TRNG.

smallstep.com

It's pretty convenient--it will issue certs for your local network, and your devices only need to trust its root CA cert. It acts as an ACME server, so pretty much any software that can get a cert from Let's Encrypt can also get a cert from the local server. Rather than using DNS validation to obtain Let's Encrypt certs for my LAN devices (including my TrueNAS server), I'm now getting them from the local CA. I know @Constantin is also using one of these.

That's neat enough, but while browsing around their site, I saw quite a bit of discussion of SSH certificates--which I, in my hitherto sheltered life, wasn't aware of, but it sounds like they address lots of issues--and I won't deny that the idea of getting a SSH cert using SSO sounded pretty intriguing. The latter idea sent me down the rabbit trail of setting up LemonLDAP::NG to act as an identity provider on my network, and while I don't quite have that part working as I'd like to, I've managed to get SSH host certs working on many of my LAN systems, and have user certs working in theory. Because I thought the whole process was a little less than intuitive, I wrote up some notes on the process in hopes they'll be helpful to others:

Edit to update the link:

SSH Certificate Configuration

Configuration to use SSH host and client certificates

wiki.familybrown.org

 

[Constantin]

I agree that SSH certificates are a much better way to scale SSH deployment than SSH by itself. Managing the dispersal of SSH info all over the place is no easy task for IT departments. In home use, this issue is not as dramatic but even so, SSH certifications can be a good way to mitigate the impacts of attacks or prevent them altogether.

For example, how many home users wipe their computers of all info before sending them in to be serviced at a depot? Especially when the IT department cannot handle the preparation of the laptop prior to servicing, SSH certs allow whatever SSH info was on the laptop to be invalidated before it even gets to the depot. Rotating the keys regularly also reduces the potential benefits to adversaries of past intercepts.

How much of a benefit SSH certifications are at a small scale like a home likely falls into the "best practices" category, just like encrypted SMB3, for example. In a larger organization where SSH is used regularly by staff to maintain servers, switches, and the like, SSH certificates seems like a pretty good fit to prevent IT mayhem.

 

[danb35]

Rotating the keys regularly also reduces the potential benefits to adversaries of past intercepts.

And it seems the Smallstep folks take short certificate lifetimes pretty seriously. You can change the defaults, of course, but if you're using their system with the default lifetimes, the depot wouldn't have any access to anything even if they wanted it.

But as I've been issuing SSH host certs, my known_hosts file has shrunk dramatically. There are still some entries for systems that aren't on my LAN and therefore wouldn't be able to get certs from my CA (as well as for some local resources that can't run the step software), but I've dropped about 120 lines from that file. It's just a pity there isn't a way to automate deployment of this feature across my various existing hosts.

 

[danb35]

while I don't quite have that part working as I'd like to

Now, with a few tweaks, I do. Smallstep posted how to set up SSO for SSH using Google:

DIY Single Sign-On for SSH

Let's set up Google SSO for SSH! We’ll use OpenID Connect (OIDC), SSH certificates, a clever SSH configuration tweak, and Smallstep’s open source packages.

smallstep.com

I've made a few adjustments to use LemonLDAP::NG as the OpenID Connect authenticator, so you can self-host this entire workflow. I'm running it under Linux; I expect it could be done in a jail as well, but that isn't how I went with it:

Install LemonLDAP::NG SSO/IAM on Nethserver

Content moved to the wiki (except the images, which I need to move over at a later time) Still in Nextcloud, go to User management: Find any users you want to have admin privileges in Nextcloud, and add them to the admin group. Make sure you add at least one. It will look like this:

community.nethserver.org

 

[Constantin]

It is, though one wrinkle was exposed to me this weekend when my CA went down. It’s likely my fault, may have to do with the NTP servers I recently stood up, etc. But the CA can become a single point failure in the home network. Not sure what the solution is… perhaps a failover CA?

I reckon they exist and have a load balancer or like manager up front noticing what parts of the infrastructure are ok, failing, etc and assigning hosts accordingly.

 

[danb35]

If not a failover CA, I wonder if it could be effectively monitored, so at least something like Zabbix could alert you if it went down. Not sure how you'd go about that though.