Anthony J. Martinez

SSH At Scale with OpenSSH Certificates

The Issue

Service and maintenance of widely deployed Linux based systems can be a challenging task. This often requires distributed global support personnel with varying levels of system access. A means of auditing when and where that access is used should be a strict requirement. In fleets of IoT devices where base configurations are common, but resources are limited, one might find a need to balance system simplicity and complex access models for support teams.

An Ideal Solution with OpenSSH Certificates

OpenSSH Certificates provide a means of gating access to Linux systems with extremely minimal overhead on either the client or server, and are supported in most every version of OpenSSH released in the last decade. If your systems are not severely deprecated this solution can work for you.

Configuration for use of certificates is quite simple, and requires no more than an understanding of a few parameters in sshd_config:

• Setting up TrustedUserCAKeys file with the public part of a Certificate Authority's keypair(s)
• Setting up one of AuthorizedPrincipalsiFile or AuthorizedPrincipalsCommand to map certificate principals to system users
• Setting HostCertificate such that the host returns a host certificate to the client, allowing for far simpler maintenance of client-side known_hosts and aiding in the abolition of dangerous options like StrictHostKeyChecking=no.

Example Flow

• Identity provider signs a token with mapping of user rights to principals for an OpenSSH certificate
• Client exchanges token and an SSH pubkey with a CA where
  • Token claims are verified
  • An OpenSSH certificate is minted from token claims using the supplied pubkey
• Client uses the returned certificate to access the target device(s).

Example IdentityToken

{
	"user_id": "someone@example.org",
	"principals": ["device_id1", "device_id2", "device_idN"],
	"nbf": 1641016800,
	"exp": 1641017100
}

Example Certificate

A number of ways exist by which one might mint an OpenSSH Certificate, ssh-kegen included.

Assuming a CA is running some process that accepts JWTs, validates the signing JWK, and verifies claim fields against some input validation defined by organizational needs, the creation of a certificate for the IdentityToken shown above might look like:

ssh-keygen \
	-I someone@example.org \
	-s ${CA_KEY_PATH} \
	-n device_id1,device_id2,device_idN \
	-z 12345678 \
	-V $(date --date=@1641016800 +%Y%m%d%H%M%S):$(date --date=@1641017100) \
	user_provided_pubkey.pub

Abstracted use case

The resulting user_provided_pubkey-cert.pub from the example above can then be returned to the user who may use the certificate to access systems where:

• The CA key that signed the certificate is trusted
• The certificate is presented within its validity period, and
• The authorized principals mechanisms match
  • The presented certificate data to
  • A local system user

When such access occurs, the authorization logs will show that:

• A pubkey was accepted for the mapped local user
• That pubkey was in fact a certificate with identity someone@example.org
• The serial number was 12345678
• The fingerprint of the signing key is also present, along with that of the certificate

Conclusion

OpenSSH versions from any non-deprecated distribution have supported certificate login for several years. A simple, and robust, solution exists for accessing distributed systems at scale. With some creativity, and a toolbox of open standards, one can provide secure and auditable access to systems over SSH. In a later post, I will share samples showing how one might configure clients and servers for OpenSSH Certificate use.