Anthony J. Martinez

SSH At Scale With OpenSSH Certificates - Practical Example

As promised in my last post, here is an example setup for how one might use machine-specific data to shape SSH access using OpenSSH Certificates. To avoid too much irriation on my local system, I created a simple test container using podman and the following Dockerfile:

# syntax=docker/dockerfile:1
FROM docker.io/alpine:latest 

RUN apk --no-cache add openssh-server bash && \
    adduser -s /bin/bash -D -u 1001 demo && \
    echo "demo:$(dd if=/dev/urandom bs=1 count=32 2>/dev/null | base64)" | chpasswd -c sha512 && \
    mkdir -p /opt/ssh/config

WORKDIR /opt/ssh

CMD ["/bin/bash", "/opt/ssh/config/run"]

The content of /opt/ssh/config/run is as follows:

#!/bin/bash

set -e

HOST_KEY="/opt/ssh/config/ssh_host_ecdsa_key"
CONF="/opt/ssh/config/sshd_config"

if [ ! -e "${HOST_KEY}" ]; then
   ssh-keygen -t ecdsa -b 256 -N '' -q -f "${HOST_KEY}"
fi

/usr/sbin/sshd -D -e -h "${HOST_KEY}" -f "${CONF}"

The reference sshd_config is:

# Setting some core values that are helpful for use in a
# system using Certificates.

HostKey /opt/ssh/config/ssh_host_ecdsa_key
HostCertificate /opt/ssh/config/ssh_host_ecdsa_key-cert.pub

LoginGraceTime 10s
PermitRootLogin no
StrictModes yes
MaxAuthTries 3
MaxSessions 10

PasswordAuthentication no
PubkeyAuthentication yes

AuthorizedKeysFile	 none

TrustedUserCAKeys /opt/ssh/config/ssh_ca_keys
AuthorizedPrincipalsCommand /opt/ssh/config/auth_principals %u
AuthorizedPrincipalsCommandUser nobody


# override default of no subsystems
Subsystem	sftp	/usr/lib/ssh/sftp-server

The magic happens in auth_principals:

#!/bin/bash

set -e

case "${1}" in
    "demo")
	echo "${HOSTNAME}-demo"
	;;
esac

While auth_principals is a fairly trivial Bash example the key points are that:

  1. AuthorizedPrincipalsCommand needs to return a string matching one of the principals encoded on the presented certificate when a user tries to login with a given username

  2. This can call upon anything the machine knows about itself and can programmatically acces. The use of HOSTNAME is just an example. As an administrator you can do as you like. Be creative!

Running the Example

After building the test container, fire it up:

$ podman run --rm -it --hostname=$(openssl rand -hex 8) -p 9022:22 -v ./config:/opt/ssh/config:Z ssh-cert-example
Server listening on 0.0.0.0 port 22.
Server listening on :: port 22.

...

Note the volume mount of a config directory which itself contains:

  1. The scripts, and configs, shown above: sshd_config, run, and auth_princpals
  2. The HostKey and HostCertificate
  3. A hello script that I will set as the ForceCommand on a sample user certificate
  4. An ssh_ca_keys file referenced in sshd_config with the public key associated with the SSH key used as a CA signing key.

Given that I created the container with a random HOSTNAME value, the container needs a little inspecting before I can proceed:

$ podman inspect strange_heisenberg | jq '.[0].Config.Hostname'
"cac0e8fd0d329d7e"
Minting a Certificate

From the information above we know that a user, demo can access the system by presenting an OpenSSH Certificate with a principal matching ${HOSTNAME}-demo. Given that the HOSTNAME variable expands to cac0e8fd0d329d7e let us sign a certificate accordingly:

$ ssh-keygen \
	-I demo@ajmartinez.com \
	-V $(date +%Y%m%d%H%M%S):$(date --date="+15 minutes" +%Y%m%d%H%M%S) \
	-z $(date +%s) \
	-n cac0e8fd0d329d7e-demo \
	-O force-command=/opt/ssh/config/hello \
	-s ../ssh_ca \
	demo.pub
Signed user key demo-cert.pub: id "demo@ajmartinez.com" serial 1642122125 for cac0e8fd0d329d7e-demo valid from 2022-01-13T19:02:05 to 2022-01-13T19:17:05

Checking the contents:

$ ssh-keygen -Lf demo-cert.pub 
demo-cert.pub:
        Type: ecdsa-sha2-nistp256-cert-v01@openssh.com user certificate
        Public key: ECDSA-CERT SHA256:qfxed1FR8kXtXMXBWTjEjwPLjBKWz0nKbthaFGGVO/E
        Signing CA: ECDSA SHA256:Tb4fK9xMEtZRnxHlXsvXaPoPj1A8vtxNXvWkb1Wpju8 (using ecdsa-sha2-nistp384)
        Key ID: "demo@ajmartinez.com"
        Serial: 1642122125
        Valid: from 2022-01-13T19:02:05 to 2022-01-13T19:17:05
        Principals: 
                cac0e8fd0d329d7e-demo
        Critical Options: 
                force-command /opt/ssh/config/hello
        Extensions: 
                permit-X11-forwarding
                permit-agent-forwarding
                permit-port-forwarding
                permit-pty
                permit-user-rc
Logging In

The easiest part of all, logging in as a client:

$ ssh -i demo -o CertificateFile=demo-cert.pub -p 9022 demo@localhost
Welcome to cac0e8fd0d329d7e! OpenSSH Certificates are cool huh?
Shared connection to localhost closed.

What trickery is this? No prompt to accept a random key fingerprint into my known_hosts?? Surely you jest!? No, I just added the ssh_ca.pub to ~/.ssh/known_hosts as a @cert-authority entry:

@cert-authority [localhost]:9022,[::1]:9022 ecdsa-sha2-nistp384 AAAAE2VjZHNhLXNoYTItbmlzdHAzODQAAAAIbmlzdHAzODQAAABhBMGDesyChnteRlL3/fkcFUQk+qDuL5dnbFPeT8oejuaDOv4UT3yLU/2bXJZlEjbknztORXuy3ViqCBQskqPkfPglyv0Uqpn4VhRbh9j1fK6MzcPg50OWDw1hioCohazx7w==
Checking Server Access

In the world of distributed SSH access without certifiate use, and with an industry worst-practice of shared accounts with shared credentials, no one ever has any clue who logged in as demo. Maybe it was someone authorized to do so. Maybe it was someone who left an organization a decade ago.

The output from my test container, for each access, looks like this:

Accepted publickey for demo from 10.0.2.100 port 53930 ssh2: ECDSA-CERT SHA256:qfxed1FR8kXtXMXBWTjEjwPLjBKWz0nKbthaFGGVO/E ID demo@ajmartinez.com (serial 1642122125) CA ECDSA SHA256:Tb4fK9xMEtZRnxHlXsvXaPoPj1A8vtxNXvWkb1Wpju8
Received disconnect from 10.0.2.100 port 53930:11: disconnected by user
Disconnected from user demo 10.0.2.100 port 53930
Accepted publickey for demo from 10.0.2.100 port 53934 ssh2: ECDSA-CERT SHA256:qfxed1FR8kXtXMXBWTjEjwPLjBKWz0nKbthaFGGVO/E ID demo@ajmartinez.com (serial 1642122125) CA ECDSA SHA256:Tb4fK9xMEtZRnxHlXsvXaPoPj1A8vtxNXvWkb1Wpju8
Received disconnect from 10.0.2.100 port 53934:11: disconnected by user
Disconnected from user demo 10.0.2.100 port 53934

And after waiting for my 15-minute validity period to expire:

Certificate invalid: expired
maximum authentication attempts exceeded for demo from 10.0.2.100 port 53940 ssh2 [preauth]
Disconnecting authenticating user demo 10.0.2.100 port 53940: Too many authentication failures [preauth]

Conclusion

The building blocks for OpenSSH certificate use are simple and accessible to admins of all skill levels. Substantial benefits exist over the use of LDAP, authorized_keys, or shared credentials:

  1. Certificate auth is lightning fast
  2. One need only maintain TrustedUserCAKeys on servers
  3. One need only maintain @cert-authority entries, which can be scoped hostnames, IPs, etc, on client systems
  4. Certificates are portable. If you have a system in an airgapped bunker, one can mint a certificate with a limited validity period attached to an ephemeral private key that will allow access to the system to someone physically present. Try that with LDAP.
  5. It is clear who accessed what and when.

While the examples given were simple and manually executed on a Bash shell, there are a number of ways one could build a highly-available (and secure) web service CA. Python and Rust both have appropriate libraries, and I am certain other languages do as well. With a little imagination, and a lot of attention to detail, you too can have secure SSH access that is both easy to deploy and easy to maintain.

There are a few limitations to be aware of, and I will cover these in another post.