Jump to content

Yubikey-SSH-FIDO

From Wikitech

This page describes how to generate FIDO/U2F authenticator-backed SSH keys, and use them for production access to WMF systems.

These keys, for instance of type ed25519-sk, are supported in OpenSSH version 8.3 or newer. This is an alternate and more modern way to add security-key based 2FA for SSH, which users may prefer to the OpenPGP method described on the Yubikey-SSH Wikitech page.

Prerequisites

  • You should already have a FIDO-compatible security key, such as a YubiKey 5 series.
  • You should set a PIN for the FIDO application on your security key
    • For YubiKey's this can be done under 'Passkeys' in the Yubico Authenticator app
    • Alternately it can be done with the ykman command-line tool.

Limitations

This method does not work on Buster hosts, so we need to keep regular key types for those. Additionally it is not currently supported/tested for SSH to our network devices, so we need to keep our regular key types for those also.

Recommendations

  • Having multiple physical security keys is recommended, so you don't get locked out if one is lost or damaged.
  • A 6-8 digit PIN is probably sufficient, as your Yubikey will permanently lock itself after 8 failed attempts.
  • Users might want to familiarise themselves with how the technology works, some links are:

Considerations

Passphrase

The ssh-keygen man page describes the new keys as follows:

FIDO keys consist of two parts: a key handle part stored in the private key file on disk, and a per-device private key that is unique to each FIDO authenticator and that cannot be exported from the authenticator hardware. These are combined by the hardware at authentication time to derive the real key that is used to sign authentication challenges.

In other words when you generate ed25519-sk type keys you still end up with two files on your local computer. One of these is the public key which can be added to 'authorized-keys' on remote systems as before. The other looks like a private key, but in fact is just the "key handle" as described above.

As normal, OpenSSH prompts you to add a passphrase to encrypt this "private" key when generating the keypair. It's unclear how much security this adds, however, as the key handle it encrypts cannot be used to authenticate on its own. As with regular keypairs if a passphrase is used the local ssh agent can cache them, so you only need to enter the passphrase once per session.

Require PIN

ssh-keygen can be run with the "verify-required" option, which will require the user to enter their FIDO PIN every time they connect to a remote host. This adds a layer of security as someone needs to know the PIN - in addition to the SSH key handle file and the physical Yubikey - to log on.

While this does add an additional layer, it also adds friction for the user. As such the instructions here do not include it, please add "-O verify-required" to the ssh-keygen commands if you wish to enable this.

Resident Keys

The SSH keys generated by ssh-keygen can optionally be made "resident" on the security key. In this mode the ssh "key handle" file will be stored on the Yubikey itself, and can retrieved from it if needed (provided the user knows the FIDO PIN for the device). This can help export the key handle to multiple computers, but that is possible in other ways and it uses storage space on the Yubikey otherwise not required.

This guide does not use resident keys, so if a user loses the generated key handle file from their ~/.ssh directory they will not be able to authenticate with that key, even if they still have the Yubikey. Users can add the "-O resident" flag when running ssh-keygen if they'd prefer resident keys.

Instructions

NOTE: Tested on Linux with OpenSSH 8.9. Not verified on any other operating systems.

1. Generate new SSH key pair(s)

Connect your Yubikey to your computer and execute the following to create a new keypair: 

   ssh-keygen -t ed25519-sk

You will be prompted to touch the security key and it should flash, please do so.

You will then be prompted for the filename to use. It is best to use a name you can associate with the physical device, especially if you have multiple Yubikeys. For example "yubi_black" and "yubi_colour" so you can tell them apart.

Next you will be prompted for a passphrase, add one if desired.

If you have multiple Yubikeys repeat the above steps multiple times, generating a keypair for each Yubikey. Make sure only one Yubikey is connected to the computer at any time.

2. Add the public keys to puppet repo for production access

The public key(s) generated in the first step can now be added to the authorized_keys file on any remote system you wish to SSH to. For production WMF systems the way to do this is via the modules/admin/data/data.yaml file in the puppet repo.

You can for instance add multiple keys like so: 

 cmooney:
   ssh_keys:
     - sk-ssh-ed25519@openssh.com AAAAGnNrLXNzaC1lZDI1NTE5QAAABHNzaDo= cmooney_yubi_black@wikilap
     - sk-ssh-ed25519@openssh.com AAAAGnNrLXNzaKetugvEfTZ+AAAABHNzaDo= cmooney_yubi_colour@wikilap
   buster_ssh_keys:
     - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKdVgr27gPI/YI57+swSnl0DRgt cmooney@wikilap

If adding multiple keys it makes sense to have a description after each that helps identify it. That makes it simpler to remove one of them in the future if a particular key is lost/damaged.

Also note the buster_ssh_keys list. Users can leave their existing non-FIDO SSH keys here. Puppet will in that case put the legacy key on our few remaining Buster-based systems, which are too old to support the FIDO key types.

3. Modify your SSH config file to use the new keys

The last thing to do is adjust your local SSH config to use the new keys when connecting to production systems.

All you need to do is change the "IdentityFile" directive for production systems, updating the path to point to the new key handle file you created. If you generated multiple keys you can add them all by using several 'IdentityFile' statements, each on a separate line.

The two-press problem

Our standard SSH config means two SSH connections are opened to reach any production host:

  1. A connection to the configured bastion host over the internet
  2. A second connection through that bastion session to the destination server

A drawback of moving to security keys with the above setup is that every ssh login requires two presses of the Yubikey, one for each of these connections. These presses are annoying as the user isn't prompted separately for each, and the timing of when to press is not clear.

Using the bastion as SOCKS5 proxy

A way around this is to use the OpenSSH "-D" (DynamicForward) option when connecting to one of our bastions. This causes OpenSSH to run a local Socks5 proxy on your machine, and to tunnel all connections made to the proxy via the remote bastion host. In this mode we:

  • SSH directly from our machine to a chosen bastion host, with -D option
    • Requires Yubikey touch to authenticate to the bastion
  • Piggy-back on this existing connection to establish subsequent connections to production hosts
    • Requires Yubikey touch to authenticate to the destination host

In other words we only make the SSH connection to the bastion once. Once it is open we can SSH to other production hosts by tunnelling over the initial connection, which saves us from authenticating to the bastion every time.

A very basic example of how one might configure this is shown below. Note the DynamicForward option for the bastion connection, and the ProxyCommand configuration for the remaining hosts. Users need to have the 'netcat-openbsd' package (or equivalent for their OS) installed for it to work: 

   ## Bastion host configured with Socks5 proxy on localhost:10800 and TCP keepalives
   Host bast
       HostName bast3007.wikimedia.org    ### change this to closest one to you
       User YOURUSERNAME
       IdentitiesOnly yes
       IdentityFile ~/.ssh/yubi_colour
       IdentityFile ~/.ssh/yubi_black
       ProxyCommand none
       TCPKeepAlive yes
       ServerAliveInterval 20
       ServerAliveCountMax 5
       DynamicForward 10800
       GatewayPorts no
   
   # Proxy production connections through the Socks5 to bastion using Netcat
   Host *.wmnet *.wikimedia.org !gerrit.wikimedia.org !bast*.wikimedia.org !gitlab.wikimedia.org
       User YOURUSERNAME
       IdentitiesOnly yes
       IdentityFile ~/.ssh/yubi_colour
       IdentityFile ~/.ssh/yubi_black
       ProxyCommand /usr/bin/nc -X 5 -x localhost:10800 %h %p


Users should in general follow the normal instructions and/or keep the other elements in their ssh config file the same.

Downsides to this approach

The main downside of this approach is that it means all your SSH connections are being tunnelled over the same TCP/SSH connection to a single bastion. That means if your SSH connection to the bastion dies or stalls then all the other sessions tunnelling through it will die, unlike if you have a separate connection to the bastion for each one.

The TCPKeepAlive and ServerAlive options for the bastion connection in the above example are there to help keep the bastion session open.

PyEz/Homer issue

For some reason after making these changes I had problems using Homer directly from my laptop. This should not affect regular Homer users who run it from a cumin host, but netops occasionally have to run it locally when modifying the code.

The issue is somehow to do with how PyEz, the Juniper library we use (that in turn runs ncclient and paramiko) to connect to routers. For some reason it wasn't using the local ssh agent, and failing as a result. I detailed in brief what was happening on the below github issue, and for now have a way around it by hacking the PyEz code to always use the agent:

https://github.com/Juniper/py-junos-eznc/issues/1353#issuecomment-2672514682

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Yubikey-SSH-FIDO&oldid=2287391"