Recently we got contacted by a customer, and started the typical remote access setup: ask them to open ssh port for our bastion server, create a new user for us, and add the corresponding ssh public key. Well, they promptly responded, and we tested access, got into their bastion server and then tried to get to one of the broken servers but, it was asking for a password. We just asked the customer for the password, and got a reply stating that they had assumed we were using agent forwarding.

The first instinct of the engineer handling the ticket was: well, lets just start ssh-agent, load the key, and forward it! Of course, after a brief discussion, it became clear it was a bad idea.

It is well known that agent forwarding (forwarding of the ssh authentication agent connection) can be considered, under certain conditions, a security risk, because when you do that a socket is created on the remote host, and any user with access to the socket would be able to authenticate using the keys loaded in the agent.

Some of you could still ask: what is the risk?, after all, the agent connection gets forwarded over ssh, the socket permissions are very restrictive, and the agent will not expose the keys, not even encrypted, it will only answer to authentication requests. Furthermore, when you read ssh-agent man page, it sounds reassuring:

(...)"The idea is that the agent is run in the user's local PC, laptop, or terminal. Authentication data need not be stored on any other machine, and authentication passphrases never go over the network. However, the connection to the agent is forwarded over SSH remote logins, and the user can thus use the privileges given by the identities anywhere in the network in a secure way."(...)

No wonder a lot of people trust ssh-agent so much.

But this is only partially true. The reality is that any user with enough privileges (root-level access, or access to the same user you are using to login) on the remote host will be able to use your forwarded agent through the local socket, and thus authenticate using the keys you loaded into the agent, potentially gaining access to other servers you manage, for as long as you keep your session open. It is worth noting that the access is limited to *using* the keys, but not fetching them, ie, the agent doesn't allow access to the private keys, but only answers authentication requests.

Because of this, it is not a good idea to forward the agent to a server you don't trust (either because you don't know and/or don't trust the other administrators, or because you can't tell for sure if the server is compromised). If in doubt, don't do it, if you really have to do it, limit the keys you add to the agent.

In general, assume that every key you add to the agent will be used by someone or something (bot) on the remote server, ask yourself if you know who will use the key and if you are okay with that.

Now, back to our customer situation. Not having a lot of time to argue or consider alternatives (customer reported an emergency), we just decided to create a new key for this customer, and use what they wanted: agent forwarding. This worked just fine, and allowed us to get into all the failing servers comfortably, and without any risks.

The first thing we had to do was create the key, something like this:

ssh-keygen -t rsa -f ~/agent_keys/new_key-id_rsa

Running ssh-keygen, and then typing in a new key destination would work too, but if you hit enter, you could unwillingly overwrite your default id_rsa file.

Now that we generated the new key, to use it with the agent we need to:

1. Start SSH agent (we don't run it by default), and have the agent run a shell.

ssh-agent bash

In my opinion, this is simpler than starting the agent and then exporting the variables, and has the advantage that the agent will automatically quit when you exit the shell, no need to kill the agent, and no risk that you forget to kill it.

2. Add the key(s) you need to use (keys will be usable by anyone with access to the agent's socket, either on the local or remote server):

ssh-add ~/agent_keys/new_key-id_rsa

3. Connect to the remote host, enabling agent forwarding, likely using the -A parameter:

ssh -A remote_host

Of course, if you modified ~/.ssh/config file to include "ForwardAgent yes", the -A parameter would be unnecessary. Remember to never set ForwardAgent to "yes" globally, unless you really know what you are doing.

4. When you are done, just exit the shell, and agent will automatically quit.

Most problems have different solutions, and this one is not an exception. We decided to use this solution, because it was aligned with what the customer expected, and was fast to implement.

I hope this little post helps increase the awareness of the associated risks of doing agent forwarding and that, in the end, helps sysadmins make informed decisions with security in mind.

Until next time!