The pitfalls of using ssh-agent, or how to use an agent safely

In a previous article we talked about how to use ssh keys and an ssh agent.

Unfortunately for you, we promised a follow up to talk about the security implications of using
such an agent. So, here we are.

If you are the impatient kind of reader, here is a a few rules of thumb you should
follow:

  1. Never ever copy your private keys on a computer somebody else has root on.
    If you do, you just shared your keys with that person.

    If you also use that key from that computer (why would you copy it, otherwise?),
    you also shared your passphrase. I generally go further and only keep my private
    keys on my personal laptop, and start all ssh sessions from there.

  2. Never ever run an ssh-agent on a computer somebody else has root on.

    Just as with the keys, I generally don’t run ssh-agents anywhere but my laptop.
    And when I say “has root on”, consider that you are both trusting that person
    to not abuse his privileges, and to do a good job at keeping the system safe,
    up to date, and without other visitors.

  3. Only forward your agent connection to machines you trust.

    As you will see further down in this article, forwarding an agent is equivalent to
    sharing your keys with anyone who managed to get root on that machine. And this
    is not theoretical: getting access to your keys takes at most a few lines of a shell
    script.

  4. Make sure your keys and your agent are unloaded when you log off your machine.

    If you are one of the old school guys that simply starts his agent with something
    like:

      if [ -z "$SSH_AUTH_SOCK" ] ; then
          eval `ssh-agent -s`
          ssh-add
      fi
    

    in his .bashrc, don’t forget that every time you open a terminal you are creating a new agent
    that nobody will ever kill. It will remain happily hanging there forever with all your
    keys ready for anyone to use.

    (the snippet of code is the one suggested on various threads, including vairous
    stackoverflow
    answers)

    To protect my agent forwarding, I personally follow a 5th rule:

  5. Use different keys for different purposes, and keep them in different agents.

    The reason for this rule is a direct consequence of the other rules,
    and is best explained with an example: let’s say in order to connect to the servers at
    work you must use ssh keys. But these servers are on a private network, so you must
    first use agent forwarding and connect to some sort of “gateway”.

    Every time you connect to the gateway with agent forwarding you give the
    ability to anyone on that machine with root to use any and every key loaded in
    your agent.

    If any of those people gains access to any other server at work, well, that’s
    life. Something my employer will need to worry about.
    At the same time, I really don’t want those people to gain access to my home
    server, personal github account, or to the VPS I use to backup my family photos.

    What I do: one key for work, one key for home, one key for backup server,
    “one key per customer”, or “per security domain”. And I do the same for
    agents, as otherwise agent forwarding will expose my keys: one agent for
    work, one for home, and so on.

    If I end up forwarding the agent to a compromised machine, the attacker will
    gain access only to machines within that domain.

    Sounds like a giant pain to manage and use? Not really, if you use something like
    ssh-ident [disclaimer: I’m one
    of the authors].

Now that we have covered the bases, let’s try to cover some of the reasons behind
those recommendations…

Starting the agent

If you do a quick search on how to use an ssh-agent, most pages will tell you to
start an agent by using something like:

$ eval `ssh-agent`

simple and fast, isn’t it? But annoying to do every time you log in. Go back on
google and search “how to automatically start ssh-agent”, and you’ll find many
suggestions to add something like:

if [ -z "$SSH_AUTH_SOCK" ] ; then
  eval `ssh-agent -s`
  ssh-add
fi

to your .bashrc. Problem solved? Not really. Now for every console you open, you
end up with a new agent. So back on google, and after some time you will find
some variation of:

SSH_ENV="$HOME/.ssh/environment"

function start_agent {
    echo "Initialising new SSH agent..."
    (umask 066; /usr/bin/ssh-agent > "${SSH_ENV}")
    . "${SSH_ENV}" > /dev/null
    /usr/bin/ssh-add;
}

# Source SSH settings, if applicable

if [ -f "${SSH_ENV}" ]; then
    . "${SSH_ENV}" > /dev/null
    ps -ef | grep ${SSH_AGENT_PID} | grep ssh-agent$ > /dev/null || {
        start_agent;
    }
else
    start_agent;
fi

(from one of the most voted answers on stackoverflow)

Which in short keeps the details of the agent in a file, tries to load it,
checks if that agent is still running (after a reboot or similar), and if
not, it starts another one.

I personally don’t like grepping for ssh-agent and checking pids, and I
don’t like the fact that the script above may break agent forwarding, as it
does not detect any agent already available.

So I much prefer versions of the script like the one here:

ssh-add -l &>/dev/null
if [ "$?" == 2 ]; then
  test -r ~/.ssh-agent && 
    eval "$(<~/.ssh-agent)" >/dev/null

  ssh-add -l &>/dev/null
  if [ "$?" == 2 ]; then
    (umask 066; ssh-agent > ~/.ssh-agent)
    eval "$(<~/.ssh-agent)" >/dev/null
    ssh-add
  fi
fi

which just queries the agent for available keys. If none can be found, it will try to
load the agent config from a file, and if still can’t connect to the agent, it will
start a new one. This version has the added benefit that if your window manager has
an agent already running, you will use it. Easy peasy, right?

Well, there are a few problems with this approach:

  1. Your agent will run forever! And keep your keys with it.

  2. You have one agent for all your keys, which violates the 5th rule at
    the top of the document.

Let’s try to solve one problem at a time, so let’s try with problem #1 first.

You could:

  1. Specify a maximum key lifetime with the -t parameter. For example,
    -t 3600 will keep your keys in memory for at most one hour.

    But what happens after an hour of inactivity? Well, your key will
    disappear, and the next time you try to use ssh it will simply
    prompt you for your password. That’s right, as the key is gone,
    it doesn’t know there was a key in the first place. It will not
    tell you “look, we need to reload your key” or “ay yo, one of
    your keys has expired, give me your passphrase again, and I’ll
    happily try to reload it”.

    This is generally taxing on my brain, as every time this happened
    to me, I had to reconcile the password prompt with the fact I
    always use the agent, and come up with
    “oh drat! my keys expired, let’s run ssh-add again”.

    Annoying, isn’t it? You can make it simpler with a few tweaks to ~/.ssh/config,
    but it still is pretty annoying.

    What I ended up doing in the past, was well, never use ssh directly:
    instead, use a shell script that would check if my keys were still
    there, and if not, call ssh-add first magically. Complicated? that is another
    thing that ssh-ident
    can do for you.

  2. Kill the ssh-agent when you are done using it. Easy peasy, no?
    You could try using .bash_logout. But if you do, and your shell ‘execs’ another
    command, crashes, your ssh terminal dies, or you use screen or tmux,
    well, it won’t work very reliably, your ssh-agent will not be killed.

    Feeling brave? Maybe you can trap EXIT 'killall ssh-agent' or something
    similar. But this still has many of the same drawbacks.

    The most reliable method I found was the exec support in ssh-agent,
    that by looking around the .net, seems also the least mentioned?

    After ssh-agent you can specify a command to run. That command will be
    started with the rigth environment variables set, and ssh-agent will keep running for as long as that command is alive.

    For example, if I type something like:

      $ exec /usr/bin/ssh-agent /bin/bash
    

    from my shell prompt, I end up in a bash that is setup correctly with
    the agent. As soon as that bash dies, or any process that replaced
    bash with exec dies, the agent exits. Simple enough, I could add it
    to my .bashrc, no? Watch out for loops, and well, you’ll be disappointed
    to find out that in each and every terminal, you will end up with a
    different ssh-agent, needing to run ssh-add every time.

    If you use a graphical interface, you can probably use this approach
    to load your window manager, so all your terminals will have an agent,
    which leads us straight into the 3rd approach…

  3. The third method is to just rely on your distribution.

    Given how many people are using ssh-agent today, many distributions just
    start your window manager with ssh-agent or some equivalent above.

    That way, you have a nice ssh-agent tied to your session, which is killed
    when you log off. Some distributions even use dbus to start and manage
    an agent, which I have not dug into yet.

    This has worked on and off for me as I upgraded laptops, changed window
    managers, login managers, and various versions or the graphical interfaces
    I use felt entitled to replace ssh-agent with something else for the sake
    of annoying their users.

To this day, the method I found most reliable and comfortable with is to 1) wrap my ssh around a
script, that 2) load agents keys as necessary, and 3) expires them after a certain timeout.

Having fun with an agent

Now that we have determined that running and killing an agent is not as easy
as it might seem, let’s look at what someone can do with root access on
a machine running your agent.

First, he may try to get your keys out of it. This is not as hard as it seems,
you can find many tutorials online on how to do it.

It boils down to dumping the memory of the ssh-agent, and looking for the keys in memory.

Second, he may try to just use your agent. This literally requires no skill or
tool whatsover. Let me give you an example, let’s start by loading an agent, a key, and
verifying it works:

$ eval `ssh-agent`
$ ssh-add ~/.ssh/my-private-key
Enter passphrase:
$ ssh-add -l
4096 0a:3c:c9:f7:d0:7a:6d:d2:c0:13:c6:0f:15:12:39:1d my-private-key

Now let’s act as another evil user who has access as root to the machine:

$ su
# ssh-add -l
Could not open a connection to your authentication agent.
# ps aux |grep bash
...
myself 32684  0.0  0.0  18028  2008 pts/5    S    17:17   0:00 /bin/bash
...
# . <(cat /proc/32684/environ |xargs -0 -i echo {} |grep SSH)
# ssh-add -l
4096 0a:3c:c9:f7:d0:7a:6d:d2:c0:13:c6:0f:15:12:39:1d my-private-key

All the attacker had to do was find the PID of one or my processes, import the
right environment variables, and well, profit! The magic was a single line of shell.

Having fun with forwarding

Turns out that the same exact trick used above works with agent forwarding: find a process
your victim is running, look at his environment, and well, configure yours
to use his agent forwarding socket. Total time to use your keys: < 1 minute.

The only improvement here is that the attacker can’t steal your keys. Also, he
can only authenticate for as long as you are logged in, both of which sound
like a win. But is this such an improvement?

Keep in mind that the attacker can write a 2 line shell script to, for
example, scan all the hosts nearby with nmap, and automatically run ssh-copy-id
to install his keys on your machine while you are logged in.

Or keep watching what you connect to, and install his key on every such
host. Hard? Not really:

while :; do servers=`pgrep -u victim -a ssh |sed -ne 's/.*ssh //p'`; 
    test -z "$servers" && { sleep 1; continue; }; 
    ssh-copy-id -i ~/.my-evil-key.pub $servers; done;

will basically intercept any ssh command you run, and install the attacker’s
keys on your remote server.

In short: even a few minutes of access to your agent will enable an attacker
to do a lot of damage, escalate the number of machines it has access to, and
install backdoors to access your system at the most convenient times.

Too many keys, github, and friends

There is one more problem with the naive approach to ssh-agents. Let’s say you go
the route of having at least one key per customer, or per “security domain”, but still use a single agent.

One thing to keep in mind is that when you try to login into a remote host, ssh
will try authentication with all the keys you have loaded, one at a time, one after
the other.

This works ok for as long as you have a few keys. As soon as you start having
many keys, with many being like more than 5, the remote server will kick you out
even before you are able to prove your identity.

That’s right: most ssh servers allow a maximum number of authentication
attempts before killing your connection. Each key you have loaded counts as an attempt,
and if you have more than a handful of keys, you will never be able to use your last ones.

Sites like github.com or gitorious also use your key to verify your identity.
If you have a work account and home account, for example, you will always submit
patches or login as the first key you have loaded in your agent, fancy, not?

Conclusions

I probably sound like a broken record by now, but something like
ssh-ident allows you to keep
different keys in different agents, easily, while loading agents and keys on
demand, keep your identities separated, and easily set a timeout while reloading all
keys as necessary.

It is not for everyone to use, but it has served me well so far, and
addresses most of the issues discussed in this document with no effort on
your side.