Where does CircleCI get ssh keys from since I don't have add_ssh_keys in config?

Deployments
1

After running tests, we deploy our app with Capistrano. Just today we ran into an issue where that task was failing with:

SSHKit::Runner::ExecuteError: Exception while executing as user@staging: disconnected: Too many authentication failures (2)

In auth logs on the server I can indeed see that it failed

Apr 21 06:48:07 staging sshd[15631]: error: maximum authentication attempts exceeded for user from 54.84.177.237 port 41100 ssh2 [preauth]

Ultimately after re-running the job everything finished successfully (as it used to in the past as well).

When trying to debug the issue, I noticed that the SSH key needed to deploy to host is added to our " Additional SSH Keys" with the correct host. However, our CircleCI config is missing the required add_ssh_keys step.

But it worked all the time without this step, so I restarted a job with SSH access to see what keys are on the CircleCI container. I found that there’s only one key in .ssh/id_rsa which doesn’t match the one added in “Additional SSH Keys”.

So then I ran ssh user@staging -vvv and to my surprise, I was able to log in! I was curious which key CircleCI uses then. This is what I found in ssh -vvv output:

debug1: Next authentication method: publickey
debug1: Offering RSA public key: 
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug1: Offering RSA public key: 
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: pkalg rsa-sha2-512 blen 535
debug2: input_userauth_pk_ok: fp SHA256:(key fingerprint here)
debug3: sign_and_send_pubkey: RSA SHA256:(key fingerprint here)
debug3: send packet: type 50
debug3: receive packet: type 52
debug1: Authentication succeeded (publickey).

So it looks like CircleCI first offered some invalid key, and then another one which fingerprint matches the key that we have under “Additional SSH Keys” in project settings.

However, usually Offering RSA public key: would be followed by the path of the key, but here it’s… empty?

How does CircleCI know which key to use then? How does it work without add_ssh_keys? Should I add add_ssh_keys to my config to prevent issues with going over maximum authentication attempts?

2

I have a similar issue, did you found out any reason?

3

Well, our actual problem here with not being able to ssh from CircleCI to our server was that we had some issues with newlines in our SSH keys. When concatenated into a big file for the allowed keys list for sshd, it sometimes rendered the CircleCI’s key invalid and CircleCI couldn’t ssh into our server.

But that probably isn’t particularly interesting. When it comes to the questions I stated in this thread, I asked CircleCI’s support and this is the response that I got:

If you added the ssh key from CircleCI’s UI it is not stored in .ssh/id_rsa. Your key will be added via the builds ssh-agent . You can confirm available keys and location with ssh-add -l . Your config is working as is because it is automatically signing all added ssh keys. Using the add_ssh_keys key and fingerprint is best practices because it adds a layer of security.

Your config may continue to work. However, we do not recommend this. By not using the add_ssh_keys key and fingerprint you run the risk of failed deploys in the future.

Hope it helps!