Where does CircleCI get ssh keys from since I don't have add_ssh_keys in config?

After running tests, we deploy our app with Capistrano. Just today we ran into an issue where that task was failing with:

SSHKit::Runner::ExecuteError: Exception while executing as user@staging: disconnected: Too many authentication failures (2)

In auth logs on the server I can indeed see that it failed

Apr 21 06:48:07 staging sshd[15631]: error: maximum authentication attempts exceeded for user from 54.84.177.237 port 41100 ssh2 [preauth]

Ultimately after re-running the job everything finished successfully (as it used to in the past as well).

When trying to debug the issue, I noticed that the SSH key needed to deploy to host is added to our " Additional SSH Keys" with the correct host. However, our CircleCI config is missing the required add_ssh_keys step.

But it worked all the time without this step, so I restarted a job with SSH access to see what keys are on the CircleCI container. I found that there’s only one key in .ssh/id_rsa which doesn’t match the one added in “Additional SSH Keys”.

So then I ran ssh user@staging -vvv and to my surprise, I was able to log in! I was curious which key CircleCI uses then. This is what I found in ssh -vvv output:

debug1: Next authentication method: publickey
debug1: Offering RSA public key: 
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug1: Offering RSA public key: 
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: pkalg rsa-sha2-512 blen 535
debug2: input_userauth_pk_ok: fp SHA256:(key fingerprint here)
debug3: sign_and_send_pubkey: RSA SHA256:(key fingerprint here)
debug3: send packet: type 50
debug3: receive packet: type 52
debug1: Authentication succeeded (publickey).

So it looks like CircleCI first offered some invalid key, and then another one which fingerprint matches the key that we have under “Additional SSH Keys” in project settings.

However, usually Offering RSA public key: would be followed by the path of the key, but here it’s… empty?

How does CircleCI know which key to use then? How does it work without add_ssh_keys? Should I add add_ssh_keys to my config to prevent issues with going over maximum authentication attempts?