SSH connection from CircleCI build server to remote server not being allowed

Deployments
1

I have had no problems making an SSH connection out from a CircleCI server to a remote Linux server. I use this for deployment, and now I’ve come to add another server to the “SSH Permissions” page of the settings but no connections can be made to the new server.

Maybe the way I am generating SSH keys is not compatible with CircleCI? I’m using CircleCI 2.0.

I’ve started from scratch again to make a note of all commands that are performed. Transcript as follows:

ssh-keygen -t rsa -b 4096 -C "deploy@mbmtest"
Generating public/private rsa key pair.
Enter file in which to save the key (/home/deploy/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/deploy/.ssh/id_rsa.
Your public key has been saved in /home/deploy/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:Pf2VpI0+KsAF/pOt1vZOfZ3nV/8hN0XlrzX3NJwoWpM deploy@mbmtest
The key's randomart image is:
+---[RSA 4096]----+
|                .|
|       .       ..|
|      . .      .o|
|       . o .. B.+|
|      . S =E.+ BB|
|       o +oo+..=X|
|        ..+  =o*B|
|         + oo +.B|
|        . o.oo  =|
+----[SHA256]-----+
deploy@mbmtest:~$ cat ~/.ssh/id_rsa.pub > ~/.ssh/authorized_keys
deploy@mbmtest:~$ ssh-keygen -E md5 -lf ~/.ssh/id_rsa
4096 MD5:18:23:7d:6a:71:52:e9:28:a8:fc:38:90:cc:c7:8b:a8 deploy@mbmtest (RSA)

Most importantly, the fingerprint:

image

The public key was entered into authorized_keys, and the private key was added to CircleCI:

image
image.png717×377 26.3 KB

You can see the same fingerprint on the relevant SSH key:

image
image.png1506×202 24.2 KB

Now within the CircleCI build server, a connection is attempted:

circleci@c4e89037b178:~$ ssh deploy@mbmtest.srv.example.com
Permission denied (publickey).

No matter what I try, Circle always gets permission denied error when trying to connect to the mbmtest server. However, it can happily make a connection to the other two hosts, mike-brewer and test-env.

Any ideas as to what I’m doing wrong are appreciated.

Edit: Tried using RSA key of 2048 length instead - no difference.
Edit: SSH keys are NOT encrypted with a passphrase.
Edit: Since taking the screenshots I have generated new keys - so any future references of fingerprints will be different from the above screenshots.

2

I responded to your ticket, adding here for others:

Can you please try adding the add_ssh_keys step? https://circleci.com/docs/2.0/configuration-reference/#add_ssh_keys

3

Tip: when you have SSH permission denied issues, use -v with one, two or three v flags - this will give you output to see what is happening. More flags means more verbosity.

1 Like
4

Thanks @drazisil - I thought it would be beneficial to leave the thread here for others that might be having the same issue.

The SSH keys I’m using are not encrypted by a passphrase.

I’ve just added the following to my config.yml:

    steps:
      - add_ssh_keys:
          fingerprints:
            - "28:13:aa:41:8a:f5:8a:aa:46:b2:2b:a7:bf:31:ca:3c"

The permission denied issue is still occurring.

Thanks @halfer for your advice. I’ve restarted the circle build with SSH so I can perform the -vvv flag manually, and this is the result:

circleci@dc08e6921f47:~$ ssh -o StrictHostKeyChecking=no deploy@mbmtest.srv.example.com
Permission denied (publickey).
circleci@dc08e6921f47:~$ ssh -vvv -o StrictHostKeyChecking=no deploy@mbmtest.sr v.example.com
OpenSSH_7.4p1 Debian-10+deb9u3, OpenSSL 1.0.2l  25 May 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolving "mbmtest.srv.example.com" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to mbmtest.srv.example.com [54.76.124.236] port 22. debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /home/circleci/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/circleci/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/circleci/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/circleci/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/circleci/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/circleci/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/circleci/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/circleci/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4p1 Debian-10+deb9u3
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.4
debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.4 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to mbmtest.srv.example.com:22 as 'deploy'
debug3: hostkeys_foreach: reading file "/home/circleci/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/circleci/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from mbmtest.srv.example.com
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:eSnFc5i2Jb1i0/kZ6/9f+L3IvMREqnO/OoCyjFOL+ac
debug3: hostkeys_foreach: reading file "/home/circleci/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/circleci/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from mbmtest.srv.example.com
debug3: hostkeys_foreach: reading file "/home/circleci/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/circleci/.ssh/known_hosts:2
debug3: load_hostkeys: loaded 1 keys from 54.76.124.236
debug1: Host 'mbmtest.srv.example.com' is known and matches the ECDSA host key.
debug1: Found key in /home/circleci/.ssh/known_hosts:1
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug2: key:  (0x55c2ae2fb730), agent
debug2: key:  (0x55c2ae2fa5c0), agent
debug2: key:  (0x55c2ae2fd910), agent
debug2: key:  (0x55c2ae2fdfe0), agent
debug2: key: /home/circleci/.ssh/id_rsa ((nil))
debug2: key: /home/circleci/.ssh/id_dsa ((nil))
debug2: key: /home/circleci/.ssh/id_ecdsa ((nil))
debug2: key: /home/circleci/.ssh/id_ed25519 ((nil))
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key:
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug1: Offering RSA public key:
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug1: Offering RSA public key:
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug1: Offering RSA public key:
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug1: Trying private key: /home/circleci/.ssh/id_rsa
debug3: no such identity: /home/circleci/.ssh/id_rsa: No such file or directory debug1: Trying private key: /home/circleci/.ssh/id_dsa
debug3: no such identity: /home/circleci/.ssh/id_dsa: No such file or directory debug1: Trying private key: /home/circleci/.ssh/id_ecdsa
debug3: no such identity: /home/circleci/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /home/circleci/.ssh/id_ed25519
debug3: no such identity: /home/circleci/.ssh/id_ed25519: No such file or directory
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey).

As far as I can see, it looks like CircleCI doesn’t have the required key set on its server, even though it’s there on the SSH Permissions page and also explicitly noted in the config.yml.

3 Likes
5

It looks like you added add_ssh_keys to your test job, did you want it on your deploy job, since that’s where you are trying to use it?

I agree to leaving the thread here, very helpful to others, good thinking!

6

Thanks - I’m an idiot… re-running on the deploy job…

7

Not an idiot, probably just either too early or too late in the day :slight_smile:

edit: (or time for the mid-day nap) :wink:

8

Unfortunately, even with the add_ssh_keys in the config.yml I’m still getting the same error.

Here’s the diff of the ssh -vvv:

Just for completeness, here’s the outcome of ssh -vvv deploy@example.com with the new settings: https://www.diffchecker.com/232h5v27

9

@g105b Can you get me an ls of the ~/.ssh directory? Are the other keys there and just this one is missing, or are none of them there?

10 
circleci@51b3597b9e8d:~$ ls -la ~/.ssh/
total 20
drwxr-xr-x 2 circleci circleci 4096 Jun 14 13:33 .
drwxr-xr-x 5 circleci circleci 4096 Jun 14 13:33 ..
-rw-r--r-- 1 circleci circleci  134 Jun 14 13:33 config
-rw------- 1 circleci circleci 1675 Jun 14 13:33 id_rsa_2813aa418af58aaa46b22ba7bf31ca3c
-rw-r--r-- 1 circleci circleci  444 Jun 14 13:33 known_hosts

I can see that the name of the id_rsa file includes the correct fingerprint for the key that is in the authorized_keys file on the remote ssh server.

11

I see it in your diff too. Can you cat me /home/circleci/.ssh/config please?

12 
circleci@51b3597b9e8d:~$ cat /home/circleci/.ssh/config
Host mbmtest.srv.example.com
  IdentitiesOnly yes
  IdentityFile /home/circleci/.ssh/id_rsa_2813aa418af58aaa46b22ba7bf31ca3c

I’ve just changed the domain name to example.com above.

13

I’ve just changed the domain name to example.com above.

I assumed :slight_smile:

Ok, next question. Can you ssh to the server from the build with the -i flag telling ssh to use that key? That will tell us if it’s an issue with the key or not.

14 
circleci@51b3597b9e8d:~$ ssh -i /home/circleci/.ssh/id_rsa_2813aa418af58aaa46b22ba7bf31ca3c deploy@mbmtest.srv.example.com
Permission denied (publickey).

Looks like the key is not working, although I can use the exact same key from another server to connect. Also, the keys that I already have in CircleCI’s SSH Permissions page do work from this build server.

15

Can you take a look at the key file itself on the build server and ensure you didn’t miss part of the header or get a weird char in there when you added it to the UI?

16

Yes, the SSH key looks perfect:

image

17

Thank you for your patience. Did you edit the ls listing, or are the other two (working) keys not listed there? Can you use ssh-keygen -l -f <keyfile> to verify what format the working ones are in?

18

The ls listing was untouched. I expect the other keys are not present in this build due to the add_ssh_keys being added to the config.yml.

Here’s the type of SSH key used:

circleci@51b3597b9e8d:~$ ssh-keygen -l -f ~/.ssh/id_rsa_2813aa418af58aaa46b22ba7bf31ca3c
2048 SHA256:zOs05cKUzYfUan8ZFklVVKLouPG7D6GhmHgsoZ+V4Ag no comment (RSA)
19

It looks like you genned the keys on the dest server. I know you said you changed the keys, but is there any chance you copied the wrong pub key to your dest server’s authorized_keys file? The fingerprint we are using doesn’t seem to match what you say you generated.

20

Ah, since I took the screenshots I tried generating a key with a different key size, so the original screenshots’ fingerprints won’t match. I’ve updated my original post with this note as to not confuse others that are having similar issues.

I don’t mind generating a new key and completely going through the process from scratch in case the problem lies in the way I’ve generated keys. Do you have any specific guide for generating keys and adding to the correct places, so I can make absolutely sure I’m doing it in the correct way?

The destination server only has the one private-public key pair. The only record in the authorized keys is the public key which I have validated to match the private key stored in Circle.

1 Like