Cannot SSH into custom docker container using ssh-agent

ssh
keys
docker

#1

For one of my projects (rsync-cache I build an integration test suite. A docker container gets build containing a prepared authorized_keys file. After building the container I load the matching private key into ssh-agent and start my integration test suite. The test suite connects via rsync+ssh to the container and performs some copy actions.

On circleci the ssh connection cannot be established as the container seems to be unreachable:

ssh: connect to host localhost port 32770: Connection refused

When I SSH into the build machine and try to connect via ssh - like shown below - the ssh-agent authentication seems to be broken. The ssh-agent authentication works fine locally (Mac OS X 10.10.5) but seems to be broken when running on circleci. It always falls back to interactive password authentication which obviously fails my build, as no interaction is done.

$ ssh -p 32770 root@localhost
Warning: Permanently added ‘[localhost]:32770’ (ECDSA) to the list of known hosts.
root@localhost’s password:
root@2b7ea32e85b2:~#

Last run: https://circleci.com/gh/matthiasbalke/rsync-cache/14
circle.yml: https://github.com/matthiasbalke/rsync-cache/blob/integration-test/circle.yml
integration test script: https://github.com/matthiasbalke/rsync-cache/blob/integration-test/integration-test/run.sh

I hope you got some hints for me.


#2

Can you try explicitly passing the key in your ssh command?

ssh -i $PATH_TO_PRIVATE_KEY -p $PORT root@localhost


#3

Thats working:

ubuntu@box451:~/rsync-cache$ ssh -i integration-test/integration-test-auth -p 32768 root@localhost
Warning: Permanently added ‘[localhost]:32768’ (ECDSA) to the list of known hosts.
root@b174c8e099d6:~#


#4

Great! Does this solve your problem?


#5

No, I’m sorry. I just tried to check the ssh connection. I want to use publickey authentication using rsync, without modifying my rsync script I’m testing. So I’m wondering, why the client (build agent) seems to prefer password based authentication.

UPDATE
I added -v to rsync’s ssh arguments to figure out what’s happening. It seems that it also does not accept the additional -i argument, which worked when using plain ssh like you told me.
This is the log:

$ cd integration-test && ./run.sh

Starting docker sshd server …
done.

Restoring cache ‘restore’ to ‘/home/ubuntu/rsync-cache/integration-test/source’ …
OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014

debug1: Reading configuration data /home/ubuntu/.ssh/config

debug1: /home/ubuntu/.ssh/config line 1: Applying options for localhost

debug1: Reading configuration data /etc/ssh/ssh_config

debug1: /etc/ssh/ssh_config line 1: Applying options for *

debug1: Connecting to localhost [127.0.0.1] port 32768.

debug1: connect to address 127.0.0.1 port 32768: Connection refused

ssh: connect to host localhost port 32768: Connection refused

Last action was not successfull. Aborting!

cd integration-test && ./run.sh returned exit code 255

When running the same script on my mac I get this log:

$ cd integration-test && ./run.sh

Starting docker sshd server …
done.

Restoring cache ‘restore’ to ‘/Volumes/HDD/Users/matthias/repositories/rsync-cache/integration-test/source’ …
OpenSSH_6.2p2, OSSLShim 0.9.8r 8 Dec 2011
debug1: Reading configuration data /Volumes/HDD/Users/matthias/.ssh/config
debug1: Reading configuration data /etc/ssh_config
debug1: /etc/ssh_config line 20: Applying options for *
debug1: Connecting to 127.0.0.1 [127.0.0.1] port 32780.
debug1: Connection established.
debug1: identity file integration-test-auth type 1
debug1: identity file integration-test-auth-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.2
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.8
debug1: match: OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.8 pat OpenSSH*
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5-etm@openssh.com none
debug1: kex: client->server aes128-ctr hmac-md5-etm@openssh.com none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: RSA 83:48:e9:26:70:c4:d3:06:f7:12:1f:0f:a6:52:c7:e2
debug1: checking without port identifier
Warning: Permanently added ‘[127.0.0.1]:32780’ (RSA) to the list of known hosts.
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /Volumes/HDD/Users/matthias/.ssh/elton.new
debug1: Authentications that can continue: publickey,password
debug1: Offering RSA public key: /Volumes/HDD/Users/matthias/.ssh/bitBucket.org
debug1: Authentications that can continue: publickey,password
debug1: Offering RSA public key: /Volumes/HDD/Users/matthias/.ssh/github
debug1: Authentications that can continue: publickey,password
debug1: Offering RSA public key: /Volumes/HDD/Users/matthias/.ssh/jenkins-cache-access
debug1: Authentications that can continue: publickey,password
debug1: Offering RSA public key: integration-test-auth
debug1: Server accepts key: pkalg ssh-rsa blen 535
debug1: read PEM private key done: type RSA
debug1: Authentication succeeded (publickey).
Authenticated to 127.0.0.1 ([127.0.0.1]:32780).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: Sending environment.
debug1: Sending env LANG = de_DE.UTF-8
debug1: Sending command: test -d "/mnt/restore"
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
debug1: channel 0: free: client-session, nchannels 1
Transferred: sent 6072, received 3508 bytes, in 0.0 seconds
Bytes per second: sent 1857473.1, received 1073125.1
debug1: Exit status 1
Cache ‘restore’ does not exist.
Done.
++ Success!

UPDATE @levlaz Can this problem be caused by the circleci specific ~/.ssh/config which forces a given private key for connections to localhost?

Host localhost
IdentitiesOnly yes
IdentityFile /home/ubuntu/.ssh/build_key.rsa

Host node0
HostName 127.0.0.1
User ubuntu
Port 22
IdentityFile ~/.ssh/build_key.rsa
IdentitiesOnly yes
LocalCommand mkdir -p $CIRCLE_ARTIFACTS/__circleci_debug && echo sshed to node0 >> $CIRCLE_ARTIFACTS/__circleci_debug/ssh_audit.log
PermitLocalCommand yes

Host github.com
IdentitiesOnly yes
IdentityFile /home/ubuntu/.ssh/id_circleci_github


#6