AWS ECR Unable to Push: no basic auth credentials

awscli
ecr
docker
aws

#1

Hi there,

Am trying to push a newly build image to AWS ECR and for some reason the docker client is completely unable to remember the login to ECR. Here is what the -deploy step looks like in my config.yml file:

      - setup_remote_docker   
      - deploy:
          name: ECR Docker Package & Push
          environment:
            - AWS_ECR_URL: 728736720051.dkr.ecr.us-east-1.amazonaws.com
          command: |
            # Only package build branches.
            [ -n "${AF_BUILD_BRANCHES##*$CIRCLE_BRANCH*}" ] && exit 0
            set -x
            VERSION="$(git describe --always --dirty)"
            BUILD_TAG="foo/bar:${VERSION}"
            REPO_TAG="${AWS_ECR_URL}/${BUILD_TAG}"
            docker build -t "$REPO_TAG" .
            eval $(aws ecr get-login --region $AWS_REGION)
            docker push "$REPO_TAG"

The most important steps here are the last two. The eval statement is what invokes the login command and the push is what I need to have happen to get the image to the proper location. The login command is successful:

++ aws ecr get-login --region us-east-1
+ eval docker login -u AWS -p <SUPER SECRET PASSWORD> -e none https://728736720051.dkr.ecr.us-east-1.amazonaws.com
++ docker login -u AWS -p <SUPER SECRET PASSWORD> -e none https://728736720051.dkr.ecr.us-east-1.amazonaws.com
Flag --email has been deprecated, will be removed in 1.13.
Login Succeeded

But when I do the push command next I get:

+ docker push 728736720051.dkr.ecr.us-east-1.amazonaws.com/foo/bar:v13.0-17-g6be4413
The push refers to a repository [728736720051.dkr.ecr.us-east-1.amazonaws.com/foo/bar]


cd5a6bdc988b: Preparing 


53cded873cfd: Preparing 


cbb60610bc99: Preparing 


bfea7a2d1282: Preparing 


77dd2c65c94f: Preparing 


22ad39b1eede: Preparing 


039c25e10efb: Preparing 


0f20784b55ff: Preparing 


2607b744b89d: Preparing 


e7b0b4cd055a: Preparing 


445ed6ee6867: Preparing 


c59fa6cbcbd9: Preparing 


8d4d1ab5ff74: Preparing 

0f20784b55ff: Waiting 

2607b744b89d: Waiting 

e7b0b4cd055a: Waiting 

445ed6ee6867: Waiting 

c59fa6cbcbd9: Waiting 

8d4d1ab5ff74: Waiting 
no basic auth credentials
Exited with code 1

Has anyone else hit this yet? Thoughts?


#2

There are a few possible causes of that http://docs.aws.amazon.com/AmazonECR/latest/userguide/common-errors-docker.html#error-403


#3

Indeed I saw that and I have eliminated all of them AFAICT. The region is being explicitly set during login and you can confirm that the repos are the same with the debug output in my first post. I was able to login successfully locally using the same build credentials so its not a token expiration issue. And I’m on a linux box so I won’t suffer from the third bullet. I’m wondering if the docker daemon is trying to save something and is unable to do so.


#4

Right… debugged the issue finally. Its a docker issue, but I’m not exactly sure what the issue really is. Obviously it is some form of issue stemming to the interaction between client and server, but I don’t know exactly what that is (or how to easily debug it for that matter). Instead let me show you my scenario. Here is the original setup. Its a Fedora 25 based docker image trying to build a docker container and upload it to AWS:

# docker version
Client:
 Version:         1.12.6
 API version:     1.24
 Package version: docker-common-1.12.6-6.gitae7d637.fc25.x86_64
 Go version:      go1.7.4
 Git commit:      ae7d637/1.12.6
 Built:           Mon Jan 30 16:15:28 2017
 OS/Arch:         linux/amd64

Server:
 Version:         17.03.0-ce
 API version:     1.26
 Package version: 
 Go version:      go1.7.5
 Git commit:      60ccb22
 Built:           Thu Feb 23 10:57:47 2017
 OS/Arch:         linux/amd64

# eval $(aws ecr get-login)
Flag --email has been deprecated, will be removed in 1.13.
Login Succeeded

# docker push 728736720051.dkr.ecr.us-east-1.amazonaws.com/foo/bar:testing
The push refers to a repository [728736720051.dkr.ecr.us-east-1.amazonaws.com/foo/bar]
4ac76077f2c7: Preparing 
no basic auth credentials

Yep. Oops… there is our original problem. Wonderful. Now lets download and install the older version of the docker client provided by Circle CI at https://s3-external-1.amazonaws.com/circle-downloads/docker-1.9.0-circleci

curl -sL -o /usr/bin/docker 'https://s3-external-1.amazonaws.com/circle-downloads/docker-1.9.0-circleci'
chmod 755 /usr/bin/docker

Great. Now lets try again after first printing out our version information:

[root@5adcbe3c1137 bin]# docker version
Client:
 Version:      1.9.0-circleci-cp-workaround
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   3cce968-dirty
 Built:        Wed Dec 16 14:48:19 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      17.03.0-ce
 API version:  1.26
 Go version:   go1.7.5
 Git commit:   60ccb22
 Built:        2017-02-23T10:57:47.731437667+00:00
 OS/Arch:      linux/amd64
[root@5adcbe3c1137 bin]# eval $(aws ecr get-login)
WARNING: login credentials saved in /root/.docker/config.json
Login Succeeded
[root@5adcbe3c1137 bin]# docker push 728736720051.dkr.ecr.us-east-1.amazonaws.com/foo/bar:testing
The push refers to a repository [728736720051.dkr.ecr.us-east-1.amazonaws.com/foo/bar]
4ac76077f2c7: Layer already exists 
testing: digest: sha256:c79345819a6882c31b41bc771d9a94fc52872fa651b36771fbe0c8461d7ee558 size: 527

WTF! The circle CI client uses an older client that uses an older API version, yet somehow that works well with their docker server and my newer client (with a newer API version) does not. This is lunacy.

If docker client is to blame here… this is a docker bug for not griping about incompatible API versions. Any reasonable client software should complain loudly if it can’t talk to the server for any reason. If this is a CircleCI issue, its because they changed something in their docker client/server interaction and failed to mention that you MUST use their client for things to work well / at all.

Either way you look at it something is definitely not stirring the Koolaid here. CircleCI folks… thoughts here?

Anyway… for any of you suffering from the BS issue, simply use their docker client as I noted above and you are golden. Cheers.


#5

It’s intended for 1.0; that’s why it has “workaround” in the version name. Just install Docker normally.


#6

It’s intended for 1.0; that’s why it has “workaround” in the version name. Just install Docker normally.

If I do that then I get the original issue described above. So I can’t do that and have my stuff work. Perhaps you meant something else?


#7

Can you please link me to those builds? I am misunderstanding something.


#8

Sure.

failure: https://circleci.com/bb/airfordable/afbeta/40
success: https://circleci.com/bb/airfordable/afbeta/43

The difference was what version of the docker client I was using. The failed one was the version packaged with Fedora 25. The successful one was the one you all provide in your public s3 bucket.


#9

Definitely don’t do this:

/bin/sh -c curl -L -o /usr/bin/docker    'https://s3-external-1.amazonaws.com/circle-downloads/docker-1.9.0-circleci'    && chmod 755 /usr/bin/docker

That’s a specific version of Docker that was revised to function properly on 1.0. Use vanilla Docker on 2.0, but maybe not the Fedora one. I’d install it the same way the Docker folks are building their own image:

You can pull it from any of their Dockerfiles;

https://hub.docker.com/_/docker/


#10

This topic was automatically closed 41 days after the last reply. New replies are no longer allowed.