[Updated] Authenticate with Docker to avoid impact of Nov. 1st rate limits

[lewang]

Hi everyone,

Docker recently announced that rate limits will apply to anonymous image pulls from Docker Hub starting on November 1st, 2020. But as long as you add Docker authentication to your pipeline config, you can avoid service disruption.

If you use the Docker executor or pull Docker images when using the machine executor on CircleCI, we encourage you to authenticate. Because the anonymous API rate limits are based on IP address, they will impact CircleCI cloud and server customers. Authenticated users get higher per-user rate limits regardless of IP.

We are currently working on a partnership with Docker to minimize the impact of this rate limit change for our users and will share more details as we get them.

If you have any questions or concerns, please feel free to post them here.

Update: Our teams are hard at work creating solutions and gathering information about this change. Our support team has put together a list of FAQs, including how to tell if your authentication was successful.

Good news!

CircleCI has partnered with Docker to ensure that our users can continue to access Docker Hub without rate limits. On November 1st, with few exceptions, you should not be impacted by any rate limits when pulling images from Docker Hub through CircleCI.

However, these rate limits may go into effect for CircleCI users in the future. That’s why we’re encouraging you and your team to add Docker Hub authentication to your CircleCI configuration and consider upgrading your Docker Hub plan, as appropriate, to prevent any impact from rate limits in the future.

Thanks and happy building!

[_joe]

Would it be feasible to have magic global environment variables for this so that we don’t have to change hundreds of lines of code?

For ECR this already exists, if we configure AWS credentials the pull works without configuring aws_auth in the manifest.

[Cbeck527]

@_joe

Would it be feasible to have magic global environment variables for this so that we don’t have to change hundreds of lines of code?

+1 – Docker Hub also provides access tokens that could help facilitate this.

I have to say, I’m a little disappointed that we were only given a ~30 day notice for this on an opt-in announcement platform.

[thekatertot]

Hi @Cbeck527!

I totally understand that it’s not much time to make all the changes you might need to make, and we’re here to help as much as we can. Regarding the timeline, we’ve been working with Docker since they announced these changes to Docker Hub, and we’re hoping for a better solution from Docker in the future. As soon as we have information from them, we can make announcements like this and start working on solutions.

There are also emails going out to all CircleCI users about this very soon, but we wanted to get the information public as soon as we could, which is why we posted here first.

[ruffsl]

@Cbeck527

+1 – Docker Hub also provides access tokens that could help facilitate this.

One annoying thing about Docker Hub’s access tokens is that they are for personal use. I.e: you’ll have to create a machine account and give it permissions to your organization’s repos if you don’t want to tie an individual team members account for CI access:

New in Docker Hub: Personal Access Tokens - Docker Blog

Feature Request: DockerHub Auth Token for third party apps

[keilw]

Hi,
I’m not sure, if I have a DockerHub account.

As there were also examples without it, let’s take a very simple JVM job:

version: 2
jobs:
  clean:
    docker:
      # specify the version you desire here
      - image: circleci/openjdk:11.0-jdk-buster-node-browsers-legacy

What would we have to add here, just “machine: true” after the job name, or is there something else?

Thanks,
Werner

[random930]

The CircleCI documentation doesn’t discuss those jobs using orbs, like aws-ecr/build-and-push-image etc. Are these not mentioned because CircleCI is caching these containers locally and not going out to Docker Hub?

[Jura]

I got the email but I struggle to really understand what this is about. Does this affect all CircleCI jobs that use Docker? What about CircleCI’s pre-built images?

From the email and Docker’s website I understand that there is a limit on anonymous pulls. That also means that some jobs probably don’t reach that limit, and thus require no changes, right? Else the wording would have been ‘anonymous pulls are disabled’ (limit = 0). (However I can’t find what the limit is.)

I’d love to hear more information from those that better understand this topic (I’m not a power user if that wasn’t apparent already ).

[alican]

Hello Friend! I’ve got confused first but now I have a decent understanding.

Let me try to answer your questions @Jura :

• Circleci pre-built images are hosted on dockerhub. Therefore this change will affect you.

• Anonymous pull count is based on source IP address. (source) I assume circleci has a pool of IP addresses and we all share those IPs.

• Anonymous pulls limit is 100 pulls / every 6 hours (source)

[bartimez]

Does this mean that the auth credentials need to be added after every image pulled in the build?

[comzeradd]

Is there a way to verify that the pull request is authenticated? Should we look for any additional message in job logs window?

[Julusian]

What is the recommended solution for when the “Build forked pull requests” option is enabled but not “Pass secrets to builds from forked pull requests”?
As those builds wont have any pull credentials they will only be able to pull images unauthenticated and so will surely fail almost all the time

[ms1111]

I’ve added auth -> username and auth -> password to my docker block so the CircleCI image pulls will be authenticated.

What about when I’m using setup_remote_docker and building Docker images? We push our images to a private registry, but the base images are pulled FROM public Docker Hub images. I’m wondering if we need to explicitly docker login to Docker Hub (in addition to our private registry) to avoid the rate limit when building images.

[thekatertot]

Just want you all to know we’re working on answers to these questions! We’re still here and still listening.

[agentreno]

Can I particularly raise @random930’s question about orbs? If an orb doesn’t expose a parameter for docker-auth, and it’s using an executor based on a docker hub image, will it be rate-limited?

This is the case for popular orbs like the aws-cli orb. The only workaround if it exceeded rate limits would be to abandon the orb temporarily in favour of a BYO jobs with the same functionality.

[blimmer]

It would be pretty cool if we could set the DockerHub auth username and password once for our organization and not need to pass the auth block for every job. It’s going to produce a ton of clutter in all of our configs.

[blimmer]

@ms1111 I just debugged with ssh on a test job. From what I can tell, CircleCI does not automatically log you into remote docker when using setup_remote_docker, even when specifying the auth parameter. I tested this by pulling down circleci/node:14.8.0 and pushing a private version of it to DockerHub as my-company/circleci-node:14.8.0.

The initial pull succeeds:

docker:
  image: my-company/circleci-node:14.8.0
  auth:
    username: $DOCKERHUB_USERNAME
    password: $DOCKERHUB_ACCESS_TOKEN

But then the docker build fails because it’s not authenticated with those same credentials:

ERROR: Service 'test' failed to build: pull access denied for *********/circleci-node, repository does not exist or may require 'docker login'

If I login again in a run step, it works:

- run: docker login -u $DOCKERHUB_USERNAME -p $DOCKERHUB_ACCESS_TOKEN

It doesn’t look like setup_remote_docker accepts an auth block (docs), which would be nice.