Docker recently announced that rate limits will apply to anonymous image pulls from Docker Hub starting on November 1st, 2020. But as long as you add Docker authentication to your pipeline config, you can avoid service disruption.
If you use the Docker executor or pull Docker images when using the machine executor on CircleCI, we encourage you to authenticate. Because the anonymous API rate limits are based on IP address, they will impact CircleCI cloud and server customers. Authenticated users get higher per-user rate limits regardless of IP.
We are currently working on a partnership with Docker to minimize the impact of this rate limit change for our users and will share more details as we get them.
If you have any questions or concerns, please feel free to post them here.
CircleCI has partnered with Docker to ensure that our users can continue to access Docker Hub without rate limits. On November 1st, with few exceptions, you should not be impacted by any rate limits when pulling images from Docker Hub through CircleCI.
However, these rate limits may go into effect for CircleCI users in the future. That’s why we’re encouraging you and your team to add Docker Hub authentication to your CircleCI configuration and consider upgrading your Docker Hub plan, as appropriate, to prevent any impact from rate limits in the future.
I totally understand that it’s not much time to make all the changes you might need to make, and we’re here to help as much as we can. Regarding the timeline, we’ve been working with Docker since they announced these changes to Docker Hub, and we’re hoping for a better solution from Docker in the future. As soon as we have information from them, we can make announcements like this and start working on solutions.
There are also emails going out to all CircleCI users about this very soon, but we wanted to get the information public as soon as we could, which is why we posted here first.
One annoying thing about Docker Hub’s access tokens is that they are for personal use. I.e: you’ll have to create a machine account and give it permissions to your organization’s repos if you don’t want to tie an individual team members account for CI access:
The CircleCI documentation doesn’t discuss those jobs using orbs, like aws-ecr/build-and-push-image etc. Are these not mentioned because CircleCI is caching these containers locally and not going out to Docker Hub?
I got the email but I struggle to really understand what this is about. Does this affect all CircleCI jobs that use Docker? What about CircleCI’s pre-built images?
From the email and Docker’s website I understand that there is a limit on anonymous pulls. That also means that some jobs probably don’t reach that limit, and thus require no changes, right? Else the wording would have been ‘anonymous pulls are disabled’ (limit = 0). (However I can’t find what the limit is.)
I’d love to hear more information from those that better understand this topic (I’m not a power user if that wasn’t apparent already ).
What is the recommended solution for when the “Build forked pull requests” option is enabled but not “Pass secrets to builds from forked pull requests”?
As those builds wont have any pull credentials they will only be able to pull images unauthenticated and so will surely fail almost all the time
I’ve added auth -> username and auth -> password to my docker block so the CircleCI image pulls will be authenticated.
What about when I’m using setup_remote_docker and building Docker images? We push our images to a private registry, but the base images are pulled FROM public Docker Hub images. I’m wondering if we need to explicitly docker login to Docker Hub (in addition to our private registry) to avoid the rate limit when building images.
It would be pretty cool if we could set the DockerHub auth username and password once for our organization and not need to pass the auth block for every job. It’s going to produce a ton of clutter in all of our configs.
@ms1111 I just debugged with ssh on a test job. From what I can tell, CircleCI does not automatically log you into remote docker when using setup_remote_docker, even when specifying the auth parameter. I tested this by pulling down circleci/node:14.8.0 and pushing a private version of it to DockerHub as my-company/circleci-node:14.8.0.