What’s Happening
Starting April 1, Docker Hub is enforcing stricter rate limits on container image pulls Usage and limits. While CircleCI customers executing a high volume of tasks may be at risk for potential disruption, the actual impact on individual customers will depend on their specific Docker plan and adherence to the fair usage policy.
Notice
The following information about rate limits applies primarily to customers using their own Docker accounts and credentials. If no credentials are provided, CircleCI’s credentials are used, which do not have the same rate limits due to our agreement with Docker. This applies specifically to Docker executors but NOT remote Docker.
Vulnerable Customers
The most susceptible customers to rate limiting are those who run thousands of CI/CD jobs daily, have loads of parallelized workflows, frequently pull public Docker images, or use shared Docker Hub credentials across teams. Note that on CircleCI, if users do not authenticate with their own credentials, they will use CircleCI’s credentials, bypassing the rate limits associated with personal Docker accounts.
Additionally, Remote Docker users face special challenges; without fallback credentials, they have higher limits when logged in, which means they can hit rate limits faster than other users.
Executors such as Machine, macOS, or Windows customers may experience rate limits if they provide their own credentials.
Identifying Rate Limit Errors
Customers may encounter sudden build failures across multiple pipelines with “429 Too Many Requests” errors. This generally indicates that they have exceeded their pull rate limit, which is particularly relevant for unauthenticated users (limited to 10 pulls/hour) and users with free accounts (limited to 100 pulls/hour). It can also occur due to persistent network issues or service outages on Docker Hub.
To troubleshoot, customers can check their rate limit usage on Docker Hub, verify their credentials, and monitor Docker Hub’s status page for any ongoing issues.
For authenticated users on Pro, Team, or Business subscriptions, there are unlimited pulls per hour, provided they adhere to fair use. When pull limits are reached, authenticated users will see a more detailed error message that includes a link to the documentation.
These errors can occur intermittently throughout the hour, and customers should review their pull usage to identify patterns. For more information on pull usage and limits, please refer to the Pulls .
Recommended Remediation
1. Implement Docker Access Token Authentication
Docker Access Tokens provide a more secure and reliable authentication method than passwords and help manage rate limits effectively.
To optimize pull requests and reduce the impact of rate limits, it is crucial to use authenticated Docker Hub pulls. Ensure that all users are authenticated when pulling images.
Additionally, it is recommended to create separate Organization Access Tokens (OATs) for each project. Each OAT operates under its own usage quota, meaning that distributing your Docker pulls across multiple tokens reduces the overall load on a single account. This way, when one token reaches its limit, only that specific project is affected, rather than all builds failing simultaneously.
Spreading Pull Requests Across Multiple Token
Set up separate OATs for each project or team instead of using one token for everything. This spreads your Docker pulls across multiple independent quotas, which gives you more total pulls before hitting limits. When one token reaches its limit, only that specific project is affected rather than all your builds failing at once.
Benefits of Organization Access Tokens (OATs)
Organization Access Tokens enable enhanced security monitoring by tracking last usage timestamps to identify and respond to suspicious activities. They offer detailed permission control to restrict each token’s access scope, minimizing potential damage if a token becomes compromised. With distributed management, all organization owners maintain management capabilities for tokens, ensuring continuity even when individual owners depart. Furthermore, OATs operate under their own Docker Hub usage quotas, which remain independent from your personal account’s limitations.
Creating an OAT:
- Log into Admin Console and select your organization.
- Navigate to Security and access > Access tokens and click Generate access token.
- Add a descriptive label and choose an expiration date.
- Configure repository access permissions (optional public repo access and add specific repositories).
- Set permission levels for each repository (Image Push or Image Pull) and save.
CircleCI Configuration Example:
version: 2.1
jobs:
build:
docker:
- image: circleci/python:3.9
steps:
- checkout
- setup_remote_docker
- run:
name: Authenticate with Docker Hub
command: |
echo "$DOCKERHUB_TOKEN" | docker login -u "$DOCKERHUB_USERNAME" --password-stdin
- run:
name: Pull docker image
command: docker pull some-image:latest
It is highly recommended to rotate tokens quarterly for security and document token ownership.
2. Distribute Workloads Across Multiple Accounts
Create dedicated Docker Hub accounts for different teams or departments. For high-volume projects, assign unique Docker Hub accounts and configure project-specific CircleCI contexts with appropriate credentials to ensure proper resource allocation and prevent rate limit issues across your organization.
Customers should store DOCKERHUB_USERNAME and DOCKERHUB_TOKEN as CircleCI environment variables and use context-specific environments for different projects:
workflows:
version: 2
docker_workflow:
jobs:
- build:
context: docker-context
Additional Resources
Docker provides extensive documentation on rate limits at Usage and limits.
For more information about access token management, visit Organization access tokens .
If customers need direct support from Docker, they can contact them at Contact | Docker or visit Docker Community Forums .
For CircleCI-specific guidance, customers can refer to (Caching dependencies - CircleCI ) and for guides on using private images (Using Docker authenticated pulls - CircleCI ).