Docker Hub rate limiting - how to prepare?

For our OSS project, while the number of pulls from CI jobs can exceed the current limits, the number of triggered CI workflows is well below this. It would be nice if dependent jobs in a workflow could be qued to the same CI worker such that the image needn’t be re-pulled if within a user defined time window for that executor. This would have the added benefit of speeding up consecutive jobs that use the same image by skipping the minutes of spin-up time in downloading the docker-image/file-caches for the next dependent job container, as well as help ensure all jobs in the same workflow are using the same sha of the docker image tag, resolving related issues where an image may update between jobs in the same workflow.

Alternatively, CircleCI could erect a transparent registry between CI workers and DockerHub and interpret the same user defined time window parameter (e.g. 06h00m00sec) for that executor config to determine when the cache of the image-tag should be considered stale and re-pull the manifest before discarding old/unused layers. This would be different from the current Docker Layer Caching feature as it would be for caching the pulling of docker image for running the job, not the layers created when building in the job.

https://circleci.com/docs/2.0/docker-layer-caching/

As an concrete example, we use a nightly job to update the CI image on DockerHub, then pull that image repeatedly for the graph of jobs in every workflow, sparing resources and maximize caching.

https://app.circleci.com/pipelines/github/ros-planning/navigation2?branch=main

1 Like