[Updated] Authenticate with Docker to avoid impact of Nov. 1st rate limits

Update

We’ve added a warning message when auth is invalid, so you should see:

Warning: No authentication provided, this pull may be subject to Docker Hub download rate limits.

2 Likes

Any word on authenticating to orbs like aws-ecr? Even if CircleCI and Docker work out a deal to avoid throttling, it seems useful to allow non-anonymous pulls. And if you don’t work out a deal, Nov 1 is only two weeks away now.

We’re working on it right now, and will hopefully have something official to share very soon that will make it easy.

@trevorr We do have a support article about workarounds for orbs right now if you’re interested!

1 Like

Any direction you can provide me on this one.

Great, thank you! :+1:

For people that look for this message, it’s approximately the fifth line in the ‘Spin up environment’ task. For example:

Build-agent version 1.0.41417-4036f5a3 (2020-10-16T14:37:07+0000)
Docker Engine Version: 18.09.6
Kernel Version: Linux 8dc5fabcbe7d 4.15.0-1077-aws #81-Ubuntu SMP Wed Jun 24 16:48:15 UTC 2020 x86_64 Linux
Starting container cibuilds/hugo:0.62.2
Warning: No authentication provided, this pull may be subject to Docker Hub download rate limits. 
  image cache not found on this host, downloading cibuilds/hugo:0.62.2
0.62.2: Pulling from cibuilds/hugo
...
1 Like

fyi, I added this to the first step of each job to automatically verify the build with a connection error when it’s running. (In short, it’s redirecting docker domains to localhost to fail the connection)

- run: |
  echo "127.0.0.1 registry-1.docker.io auth.docker.io index.docker.io dseasb33srnrn.cloudfront.net production.cloudflare.docker.com" | sudo tee -a /etc/hosts
  sleep 5

You would still have to manually verify the docker executor image yourself.

Any recommendations on how to enable authenticating with Docker in forked PRs?

So far, the only way I can think of is to enable “Pass secrets to builds from forked pull requests” and then utilize contexts to limit which env variables are accessible to the jobs being run in forked PRs.

In other words, it seems that if you’re using Docker executors in jobs that are exposed to forked PRs, there is no way to protect your Docker username and access key.

Would be nice if CircleCI added project or org-level authentication that would take place outside of the pipeline.

1 Like

I really want to suggest (again) that something higher level that required less adjustment of individual steps would really have been ideal.

Also, w/r/t orbs, it would be nice if Circle could publish a convention (e.g., DOCKERHUB_USER / DOCKERHUB_PASSWORD that would be relatively standard so that orbs could (if people followed the convention) take those from environment and / or a context without additional parameter configuration.

1 Like

As far as both orbs and the convention, I see a very interesting comment from @KyleTryon in the following comment with a little tease:

Also, this confirms that DOCKERHUB_USERNAME would probably be the convention for anything that did need to have those magic env vars (re: my post above)

1 Like

Good news!

CircleCI has partnered with Docker to ensure that our users can continue to access Docker Hub without rate limits. On November 1st, with few exceptions, you should not be impacted by any rate limits when pulling images from Docker Hub through CircleCI.

However, these rate limits may go into effect for CircleCI users in the future. That’s why we’re encouraging you and your team to add Docker Hub authentication to your CircleCI configuration and consider upgrading your Docker Hub plan, as appropriate, to prevent any impact from rate limits in the future.