Docker Hub failures

For the last 3 days, our team has had random failures between Docker Hub and CirclecCI. We have multiple accounts with 3+ paid nodes, so this is becoming frustrating quickly.

examples:

  • Error response from daemon: error parsing HTTP 404 response body: invalid character ‘p’ after top-level value: “404 page not found\n”

  • Starting container circleci/postgres:9.6.5-alpine-ram
    image is cached as circleci/postgres:9.6.5-alpine-ram, but refreshing…

    Error response from daemon: unauthorized: authentication required

  • error pulling image configuration: unknown blob
    Received ExitFailure 1 when running

We are using the correct DockerHub credentials. The same CI pipeline has not had Docker login problems for 2+ years so I am certain the credentials work just fine.

What can be done? I have reached out to CircleCI by contact form, on Twitter, and now here.

On the CircleCI status page, they indicate they found the error and there are “very few errors” remaining, yet 50% of my builds are still failing.

Some Docker Hub Operations are degraded

Resolved - We are seeing very few issues but still they are happening. We are closing the incident but if we see a spike we will re-open it.
Nov 24, 23:09 UTC
Update - Apologies for the “fix has been implemented” message, it has not - I blame the pumpkin pie! We are monitoring the situation and will update
Nov 24, 21:10 UTC
Monitoring - A fix has been implemented and we are monitoring the results.
Nov 24, 21:08 UTC
Identified - Our Support team has identified that some operations with Docker Hub are failing. We will monitor Docker Hub and notifiy you when it has resolved.

1 Like

Hi,

Has this been resolved for you? There was a some left over issues that affected myself as well between our platform and Docker Hub. Their status page said it was green as well but everything is working for me now as of yesterday.

My builds have now started to finish successfully as well. Thank you for looking into it.

1 Like

Yes, all is good now. Thank you.

I do want to mention one more thing, now that I am significantly less frustrated.

This problem affected our team for days. In fact, I have three commercial products using CircleCI that were affected.

During that time, we only briefly saw any indication of a problem on CircleCI’s status page. We spent effort trying to solve and work around a problem that wasn’t our fault, because there was not enough transparency on CircleCI’s end for us to understand what was happening.

I tried all the communication channels available to me with CircleCI and was still unable to receive any confirmation of a problem.

Nearly the entire time that notification was up, we saw " Resolved - We are seeing very few issues…". Yet, our builds were effectively unavailable for 3 days (check our logs).

The lack of transparency on CircleCI’s end on this issue is a bummer, and the only incident that has made me truly lose confidence in relying on Circle over the years. I hope in the future this kind of issue can be more effectively communicated, even if it’s only lingering for a “small” amount of users.

Thanks for checking up.

1 Like

Thank you for taking the time to provide this feedback. I’ll share this internally with the larger team.

I’ve been experiencing this issue over the past week while trying to pull the postgres:9.6-ram image.

We pull multiple images from Docker Hub for each build, but the only one that seems to be returning this intermittent 404 or authentication error is the circleci/postgres:9.6-ram one. I’ve had it happen as recently as yesterday when queueing up multiple workflows.

An interesting thing to note is that this is the only circleci image that we utilize, all the others come from their respective organization’s repositories which seem to be progressing without issue.

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.