Docker push/pull jobs hang when using private AWS ECR repository

We first started seeing at around 4pm PST yesterday (Sept 10, 2019) that jobs which pull or push containers to our private AWS ECR docker repository hang indefinitely. They have 0 output and have to be manually canceled, or they will seemingly be killed after a hard 5 hour limit. Rerunning the failed jobs will occasionally work.

Looking at the ECR side, I can see that the docker push commands do succeed and push containers to the repository. However CircleCI will often not progress beyond this point.

We haven’t had any changes in our Circle configuration or infrastructure for a long while, and we’ve never had a problem with this until yesterday. Now it’s destroyed our workflow as we have to manually go and cancel workflows which are stuck.

Examples:
https://app.circleci.com/jobs/bitbucket/TheClarityProject/clir/68701
https://app.circleci.com/jobs/bitbucket/TheClarityProject/clir/68735
app (dot) circleci (dot) com/jobs/bitbucket/TheClarityProject/clir/68745 (manually canceled after 56 minutes)

This appears to be related to Steps with progress output stop running . Switching our docker push/pull commands to

docker push <...> | cat

appears to have fixed the problem by suppressing the progress bars.