Build docker image fails randomly with "Error response from daemon: container is marked for removal and cannot be started"

Hi all!

We’re experiencing random failures of our docker image builds. The error we get is “Error response from daemon: container is marked for removal and cannot be started”. The error happens randomly: sometimes the build is successful and sometimes it fails without anything being changed from our side. We started to get this error on 15 March and we did not have it before (the environment, build scripts and images were not changed for at least 1 month).

Any thoughts on what could be the problem?

We’re seeing the the same issue when using the buildx multiarch image. I first started seeing it on March 10.

Upgrading the machine executor we were using from ubuntu-2004:202010-01 to ubuntu-2004:202101-01 along with upgrading from buildx 0.4.1 to 0.5.1 seems to have fixed it.

I was incorrect; the issue has started reoccurring.

EDIT:
After some additional investigation, it seems that this was happening due to the multiarch container being cached across builds. I also noticed that we weren’t logging into Docker. To resolve this I added the following to the beginning of each build:

echo $PASSWORD | docker login --username $USERNAME --password-stdin
docker rm -f buildx_buildkit_docker-multiarch0

The name of your multiarch builder differs from mine; make sure to take that into account.

We haven’t seen the issue occur since making these changes.

1 Like

Thanks for your help!

We have handled it in a similar way by adding:
docker container prune -f
It has solved the issue for us.

Upon advice from support, we’ll also add a debug step that uploads the buildkit logs as artifacts in CircleCI to check further.

2 Likes