CircleCI was unable to run the job runner

Hello everyone.

This was caused by a change to fix a bug with the mongodb service image, which was running into a problem with zombie processes.

The fix in this case was to run docker containers with the --init flag enabled, which causes docker to run a tini init process as PID1. We expected this not to cause any issues, but as it turns out, more people run init daemons inside their containers than we expected.

In most cases, your best bet for long term stability and upgrades will be to use a list of individually versioned containers that run the same versions of each dependency as your production system. CircleCI produces a number of pre-built containers for use, but we don’t restrict you to using those.

We will be looking into a way to support the mongodb container as well as containers which need to run their own init daemon, but until then I would recommend the following:

  • Remove the command: /sbin/init line
  • Use separate containers for any backing services you need (see Databases and Database Configuration Examples)
  • Consider moving away from the circleci/build-image container image, as it is no longer being updated
    • If you need ubuntu specifically, you should be able to use any Ubuntu derived container Docker Hub

I hope that helps provide some context for why this happened.

4 Likes