Aug 13-14 'Unable to run the job runner' issue

Jobs using the image circleci/build-image failed to run starting on August 13th with the following message:

CircleCI was unable to run the job runner because we were unable to execute commands in build container.
This typically means that the build container entrypoint or command is terminating the container prematurely, or interfering with executed commands. Consider clearing entrypoint/command values and try again.

This was caused by a change to fix a bug with the mongodb service image, which was running into a problem with zombie processes .

The fix in this case was to run docker containers with the --init flag enabled, which causes docker to run a tini init process as PID1. We expected this not to cause any issues, but as it turns out, more people run init daemons inside their containers than we expected.

In most cases, your best bet for long term stability and upgrades will be to use a list of individually versioned containers that run the same versions of each dependency as your production system. CircleCI produces a number of pre-built containers for use, but we don’t restrict you to using those.

We will be looking into a way to support the mongodb container as well as containers which need to run their own init daemon, but until then I would recommend the following:

  • Remove the command: /sbin/init line
  • Use separate containers for any backing services you need (see Databases and Database Configuration Examples)
  • Consider moving away from the circleci/build-image container image, as it is no longer being updated
    • If you need ubuntu specifically, you should be able to use any Ubuntu derived container Docker Hub