CircleCI seems to be caching an old docker image

Build Environment
, , ,
1

All our repos and images are private, so I am just using placeholder names below as an example

Builds were failing due to an image we were using having an older version of node installed on it (8.3.1). So I created a new tag for that image with an updated version of node (12.3.1) but it seems as though Circle is still using the old image.

In the failed build “Spinning up environment” shows:

Starting container org/image:12.3.1
  image cache not found on this host, downloading org/image:12.3.1

So it looks like the image is not caching.

Now the failing command is “lint”

#!/bin/bash -eo pipefail
npm run lint

> assistant@2.2.4 lint /assistant
> ng lint

You are running version v8.12.0 of Node.js, which is not supported by Angular CLI 8.0+.
The official Node.js version that is supported is 10.9 or greater.

As the error message above shows, we are using an old version of node.

I have rerun the build job with SSH, SSH’d onto that box and run the same command successfully.

ssh command provided by CircleCI
$ node --version
v12.3.1
$ npm run lint
successfully runs

Also to sanity check I have also created my own instance of that image and can do the same, using the following steps (the image is private so you won’t be able to do this).

$ docker pull org/image:12.3.1
$ docker run -d --name builder org/image:12.3.1 tail -f
$ docker exec -it builder /bin/bash

Then

$ node --version
v12.3.1

So despite pulling the brand new image when it spins up the environment, it still looks as though CircleCI is using an old image.

The config.yml looks like this

version: 2.0
jobs:

  test:
    docker:
      - image: org/image:12.3.1
        auth:
          username: correct_deets
          password: $correct_deets


    working_directory: /assistant

    steps:
      - checkout

      - restore_cache:
          keys:
            - npm-cache-{{ checksum "package-lock.json" }}

      - run: npm i

      - save_cache:
          key: npm-cache-{{ checksum "package-lock.json" }}
          paths:
            - node_modules

      - run:
          name: lint
          command: npm run lint
2

I added a sanity check to the build as well:

#!/bin/bash -eo pipefail
node -v
v8.12.0

And then SSH’ing onto the box and running the same:

$ node -v
v12.3.1
3

For clarification, does the 12.3.1 point to both the old and the new images? I believe that in these circumstances, it should pull the newer one.

Edit, to clarify: you can’t have the same tag on more than one different image. However, I think you can have 12.3.1.1, 12.3.1.2, etc - and then Docker works it out, based on semver principles. I assume that is how one can pull a broad version tag and get all the recent minor updates.

4

It pulls the latest image. I confirmed this by pulling the 12.3.1 docker image and then running an instance locally outside of CI

5

Ah, gotcha. In the CircleCI pull step, check the sha256 hash for an old commit (where the old image was correct) and for a newer commit (where the new image would be expected). If they are the same, then that would probably be a platform bug, which I would suggest a support ticket is logged for.

You could work around it in the meantime by using another tag name for the same image.

6

Thanks @halfer - I have raised a support request with Circle CI.

I can see the hash being used on the failed build:

using image org/image@sha256:etc

Again I manually pulled this image outside of CircleCI and could get the correct version.

Not sure what is going on here.