Docker "--cache-from" flag not working


#1

Docker supports the ability to specify where it should fetch it cache from --cache-from however in all of my testing on CircleCI 2.0 this doesn’t appear to work. If there was some insight from the CircleCI team as to why it doesn’t work that would be really useful.

This is a great little post that help to describe it in detail…


Building a Docker image in 2.0
#2

Are you trying to do this with the setup_remote_docker or with another setup?


#3

We support Docker layer caching natively, but you need to get whitelisted for it:

https://circleci.com/docs/2.0/docker-layer-caching/

If you haven’t gotten whitelisted, it won’t work.


#4

@rohara Not sure why that’s a good replacement –

With reusable: true, CircleCI will try to reuse your Remote Docker Environment; every layer you built in a previous job will be accessible there. We do our best to reuse the previous environment when we can, but in some cases your job may find itself in a clean environment, even if you’ve specified reusable: true.

Depending on the job you’ll typically end up in a clean environment and forced to rebuild your layers.

@Eric

Yes, it’s with setup_remote_docker

What I’m running is:

version: 2

jobs:
  build:
    docker:
      - image: docker:17.03.0-ce-git

    environment:
      - AWS_REGION: us-east-1
      - CONTAINER_IMAGE: my-project-repository

    working_directory: ~/build

    steps:
      - run:
          name: Install AWS CLI
          command: |
            apk add --update python python-dev py-pip build-base jq
            pip install --upgrade awscli
            aws --version
            aws configure set default.region $AWS_REGION
            aws configure set default.output json

      - setup_remote_docker

      - checkout

      - run:
          name: Build Image
          command: |
            docker build \
                -f docker/Dockerfile.base --rm=false \
                -t ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/${CONTAINER_IMAGE}:${CIRCLE_BRANCH} \
                -t ${CONTAINER_IMAGE} \
                .

      - run:
          name: Run unit tests
          command: |
            docker run ${CONTAINER_IMAGE} npm test

      - deploy:
          name: Tag and push a production image
          command: |
           if [[ $CIRCLE_BRANCH = production ]]; then
             docker tag ${CONTAINER_IMAGE} ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/${CONTAINER_IMAGE}:1.0.${CIRCLE_BUILD_NUM}
           fi

      - deploy:
          name: Deploy to docker repository
          command: |
            eval $(aws ecr get-login --region ${AWS_REGION})
            docker push ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/${CONTAINER_IMAGE}

#5

I should have mentioned is what I would like to be able to do is take the following run step and change it to have the cache-from as such:

  - run:
      name: Build Image
      command: |
        eval $(aws ecr get-login --region us-east-1)
        repo=${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/${CONTAINER_IMAGE}
        if aws ecr describe-images --repository-name=my-repo --image-ids imageTag=${CIRCLE_BRANCH} ; then tag=${CIRCLE_BRANCH} ; else tag=latest ; fi

        docker build \
            --cache-from ${repo}:${tag} \
            -f docker/Dockerfile.base --rm=false \
            -t ${repo}:${CIRCLE_BRANCH} \
            -t ${CONTAINER_IMAGE} \
            .

#6

Hi Koblas,

Are you pulling the image manually? I believe the ‘–cache-from’ flag will not pull the image for you, you have to manually pull it.


#7

Hi, I’m having the same problem, not seeing the layers getting reused with --cache-from. I pull the image beforehand.

- run:
      name: Build the Docker image
      command: |
        docker pull $DOCKER_IMAGE:latest
        docker build --cache-from $DOCKER_IMAGE:latest -t $DOCKER_IMAGE:$CIRCLE_SHA1 .

#8

I did try to manually pull the image before the --cache-from. However, this didn’t appear to have any effect.


#9

Hmm, weird. I have used it with the machine executor and it works for me, haven’t tried it with the docker executor.


#10

@koblas @dominikkrejcik That article says this flag was added in Docker 1.13. Are you both installing this version? If so, can you share links to builds where you tried this?


#11

I detailed a bunch of attempts in this post -

Let me know if this answers your question.


#12

Yes, I am using Docker 17.03.0-ce. I got it working in the end, the key was to build the image on Circle, push it and pull that exact image the in the next build (as opposed to using one that was built from the same commit but locally).


#13

@dominikkrejcik - Can you provide an example, would be helpful to me.


#14

I don’t have an example but you can build a new config containing the sha of the new image in question to pull in that specific image.


#15
docker pull $DOCKER_IMAGE:latest
docker build --cache-from $DOCKER_IMAGE:latest \
             -t $DOCKER_IMAGE:$CIRCLE_SHA1 .
docker tag $DOCKER_IMAGE:$CIRCLE_SHA1 $DOCKER_IMAGE:latest
docker push $DOCKER_IMAGE:latest

Docker image is an env var we set in our config.

Run this twice, and you should be seeing caching working the second time.


#16

Will say that I put that back into my config.yaml and I don’t seen any performance changes. It’s still running all of the steps in the base Dockerfile. I know they work, since when run on my local machine I see the performance improvements.


#17

Hey, it seems to be related with an issue with Docker : https://github.com/docker/docker/issues/31613


#18

@koblas @b4stien

I wanted to thank you both. Spent a day getting this working and that GitHub ticket seems to point to the correct workaround, which is to make sure that you pin the base image with a digest in the Dockerfile. E.g. if your image is based on python:3.6-alpine, then modify it so it starts with FROM python:3.6-alpine@sha256:<digest hash>.

You can find the current digest under RepoDigests with docker inspect <base image>. Once that’s set up and your latest image is built using that base image, --cache-from will correctly use the cached base image instead of redoing all the build steps. FYI, this is under docker executor.

@Eric

It seems to me that correctly working --cache-from seems to obviate the need for https://circleci.com/docs/2.0/docker-layer-caching/. Is there any additional benefit from using CircleCI’s native caching over --cache-from?


#19

The benefit is we keep your containers running. It’s not necessarily a benefit in all use cases, but it greatly simplifies a lot of common workflows.


#20