Docker_layer_caching inconsistencies

Build Environment
, , , ,
1

Hi,

I use docker_layer_caching to cache the Docker image between test runs and jobs but there are many times this caching doesn’t even caches the image between jobs in the same test run.

In my setup, the docker image will change only if one of these files changes: Dockerfile, docker-compose.yml, Pipfile.lock (python requirements file).
But the docker image build is sometimes run, even if these files didn’t changed.

In the summarized version of my .circleci/config.yml below, we can see that I run a build step, then I persist_to_workspace and open 6 parallel jobs with attach_workspace where tests are run.

In this workflow, several times the following happens:

  • There is no changes to docker related files,
  • No docker build is run in build step
  • But docker build is run in 1, 2 or 3 out of the 6 parallel machines.

What can I do to correctly cache docker image and only run docker build if there are changes to the related files?

Can I ensure the same build is used by the 6 parallel jobs with persist_to_workspace/attach_workspace?

Thank you.

A summarized version of my config.yml:

version: 2.1

orbs:
  codecov: codecov/codecov@1.2.0

commands:
  setup:
    description: "Setting up the test environment"
      - run:
          name: Build containers
          command: |
            docker-compose build
    ...

executors:
  test-machine:
    machine:
      image: ubuntu-2004:202201-02
      docker_layer_caching: true

jobs:
  build:
    executor: test-machine
    steps:
      - checkout
      - setup
      - chmod
      - persist_to_workspace:
          root: .
          paths: .

  tests:
    executor: test-machine
    parallelism: 6
    steps:
      - attach_workspace:
          at: .
      - run:
          name: Run tests
          command: |
            sudo chmod -R 777 .
            set -x
            TEST_FILES=$(circleci tests glob "**/test_*.py" | circleci tests split --split-by=timings --timings-file timing.json)
            echo $TEST_FILES | tr ' ' '\n' | sort | uniq | tr '\n' ' ' > test_files.txt
            cp .env.test .env
            docker-compose run app sh -c "until psql -d postgres -c '\q'; do sleep 1; done; pytest $(cat test_files.txt)"

workflows:
  version: 2

  git event checks:
    jobs:
      - build

      - tests:
          requires:
            - build