I use docker_layer_caching to cache the Docker image between test runs and jobs but there are many times this caching doesn’t even caches the image between jobs in the same test run.
In my setup, the docker image will change only if one of these files changes: Dockerfile, docker-compose.yml, Pipfile.lock (python requirements file).
But the docker image build is sometimes run, even if these files didn’t changed.
In the summarized version of my .circleci/config.yml below, we can see that I run a build step, then I persist_to_workspace and open 6 parallel jobs with attach_workspace where tests are run.
In this workflow, several times the following happens:
- There is no changes to docker related files,
- No docker build is run in build step
- But docker build is run in 1, 2 or 3 out of the 6 parallel machines.
What can I do to correctly cache docker image and only run docker build if there are changes to the related files?
Can I ensure the same build is used by the 6 parallel jobs with persist_to_workspace/attach_workspace?
A summarized version of my config.yml:
version: 2.1 orbs: codecov: firstname.lastname@example.org commands: setup: description: "Setting up the test environment" - run: name: Build containers command: | docker-compose build ... executors: test-machine: machine: image: ubuntu-2004:202201-02 docker_layer_caching: true jobs: build: executor: test-machine steps: - checkout - setup - chmod - persist_to_workspace: root: . paths: . tests: executor: test-machine parallelism: 6 steps: - attach_workspace: at: . - run: name: Run tests command: | sudo chmod -R 777 . set -x TEST_FILES=$(circleci tests glob "**/test_*.py" | circleci tests split --split-by=timings --timings-file timing.json) echo $TEST_FILES | tr ' ' '\n' | sort | uniq | tr '\n' ' ' > test_files.txt cp .env.test .env docker-compose run app sh -c "until psql -d postgres -c '\q'; do sleep 1; done; pytest $(cat test_files.txt)" workflows: version: 2 git event checks: jobs: - build - tests: requires: - build