Docker images frequently not being passed with DLC to subsequent steps

Build Environment Running Tests
1

I’m trying to build an app image in job 1 and run 2 sets of tests in parallel jobs 2 and 3 with following config:

version: 2.1
jobs:
 build_candidate_artifacts:
    docker:
      - image: cimg/node:16.19.0
    resource_class: medium+
    steps:
      - setup_remote_docker:
          version: 20.10.18
          docker_layer_caching: true
      - checkout
      - run:
          name: Build test env Docker image
          command: docker compose -f ./docker/docker-compose.test.yml build --progress plain
  
  run_e2e_tests:
    docker:
      - image: cimg/base:2023.01
    resource_class: small
    steps:
      - setup_remote_docker:
          version: 20.10.18
          docker_layer_caching: true
      - checkout
      - run:
          name: Launch app network
          command: docker compose -f docker/docker-compose.test.yml up -d --wait
      - run:
          name: Build E2E runner
          command: docker build -f Dockerfile.e2e -t e2e-runner .
          working_directory: e2e
      - run:
          name: Run E2E tests
          command: docker run --network app-test --name e2e-container -it e2e-runner yarn ci
          working_directory: e2e
      # ...
  # - run_backend_tests with similar command

workflows:
  run-all-tests:
    jobs:
      - build_candidate_artifacts:
      - run_e2e_tests:
          requires:
            - build_candidate_artifacts
      - run_backend_tests:
           requires:
             - build_candidate_artifacts

It works well on some of the days, on other I have to re-run the workflow up to 5 times for the test job to run properly. The issues faced include:

  • run_e2e_tests job rebuilds Docker images from scratch which fails, because there are required dependencies in build job
  • only one of run_e2e_tests and run_backend_tests jobs picks up the built image, second one starts building from scratch. order has been random
  • the app-test network reports as up and healthy, but test-app container does not respond to http requests

I had hoped that DLC will allow to speed up the pipeline. Though, I could not find in the docs whether such setup is guaranteed to work.

Please advise what is the correct way to build a candidate Docker image, run tests on it and in case of success promote it to staging/production. Currently it fails unexpectedly that does not allow me to use it for main branch process.

2

The main docs for DLC can be found here

https://circleci.com/docs/docker-layer-caching/

I’ll have to take a guess as I do not use DLC (I use self-hosted runners) but the section titled " How DLC works" seems to match what you are3 seeing. So you need to add a parallelism value to cause 2 copies of the image cache to be generated so that each of your following test jobs can have access to an image cache.

The write-up regarding parallelism can be found here