Does docker_layer_caching support buildx?

My CircleCI configuration has a job defined with a machine executor using docker_layer_caching. This currently works as expected using docker build. However, when moving to a custom builder to target multiple platforms, I do not see subsequent runs using a cache. The documentation for this option does not say anything regarding using buildx.

The main change in the job is:

docker buildx create --use --name myname
docker buildx build …

It’s plausible to me that a custom builder may need to be configured in some way to leverage the cache that CircleCI is meant to add. Is there any advice on what that could look like?

Thanks

hi @stackptr

Apologies for the late follow-up here!

I would like to share that indeed, I can confirm that docker buildx is not playing nice with Docker Layer Caching (DLC) on CircleCI builds;
I also could not get my own builds utilizing DLC when using docker buildx build specifically.

We also see this mentioned in the CircleCI aws-ecr Orb here:

In particular, the aws-ecr Orb has moved to using docker buildx for building images.
As such, developers using the current aws-ecr Orb (at 8.1.2) are also seeing DLC not utilized in their builds.

Unfortunately, I do not have an answer / solution for this yet!
Please allow us to reach out and confirm with our Engineering team on this, and follow up here :bowing_man:

If you have any findings, please let us know here too as well.

Thank you for your understanding in this matter.

As a follow-up, I would like to share solutions and some explanation around Docker Buildx and the Docker Layer Cache (DLC) feature on CircleCI.

Re: DLC on CircleCI

When you build a Docker image, its layers are saved within the Docker root directory.
The Docker root directory is typically /var/lib/docker for most setups.

However, builds on CircleCI (or any CI) are ephemeral.
The next build is likely on a different virtual machine (VM) and so we need a way to “pass” around the stored layers between builds.

Hence, DLC on CircleCI works by storing and retrieving these layers in an external volume instead.
These volumes are attached to the remote machine running the Docker daemon.
The idea is that these volumes will likely include all, if not some of the layers as they get passed around.

More information: Enabling Docker Layer Caching - CircleCI

Re: How does this impact Docker Buildx

With Docker Buildx, developers can choose where the cached layers should be stored and retrieved.

There are many options, but this includes the local cache (folder), or a registry cache (i.e., saving the layers in a Docker repo on a registry like Docker Hub).

See the following options for the docker buildx build command:

With such options, you do not necessarily need the DLC feature in order to pass around the cache storage between builds.

Example: Cache from local folder

jobs:
  build-push-local-docker:
    docker:
    - image: cimg/base:stable
    resource_class: medium
    environment:
    - DOCKER_REGISTRY: docker.io
    - DOCKER_USER: <user_or_org>
    - DOCKER_LOGIN: <login_user>
    steps:
    - checkout
    - setup_remote_docker:
        version: 20.10.14
    - run:
        name: Check Docker settings (for buildx)
        command: |
          docker version
          docker buildx version
          docker context inspect
    - run:
        name: Setup docker buildx
        command: |
          docker context create circleci
          docker buildx create --use circleci

          docker buildx ls
          docker context inspect circleci
    - run:
        name: Login to registry
        command: |
          # login credentials should be provided via context or project environment variables.
          echo $DOCKER_PASSWORD | docker login $DOCKER_REGISTRY --username $DOCKER_LOGIN --password-stdin
    - restore_cache:
        keys:
        - hello-buildx-docker-{{ arch }}-{{ .Branch }}-
        - hello-buildx-docker-{{ arch }}-
    - run:
        name: Docker buildx with local cache
        command: |
          docker buildx build --progress=plain \
            --tag="${DOCKER_REGISTRY}/${DOCKER_USER}/hello-buildx:${CIRCLE_SHA1}" \
            --cache-to=type=local,mode=max,dest=/tmp/dockercache \
            --cache-from=type=local,src=/tmp/dockercache \
            --output=type=docker \
            .

          docker image ls
    - save_cache:
        key: hello-buildx-docker-{{ arch }}-{{ .Branch }}-{{ checksum "/tmp/dockercache/index.json" }}
        paths:
        - /tmp/dockercache
    - run:
        name: Publish image
        command: |
          docker image tag "${DOCKER_REGISTRY}/${DOCKER_USER}/hello-buildx:${CIRCLE_SHA1}" "${DOCKER_REGISTRY}/${DOCKER_USER}/hello-buildx:local-docker"
          docker image push "${DOCKER_REGISTRY}/${DOCKER_USER}/hello-buildx:local-docker"
    - run:
        name: Prune cache
        command: |
          docker buildx prune

This solution uses the local cache option for --cache-from and --cache-to.
With the local cache strategy, we can take advantage of CircleCI’s restore_cache and save_cache steps to load and save the cached layers between builds!

Hope this helps! :nerd_face:

I have a sample repo showcasing the registry and local cache strategies:

1 Like