Docker Layer Caching not working

Build Environment Caching Dependencies
1

tl;dr:

DCL is not working for my project in the free plan. Is there some project configuration I need to change? See details below.

Hey folks, I do have the following configuration to build a Docker image:

  docker-build:
    docker:
      - image: cimg/base:stable
    steps:
      - checkout
      - setup_remote_docker:
          docker_layer_caching: true
      - run:
          name: List docker images
          command: |
            docker image ls --all
      - run:
          name: Docker pull build image
          command: |
            image=$(cat Dockerfile | grep "AS build" | awk -F" " '{ print $2 }' | xargs)
            docker pull "$image"
      - run:
          name: Build Docker image
          command: |
            docker build --progress plain -t "imagename:latest" .

And there is nothing really special about the Dockerfile:

FROM gradle:8.6.0-jdk21 AS build

# Install Node JS
RUN apt-get update -y && apt-get install --no-install-recommends -y curl \
    && curl -fsSL https://deb.nodesource.com/setup_20.x | bash - \
    && apt-get install -y --no-install-recommends nodejs \
    && node -v

# Better layer caching by installing dependencies first
WORKDIR /app
COPY package.json package-lock.json /app/
COPY settings.gradle.kts build.gradle.kts gradlew gradle.properties /app/
COPY gradle /app/gradle
COPY buildSrc /app/buildSrc

# Will generate a cached layer that contains all the dependencies
RUN mkdir -p /app/src/main/jte && \
  npm install && \
  gradle -Dsonar.gradle.skipCompile=true --console plain --no-configuration-cache classes -x assetsPipeline

# Build application
COPY . /app/
RUN gradle -Dsonar.gradle.skipCompile=true --console plain --no-configuration-cache \
      clean shadowJar -x test -x accessibilityTest \
      && mv -vf build/libs/*.jar app.jar

FROM eclipse-temurin:21.0.2_13-jre-alpine

COPY --from=build /app/app.jar .
EXPOSE 8080
HEALTHCHECK CMD curl -f "http://localhost:8080/" || exit 1

RUN addgroup --system hyginia && adduser --system -G hyginia hyginia
USER hyginia

ENTRYPOINT [ "java", "-jar", "app.jar" ]

It is trying to use Docker layer caching as much as possible. When I run it locally, I see this in the second run (same command as in the CircleCI configuration above):

#0 building with "desktop-linux" instance using docker driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 2.23kB done
#1 DONE 0.0s

#2 [internal] load metadata for docker.io/library/gradle:8.6.0-jdk21
#2 ...

#3 [internal] load metadata for docker.io/library/eclipse-temurin:21.0.2_13-jre-alpine
#3 DONE 0.3s

#2 [internal] load metadata for docker.io/library/gradle:8.6.0-jdk21
#2 DONE 0.3s

#4 [internal] load .dockerignore
#4 transferring context: 279B done
#4 DONE 0.0s

#5 [build  1/10] FROM docker.io/library/gradle:8.6.0-jdk21@sha256:ac6222e64c5ce39bc47b78a4f26ce0599f70bdfff5e0c95d0f9372a4c356421f
#5 DONE 0.0s

#6 [stage-1 1/3] FROM docker.io/library/eclipse-temurin:21.0.2_13-jre-alpine@sha256:fb4150a30569aadae9d693d949684a00653411528e62498b9900940c9b5b8a66
#6 DONE 0.0s

#7 [internal] load build context
#7 transferring context: 83.50kB 0.0s done
#7 DONE 0.0s

#8 [build  4/10] COPY package.json package-lock.json /app/
#8 CACHED

#9 [build  5/10] COPY settings.gradle.kts build.gradle.kts gradlew gradle.properties /app/
#9 CACHED

#10 [build  7/10] COPY buildSrc /app/buildSrc
#10 CACHED

#11 [build  8/10] RUN mkdir -p /app/src/main/jte &&   npm install &&   gradle -Dsonar.gradle.skipCompile=true --console plain --no-configuration-cache classes -x assetsPipeline
#11 CACHED

#12 [build  2/10] RUN apt-get update -y && apt-get install --no-install-recommends -y curl     && curl -fsSL https://deb.nodesource.com/setup_20.x | bash -     && apt-get install -y --no-install-recommends nodejs     && node -v
#12 CACHED

#13 [build  6/10] COPY gradle /app/gradle
#13 CACHED

#14 [build  3/10] WORKDIR /app
#14 CACHED

#15 [build  9/10] COPY . /app/
#15 CACHED

Meaning that the cache is working as expected.

But on CircleCI, I see:

#1 [internal] load build definition from Dockerfile
#1 sha256:92c8aa583b9d7e47cec2dd909680e4ca343c82e524fc5be016e346864cef26bb
#1 transferring dockerfile: 2.25kB done
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 sha256:44ca9db220276377e925c46fbace9fd75659559bff0714ec053e4ecb496fe115
#2 transferring context: 279B done
#2 DONE 0.0s

#4 [internal] load metadata for docker.io/library/gradle:8.6.0-jdk21
#4 sha256:5bb5d20a8447f1578d5eafee569ce9311bd890a027ed32934a8ee29c284fee6f
#4 DONE 0.0s

#3 [internal] load metadata for docker.io/library/eclipse-temurin:21.0.2_13-jre-alpine
#3 sha256:8066503c00222779dcc38b4324444d6ac9e2fcfeee5ecd24b46eaf3002672066
#3 DONE 0.1s

#16 [stage-1 1/3] FROM docker.io/library/eclipse-temurin:21.0.2_13-jre-alpine@sha256:fb4150a30569aadae9d693d949684a00653411528e62498b9900940c9b5b8a66
#16 sha256:5a6d6202b1e781bfee384ded732518c70523415768978a15a675c4cb2723a805
#16 DONE 0.0s

#15 [build  1/10] FROM docker.io/library/gradle:8.6.0-jdk21
#15 sha256:da7258c1d91e7a806c057ece6a581e7f02a77256458f6f3e8ebfaf7e441fe29e
#15 DONE 0.0s

#12 [internal] load build context
#12 sha256:5440e721d5735a6d9ff8b7624818d87a0eac908f800dc75592b615894a174a91
#12 transferring context: 18.68MB 0.1s done
#12 DONE 0.2s

#15 [build  1/10] FROM docker.io/library/gradle:8.6.0-jdk21
#15 sha256:da7258c1d91e7a806c057ece6a581e7f02a77256458f6f3e8ebfaf7e441fe29e
#15 DONE 0.0s

#14 [build  2/10] RUN apt-get update -y && apt-get install --no-install-recommends -y curl     && curl -fsSL https://deb.nodesource.com/setup_20.x | bash -     && apt-get install -y --no-install-recommends nodejs     && node -v
#14 sha256:dcdc6e129346f573baea44c7aae70ba09c2acf900992e99e1f000f4a6b5d58b0
#14 0.254 Get:1 http://archive.ubuntu.com/ubuntu jammy InRelease [270 kB]
#14 0.329 Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB]
#14 0.347 Get:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB]
#14 0.422 Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages [1,792 kB]
#14 0.488 Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 Packages [17.5 MB]
#14 0.639 Get:6 http://archive.ubuntu.com/ubuntu jammy/restricted amd64 Packages [164 kB]
#14 0.641 Get:7 http://archive.ubuntu.com/ubuntu jammy/multiverse amd64 Packages [266 kB]
#14 0.654 Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,343 kB]
#14 0.666 Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,735 kB]
#14 0.681 Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,822 kB]
#14 0.697 Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [50.4 kB]
#14 0.698 Get:12 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.1 kB]
#14 0.698 Get:13 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [50.4 kB]
#14 1.008 Get:14 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB]
#14 1.393 Get:15 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [44.6 kB]
#14 1.487 Get:16 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,784 kB]
#14 1.799 Get:17 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,456 kB]
#14 1.865 Get:18 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [1,070 kB]
#14 2.129 Fetched 29.7 MB in 2s (16.4 MB/s)
#14 2.129 Reading package lists...
#14 3.064 Reading package lists...
#14 4.030 Building dependency tree...
#14 4.288 Reading state information...
#14 4.524 curl is already the newest version (7.81.0-1ubuntu1.15).
#14 4.524 0 upgraded, 0 newly installed, 0 to remove and 5 not upgraded.
#14 4.630 2024-02-15 16:45:06 - Installing pre-requisites
#14 4.724 Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease
#14 4.733 Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
#14 4.749 Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
#14 4.834 Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease
#14 5.030 Reading package lists...
#14 5.971 Reading package lists...
#14 6.860 Building dependency tree...
#14 7.075 Reading state information...
#14 7.289 ca-certificates is already the newest version (20230311ubuntu0.22.04.1).
#14 7.289 curl is already the newest version (7.81.0-1ubuntu1.15).
#14 7.289 The following additional packages will be installed:
#14 7.289   dirmngr gnupg-l10n gnupg-utils gpg gpg-agent gpg-wks-client gpg-wks-server
#14 7.290   gpgconf gpgsm libassuan0 libksba8 libnpth0 pinentry-curses
#14 7.291 Suggested packages:
#14 7.291   dbus-user-session libpam-systemd pinentry-gnome3 tor parcimonie xloadimage
#14 7.291   scdaemon pinentry-doc
#14 7.328 The following NEW packages will be installed:
#14 7.328   apt-transport-https dirmngr gnupg gnupg-l10n gnupg-utils gpg gpg-agent
#14 7.329   gpg-wks-client gpg-wks-server gpgconf gpgsm libassuan0 libksba8 libnpth0
#14 7.330   pinentry-curses
#14 7.376 0 upgraded, 15 newly installed, 0 to remove and 5 not upgraded.
#14 7.376 Need to get 2,311 kB of archives.
#14 7.376 After this operation, 6,007 kB of additional disk space will be used.
#14 7.376 Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 apt-transport-https all 2.4.11 [1,510 B]
#14 7.390 Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 libassuan0 amd64 2.5.5-1build1 [38.2 kB]
#14 7.410 Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 gpgconf amd64 2.2.27-3ubuntu2.1 [94.2 kB]
#14 7.433 Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libksba8 amd64 1.6.0-2ubuntu0.2 [119 kB]
#14 7.446 Get:5 http://archive.ubuntu.com/ubuntu jammy/main amd64 libnpth0 amd64 1.6-3build2 [8,664 B]
#14 7.447 Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 dirmngr amd64 2.2.27-3ubuntu2.1 [293 kB]
#14 7.472 Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 gnupg-l10n all 2.2.27-3ubuntu2.1 [54.4 kB]
#14 7.473 Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 gnupg-utils amd64 2.2.27-3ubuntu2.1 [308 kB]
#14 7.477 Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 gpg amd64 2.2.27-3ubuntu2.1 [519 kB]
#14 7.488 Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 pinentry-curses amd64 1.1.1-1build2 [34.4 kB]
#14 7.489 Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 gpg-agent amd64 2.2.27-3ubuntu2.1 [209 kB]
#14 7.493 Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 gpg-wks-client amd64 2.2.27-3ubuntu2.1 [62.7 kB]
#14 7.493 Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 gpg-wks-server amd64 2.2.27-3ubuntu2.1 [57.5 kB]
#14 7.494 Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 gpgsm amd64 2.2.27-3ubuntu2.1 [197 kB]
#14 7.497 Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 gnupg all 2.2.27-3ubuntu2.1 [315 kB]
#14 8.245 debconf: delaying package configuration, since apt-utils is not installed
#14 8.283 Fetched 2,311 kB in 0s (14.3 MB/s)
#14 8.306 Selecting previously unselected package apt-transport-https.
(Reading database ... 13624 files and directories currently installed.)
#14 8.315 Preparing to unpack .../00-apt-transport-https_2.4.11_all.deb ...
#14 8.319 Unpacking apt-transport-https (2.4.11) ...
#14 8.362 Selecting previously unselected package libassuan0:amd64.
#14 8.363 Preparing to unpack .../01-libassuan0_2.5.5-1build1_amd64.deb ...
#14 8.370 Unpacking libassuan0:amd64 (2.5.5-1build1) ...
#14 8.396 Selecting previously unselected package gpgconf.
#14 8.398 Preparing to unpack .../02-gpgconf_2.2.27-3ubuntu2.1_amd64.deb ...
#14 8.400 Unpacking gpgconf (2.2.27-3ubuntu2.1) ...
#14 8.435 Selecting previously unselected package libksba8:amd64.
#14 8.437 Preparing to unpack .../03-libksba8_1.6.0-2ubuntu0.2_amd64.deb ...
#14 8.440 Unpacking libksba8:amd64 (1.6.0-2ubuntu0.2) ...
#14 8.484 Selecting previously unselected package libnpth0:amd64.
#14 8.485 Preparing to unpack .../04-libnpth0_1.6-3build2_amd64.deb ...
#14 8.488 Unpacking libnpth0:amd64 (1.6-3build2) ...
#14 8.525 Selecting previously unselected package dirmngr.
#14 8.527 Preparing to unpack .../05-dirmngr_2.2.27-3ubuntu2.1_amd64.deb ...
#14 8.541 Unpacking dirmngr (2.2.27-3ubuntu2.1) ...
#14 8.575 Selecting previously unselected package gnupg-l10n.
#14 8.577 Preparing to unpack .../06-gnupg-l10n_2.2.27-3ubuntu2.1_all.deb ...
#14 8.580 Unpacking gnupg-l10n (2.2.27-3ubuntu2.1) ...
#14 8.612 Selecting previously unselected package gnupg-utils.
#14 8.614 Preparing to unpack .../07-gnupg-utils_2.2.27-3ubuntu2.1_amd64.deb ...
#14 8.617 Unpacking gnupg-utils (2.2.27-3ubuntu2.1) ...
#14 8.650 Selecting previously unselected package gpg.
#14 8.652 Preparing to unpack .../08-gpg_2.2.27-3ubuntu2.1_amd64.deb ...
#14 8.655 Unpacking gpg (2.2.27-3ubuntu2.1) ...
#14 8.698 Selecting previously unselected package pinentry-curses.
#14 8.700 Preparing to unpack .../09-pinentry-curses_1.1.1-1build2_amd64.deb ...
#14 8.704 Unpacking pinentry-curses (1.1.1-1build2) ...
#14 8.742 Selecting previously unselected package gpg-agent.
#14 8.744 Preparing to unpack .../10-gpg-agent_2.2.27-3ubuntu2.1_amd64.deb ...
#14 8.747 Unpacking gpg-agent (2.2.27-3ubuntu2.1) ...
#14 8.805 Selecting previously unselected package gpg-wks-client.
#14 8.807 Preparing to unpack .../11-gpg-wks-client_2.2.27-3ubuntu2.1_amd64.deb ...
#14 8.811 Unpacking gpg-wks-client (2.2.27-3ubuntu2.1) ...
#14 8.842 Selecting previously unselected package gpg-wks-server.
#14 8.844 Preparing to unpack .../12-gpg-wks-server_2.2.27-3ubuntu2.1_amd64.deb ...
#14 8.847 Unpacking gpg-wks-server (2.2.27-3ubuntu2.1) ...
#14 8.878 Selecting previously unselected package gpgsm.
#14 8.879 Preparing to unpack .../13-gpgsm_2.2.27-3ubuntu2.1_amd64.deb ...
#14 8.883 Unpacking gpgsm (2.2.27-3ubuntu2.1) ...
#14 8.920 Selecting previously unselected package gnupg.
#14 8.922 Preparing to unpack .../14-gnupg_2.2.27-3ubuntu2.1_all.deb ...
#14 8.925 Unpacking gnupg (2.2.27-3ubuntu2.1) ...
#14 8.969 Setting up libksba8:amd64 (1.6.0-2ubuntu0.2) ...
#14 8.980 Setting up apt-transport-https (2.4.11) ...
#14 8.988 Setting up libnpth0:amd64 (1.6-3build2) ...
#14 8.997 Setting up libassuan0:amd64 (2.5.5-1build1) ...
#14 9.009 Setting up gnupg-l10n (2.2.27-3ubuntu2.1) ...
#14 9.019 Setting up gpgconf (2.2.27-3ubuntu2.1) ...
#14 9.027 Setting up gpg (2.2.27-3ubuntu2.1) ...
#14 9.035 Setting up gnupg-utils (2.2.27-3ubuntu2.1) ...
#14 9.044 Setting up pinentry-curses (1.1.1-1build2) ...
#14 9.062 Setting up gpg-agent (2.2.27-3ubuntu2.1) ...
#14 9.546 Setting up gpgsm (2.2.27-3ubuntu2.1) ...
#14 9.555 Setting up dirmngr (2.2.27-3ubuntu2.1) ...
#14 9.693 Setting up gpg-wks-server (2.2.27-3ubuntu2.1) ...
#14 9.702 Setting up gpg-wks-client (2.2.27-3ubuntu2.1) ...
#14 9.712 Setting up gnupg (2.2.27-3ubuntu2.1) ...
#14 9.721 Processing triggers for libc-bin (2.35-0ubuntu3.6) ...
#14 10.02 Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease
#14 10.02 Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
#14 10.04 Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
#14 10.07 Get:4 https://deb.nodesource.com/node_20.x nodistro InRelease [12.1 kB]
#14 10.13 Hit:5 http://security.ubuntu.com/ubuntu jammy-security InRelease
#14 10.32 Get:6 https://deb.nodesource.com/node_20.x nodistro/main amd64 Packages [5,184 B]
#14 10.39 Fetched 17.3 kB in 0s (41.8 kB/s)
#14 10.39 Reading package lists...
#14 11.28 2024-02-15 16:45:13 - Repository configured successfully. To install Node.js, run: apt-get install nodejs -y
#14 11.32 Reading package lists...
#14 12.22 Building dependency tree...
#14 12.42 Reading state information...
#14 12.64 The following NEW packages will be installed:
#14 12.64   nodejs
#14 12.74 0 upgraded, 1 newly installed, 0 to remove and 5 not upgraded.
#14 12.74 Need to get 31.4 MB of archives.
#14 12.74 After this operation, 195 MB of additional disk space will be used.
#14 12.74 Get:1 https://deb.nodesource.com/node_20.x nodistro/main amd64 nodejs amd64 20.11.1-1nodesource1 [31.4 MB]
#14 13.17 debconf: delaying package configuration, since apt-utils is not installed
#14 13.20 Fetched 31.4 MB in 0s (78.9 MB/s)
#14 13.27 Selecting previously unselected package nodejs.
(Reading database ... 13841 files and directories currently installed.)
#14 13.28 Preparing to unpack .../nodejs_20.11.1-1nodesource1_amd64.deb ...
#14 13.28 Unpacking nodejs (20.11.1-1nodesource1) ...
#14 16.48 Setting up nodejs (20.11.1-1nodesource1) ...
#14 16.52 v20.11.1
#14 DONE 17.2s

#13 [build  3/10] WORKDIR /app
#13 sha256:afe7338fe12eb1c203cd047f172e6e8a78b78a9618d4f541cdb55a45cd46d9fb
#13 DONE 0.0s

#11 [build  4/10] COPY package.json package-lock.json /app/
#11 sha256:9d5aeea839e17522742c7bd3474a9752597055f3d4ca8b2128a61860f5a5637e
#11 DONE 0.0s

#10 [build  5/10] COPY settings.gradle.kts build.gradle.kts gradlew gradle.properties /app/
#10 sha256:c19702537887219033ca091dd8da4121c127ba27c254e86b1c81518d3087908d
#10 DONE 0.1s

#9 [build  6/10] COPY gradle /app/gradle
#9 sha256:75401c15aaa43fe43ae1dd4be9c910e0e99007fe4d1a0a565b84c69f82ff2f99
#9 DONE 0.0s

#8 [build  7/10] COPY buildSrc /app/buildSrc
#8 sha256:b48ee17955f4943e0444310920fa258999626362b6e2f5a88109c0a6ffc98976
#8 DONE 0.0s

#7 [build  8/10] RUN mkdir -p /app/src/main/jte &&   npm install &&   gradle -Dsonar.gradle.skipCompile=true --console plain --no-configuration-cache classes -x assetsPipeline
#7 sha256:fba430340d346085b06093c1fe2e71e48fd342e30cdcffe0048dca12a9e9870a
#7 7.562 npm WARN deprecated uuid@2.0.3: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
#7 11.38 
#7 11.38 added 669 packages, and audited 670 packages in 11s
#7 11.38 
#7 11.38 126 packages are looking for funding
#7 11.38   run `npm fund` for details
#7 11.38 
#7 11.38 found 0 vulnerabilities
#7 11.38 npm notice 
#7 11.38 npm notice New minor version of npm available! 10.2.4 -> 10.4.0
#7 11.38 npm notice Changelog: <https://github.com/npm/cli/releases/tag/v10.4.0>
#7 11.38 npm notice Run `npm install -g npm@10.4.0` to update!
#7 11.38 npm notice 
#7 12.00 
#7 12.00 Welcome to Gradle 8.6!
#7 12.00 
#7 12.00 Here are the highlights of this release:
#7 12.00  - Configurable encryption key for configuration cache
#7 12.00  - Build init improvements
#7 12.00  - Build authoring improvements
#7 12.00 
#7 12.00 For more details see https://docs.gradle.org/8.6/release-notes.html
#7 12.00 
#7 12.20 Starting a Gradle Daemon (subsequent builds will be faster)
#7 65.50 > Task :buildSrc:checkKotlinGradlePluginConfigurationErrors
#7 77.80 > Task :buildSrc:pluginDescriptors
#7 77.80 > Task :buildSrc:processResources NO-SOURCE
#7 87.90 > Task :buildSrc:compileKotlin
#7 87.90 > Task :buildSrc:compileJava NO-SOURCE
#7 87.90 > Task :buildSrc:compileGroovy NO-SOURCE
#7 87.90 > Task :buildSrc:classes UP-TO-DATE
#7 88.00 
#7 88.00 > Task :buildSrc:jar
#7 88.00 :jar: No valid plugin descriptors were found in META-INF/gradle-plugins
#7 137.3 
#7 137.3 > Task :checkKotlinGradlePluginConfigurationErrors
#7 137.4 
#7 137.4 > Task :generateJte
#7 137.4 Extension gg.jte.nativeimage.NativeResourcesExtension generated 0 files.
#7 137.4 Extension gg.jte.models.generator.ModelExtension generated 3 files.
#7 160.6 
#7 160.6 > Task :kspKotlin
#7 160.8 > Task :processResources NO-SOURCE
#7 163.6 > Task :compileKotlin
#7 163.6 > Task :compileJava NO-SOURCE
#7 163.6 > Task :classes UP-TO-DATE
#7 163.6 
#7 163.6 BUILD SUCCESSFUL in 2m 32s
#7 163.6 8 actionable tasks: 8 executed
#7 DONE 164.5s

I cannot see a project settings that would enable/disable this, and “DLC Teardown” apparently is running successfully:

Uploading: success after 8.875268467s.
Save DLC success after 15.220564772s.
2

I’ve had to simplify your Dockerfile to some that only does the “RUN apt-get …” line as I do not have the rest of your environment, but I have used your config.yml file.

The output of the “Build Docker image” step shows the following (in part)

#1 [internal] load build definition from Dockerfile
#1 sha256:f8c7b11bbaf4ac9b7f7c20cf2789e0d2b5f3596dfe629a120ffaad9d95569d32
#1 transferring dockerfile: 2.55kB done
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 sha256:de20a3691006fd22a4cd7f05d8d3f952b44ef5d6c817e8da7299c8c479481b15
#2 transferring context: 2B done
#2 DONE 0.0s

#3 [internal] load metadata for docker.io/library/gradle:8.6.0-jdk21
#3 sha256:5bb5d20a8447f1578d5eafee569ce9311bd890a027ed32934a8ee29c284fee6f
#3 DONE 0.0s

#5 [1/2] FROM docker.io/library/gradle:8.6.0-jdk21
#5 sha256:da7258c1d91e7a806c057ece6a581e7f02a77256458f6f3e8ebfaf7e441fe29e
#5 DONE 0.0s

#4 [2/2] RUN apt-get update -y && apt-get install --no-install-recommends -y curl && curl -fsSL https://deb.nodesource.com/setup_20.x | bash - && apt-get install -y --no-install-recommends nodejs && node -v
#4 sha256:dcdc6e129346f573baea44c7aae70ba09c2acf900992e99e1f000f4a6b5d58b0
#4 CACHED

So for my ‘Free’ account I do see caching activated for a basic step in your build process.

In your environment, the first thing to check is the output shown under the “Spin up environment” as it should include the lines “Restoring DLC” and “Restore DLC success after xx.xxxxs”

One possible issue is, just how large is your final image? The CircleCI cache is designed for small images, such as the updating of a base OS distro, rather than a full-size application environment being placed on top of an OS distro. The following published at the end of last year states the current limit is 15 GiB.

I also build application-size docker environments but for other reasons use self-hosted runners. These execute the docker workflow on systems I have defined elsewhere. This provides me with environments with the limits being my allocated storage space rather than any set by CircleCI.

3

Thank you, @rit1010!

It seems that most of the layers are under 100MB, and the final image is 298MB. Also, I can see this in the “Spin up environment” build step:

machine-agent version 1.0.70629-b1d9199 (image: "ubuntu-2204:2024.02.7", public ip: "34.207.106.106", provider: "EC2")
Restoring DLC
running command: service [docker stop]
Warning: Stopping docker.service, but it can still be activated by:
  docker.socket
  success after: 573.812055ms
running command: systemctl [stop docker.socket]
  success after: 3.216813ms
Downloading: downloaded 1.757GiB, expanded to 2.283GiB, ratio 0.77, after 23.338002339s.
running command: mount [-t ext4 -o loop,discard /tmp/.dlc.img /var/lib/docker]
  success after: 1.727353256s
running command: service [docker start]
  success after: 959.063739ms
running command: systemctl [start docker.socket]
  success after: 2.81429ms
Restore DLC success after 26.651801974s.
docker-agent version 1.0.17363-bb81d4b
Downloading docker-agent: success after 162.112569ms.

I guess the main line there is:

Downloading: downloaded 1.757GiB, expanded to 2.283GiB, ratio 0.77, after 23.338002339s.

This includes the build image as well. I will change my build config to debug how DLC is pruning too much of the cache before saving it. Thanks for sharing that page.

As soon as I got more info, I will report back here.

Best.

4

Alright, some extra information:

The “Spin up Environment” output:

machine-agent version 1.0.70629-b1d9199 (image: "ubuntu-2204:2024.02.7", public ip: "44.204.123.155", provider: "EC2")
Restoring DLC
running command: service [docker stop]
Warning: Stopping docker.service, but it can still be activated by:
  docker.socket
  success after: 776.780374ms
running command: systemctl [stop docker.socket]
  success after: 4.475175ms
Downloading: downloaded 1.576GiB, expanded to 1.824GiB, ratio 0.86, after 28.179848961s.
running command: mount [-t ext4 -o loop,discard /tmp/.dlc.img /var/lib/docker]
  success after: 65.930057ms
running command: service [docker start]
  success after: 1.369864711s
running command: systemctl [start docker.socket]
  success after: 3.748181ms
Restore DLC success after 30.451412879s.
docker-agent version 1.0.17363-bb81d4b
Downloading docker-agent: success after 284.254101ms.

Then, there is a step to list all the images:

- run:
    name: List docker images
    command: |
      docker image ls --all --digests

It outputs:

REPOSITORY                                   TAG       DIGEST                                                                    IMAGE ID       CREATED             SIZE
*******                                      latest    <none>                                                                    bb3f5714fb72   1 second ago        296MB
<none>                                       <none>    <none>                                                                    ebe8243d19ec   50 minutes ago      296MB
<none>                                       <none>    <none>                                                                    4b043bebc885   About an hour ago   296MB
<none>                                       <none>    <none>                                                                    c29b03fbfe62   2 hours ago         296MB
<none>                                       <none>    <none>                                                                    b1515d914880   2 hours ago         296MB
<none>                                       <none>    <none>                                                                    c68acb2d2114   6 hours ago         296MB
<none>                                       <none>    <none>                                                                    ae7e2eda8d14   6 hours ago         296MB
<none>                                       <none>    <none>                                                                    4de63241a53a   14 hours ago        296MB
<none>                                       <none>    <none>                                                                    07658aa851fd   14 hours ago        296MB
<none>                                       <none>    <none>                                                                    51ea7a221a17   15 hours ago        296MB
<none>                                       <none>    <none>                                                                    e4eaa71c23c8   23 hours ago        296MB
<none>                                       <none>    <none>                                                                    83e160f8bd7f   23 hours ago        296MB
<none>                                       <none>    <none>                                                                    14f3cf3d07db   29 hours ago        296MB
<none>                                       <none>    <none>                                                                    d26c17f80b7a   29 hours ago        296MB
<none>                                       <none>    <none>                                                                    1bfbf6c3b7a4   2 days ago          296MB
<none>                                       <none>    <none>                                                                    8357de47b9e4   2 days ago          296MB
<none>                                       <none>    <none>                                                                    cbdd42ea11ce   2 days ago          296MB
<none>                                       <none>    <none>                                                                    645a268f49a9   2 days ago          296MB
<none>                                       <none>    <none>                                                                    16086f91cead   2 days ago          296MB
public.ecr.aws/eks-distro/kubernetes/pause   3.6       sha256:c38d6dd4c0a53ccbf200f85a1057f273040e4ad4a29171e1cdcc0e4414f1b12c   3c69a9ca2c95   4 months ago        6.62MB
cimg/base                                    stable    sha256:35e5e29930ab565475a4f2aa9b4124998ed67dbc7b0e2dd5f420a4189d08d0d2   1db3a73cfdc2   18 months ago       1.15GB

So, I can see there are a few images there that are at least 2 days old. I changed my Dockerfile to remove the mkdir command and instead copy a .keep file that is never changed, but that didn’t help either.

I’m not sure what would be a good next step here.

5

Beyond resetting the cache I have no ideas to put forward. This can be done by

  • go into the CircleCI web GUI
  • Select the project setting for the project that is having issues
  • Select the Docker Layer Caching page
  • Delete Cache Contents

The logic being is that a reset is always a good start and the fact that if you sum up the images you have listed it is just over 13GB and CircleCI has not documented how they keep the cache below 15GB - from your out put it is clear that it is not based on creation date.

6

Thanks!

Deleting the cache didn’t work. I did add some extra debugging steps here:

- run:
    name: List docker cache
    command: |
      docker system df --verbose

And it outputs:

Build cache usage: 2.247GB

CACHE ID       CACHE TYPE     SIZE      CREATED              LAST USED                USAGE     SHARED
fhw1v8y9pc0c   regular        0B        53 minutes ago       53 minutes ago           1         true
833nt52wjwkn   regular        0B        53 minutes ago       53 minutes ago           1         true
w62bm0dk3dqr   regular        0B        53 minutes ago       53 minutes ago           1         true
z8dc7m6yz7b5   regular        0B        53 minutes ago       53 minutes ago           1         true
qx34edsem2m5   regular        93.8MB    50 minutes ago       50 minutes ago           1         true
0j7b5k7vir61   regular        4.71kB    50 minutes ago       50 minutes ago           1         true
x5vt0960jzv0   regular        0B        53 minutes ago       50 minutes ago           1         true
oq09yj2nbttq   regular        93.8MB    42 minutes ago       42 minutes ago           1         true
m3i9lm89ux6v   regular        4.71kB    42 minutes ago       42 minutes ago           1         true
0qpnc5tv4xah   regular        4.71kB    33 minutes ago       33 minutes ago           1         true
kl2ghcyfjx1n   regular        93.8MB    33 minutes ago       33 minutes ago           1         true
t0w2p2jbmjim   regular        0B        3 minutes ago        3 minutes ago            1         false
vf5hth55z02x   regular        0B        3 minutes ago        3 minutes ago            1         false
ig8ffu5b42yb   regular        0B        3 minutes ago        3 minutes ago            1         false
y1wlz1d6p3rq   regular        0B        3 minutes ago        3 minutes ago            1         false
nqctrhycoihi   regular        0B        3 minutes ago        3 minutes ago            1         false
j6xnyi22p8s9   regular        0B        3 minutes ago        3 minutes ago            1         false
pipl1odfzfc2   regular        0B        3 minutes ago        3 minutes ago            1         false
8to49fkt2omc   regular        0B        3 minutes ago        3 minutes ago            1         false
xwo097end8gz   source.local   2.12kB    3 minutes ago        Less than a second ago   1         false
q03ynlkkms1k   regular        293kB     3 minutes ago        Less than a second ago   1         false
nbj147s02rzr   regular        22.3kB    3 minutes ago        Less than a second ago   1         false
s1tplq9esq6d   regular        0B        3 minutes ago        Less than a second ago   1         false
iv2es7m5uyy5   regular        0B        3 minutes ago        Less than a second ago   1         false
r7ub1thlqhdm   source.local   18.7MB    3 minutes ago        Less than a second ago   1         false
th3yej0zixhl   regular        43.7kB    3 minutes ago        Less than a second ago   1         false
h5v3s2o2ev7u   regular        11.4kB    3 minutes ago        Less than a second ago   1         false
iass0c6bggyw   regular        1.86GB    About a minute ago   Less than a second ago   1         false
ur9ngnwamjpo   regular        18.7MB    About a minute ago   Less than a second ago   1         false
lcxc3qsdihtp   regular        130MB     About a minute ago   Less than a second ago   1         false
mis6ec6114zz   source.local   237B      3 minutes ago        Less than a second ago   1         false
s290ukm7on4w   regular        214MB     3 minutes ago        Less than a second ago   1         false
sdxzbiuhvddg   regular        4.71kB    1 second ago         Less than a second ago   1         true
m2oba5cfih3t   regular        0B        3 minutes ago        Less than a second ago   1         false
wtg85fqofrhd   regular        93.8MB    1 second ago         Less than a second ago   1         true
CircleCI received exit code 0

Test Docker image

Now, when I look at the “DLC Teardown” output, it shows:

Saving DLC
running: remove running containers
  success after: 297.200685ms
running: prune containers
  success after: 365.237µs
running: prune volumes
  success after: 474.198µs
running: prune networks
  success after: 410.036µs
running: prune images
  success after: 3.861573555s
deleted 2 images: 
- sha256:3c69a9ca2c95e52830d24b1ba41368bc7aba41f7144cdd0f02318ad6d596df19
- sha256:1db3a73cfdc23c86c60c414549a49f57a17eb9b440e3effe2c519fd82f969895
deleted 22 build caches: 
- r7ub1thlqhdmj7mlm2g640m2i
- mis6ec6114zz7ezj1cjrji5ot
- xwo097end8gzu27owpliyujtz
- lcxc3qsdihtpmyc6pc2cm103e
- ur9ngnwamjpo668dqri3d350k
- iass0c6bggywrqytssrz24qo6
- s1tplq9esq6d2w2abnhrf5a4i
- h5v3s2o2ev7urxpgbnwc2inpp
- th3yej0zixhlct9o5d7ni6rtl
- nbj147s02rzri87q2ojc8kwaz
- q03ynlkkms1kvqu1s5wty63yg
- m2oba5cfih3tf6vhn2ioqlnie
- s290ukm7on4wol9ljj37690d5
- iv2es7m5uyy554ffit0vk85pm
- 8to49fkt2omc36oqodfq6u239
- pipl1odfzfc2onxkzrvqhvv0b
- j6xnyi22p8s9012tdapbxnh4f
- nqctrhycoihibnqx7bilz21mw
- y1wlz1d6p3rqgsuwehe00l8po
- ig8ffu5b42ybvpg4alips5g18
- vf5hth55z02xrxie0btjswr7c
- t0w2p2jbmjiman2uhce9i87jv
running command: service [docker stop]
Warning: Stopping docker.service, but it can still be activated by:
  docker.socket
  success after: 419.192206ms
running command: systemctl [stop docker.socket]
  success after: 3.580059ms
running command: fstrim [-v /var/lib/docker]
/var/lib/docker: 97.8 GiB (105034686464 bytes) trimmed
  success after: 31.452285ms
running command: umount [/var/lib/docker]
  success after: 93.721429ms
Uploading: success after 3.693817503s.
Save DLC success after 10.299251834s.

So, these were deleted (all new cache entries) before caching:

t0w2p2jbmjim   regular        0B        3 minutes ago        3 minutes ago            1         false
vf5hth55z02x   regular        0B        3 minutes ago        3 minutes ago            1         false
ig8ffu5b42yb   regular        0B        3 minutes ago        3 minutes ago            1         false
y1wlz1d6p3rq   regular        0B        3 minutes ago        3 minutes ago            1         false
nqctrhycoihi   regular        0B        3 minutes ago        3 minutes ago            1         false
j6xnyi22p8s9   regular        0B        3 minutes ago        3 minutes ago            1         false
pipl1odfzfc2   regular        0B        3 minutes ago        3 minutes ago            1         false
8to49fkt2omc   regular        0B        3 minutes ago        3 minutes ago            1         false
xwo097end8gz   source.local   2.12kB    3 minutes ago        Less than a second ago   1         false
q03ynlkkms1k   regular        293kB     3 minutes ago        Less than a second ago   1         false
nbj147s02rzr   regular        22.3kB    3 minutes ago        Less than a second ago   1         false
s1tplq9esq6d   regular        0B        3 minutes ago        Less than a second ago   1         false
iv2es7m5uyy5   regular        0B        3 minutes ago        Less than a second ago   1         false
r7ub1thlqhdm   source.local   18.7MB    3 minutes ago        Less than a second ago   1         false
th3yej0zixhl   regular        43.7kB    3 minutes ago        Less than a second ago   1         false
h5v3s2o2ev7u   regular        11.4kB    3 minutes ago        Less than a second ago   1         false
iass0c6bggyw   regular        1.86GB    About a minute ago   Less than a second ago   1         false
ur9ngnwamjpo   regular        18.7MB    About a minute ago   Less than a second ago   1         false
lcxc3qsdihtp   regular        130MB     About a minute ago   Less than a second ago   1         false
mis6ec6114zz   source.local   237B      3 minutes ago        Less than a second ago   1         false
s290ukm7on4w   regular        214MB     3 minutes ago        Less than a second ago   1         false
m2oba5cfih3t   regular        0B        3 minutes ago        Less than a second ago   1         false

Apart from this entry:

iass0c6bggyw   regular        1.86GB    About a minute ago   Less than a second ago   1         false

None of these are large or approaching the cache size limit. For the sake of completeness, these are new and were NOT deleted:

sdxzbiuhvddg   regular        4.71kB    1 second ago         Less than a second ago   1         true
wtg85fqofrhd   regular        93.8MB    1 second ago         Less than a second ago   1         true

Should I open a support ticket? Is that even possible for free plans?

7

The basic working test I did above was on a free account so the cache is available.

A support ticket would seem the only option, but it may take a while on a free account.

Your latest DLC Teardown log indicates that CircleCI’s automated processes are deleting all the layers, but there is no public info on how that is decided.

8

Alright, it seems I was able to make some progress. I followed this article to debug caching issues:

And I noticed that a few directories were not in my .dockerignore file. For example, I added .git. Those affected only the last layer (the image itself), but it is still good to sanitize this aspect.

I also added the following argument to my docker image build command:

--build-arg BUILDKIT_INLINE_CACHE=1

Following the docs from here: docker build | Docker Docs

I don’t fully understand how this affected the cache creation, but now re-running the job to build the docker image, with no changes at all, runs instantly (since it uses the cache).

When there are changes to the files, though, even if they don’t affect the previous layers, the build uses no cache at all. :frowning: