Recommended dependencies caching for a Gradle setup is wrong

gradle
cache
circle.yml

#1

Your suggested Gradle setup contains

- restore_cache:
    keys:
    - v1-dependencies-{{ checksum "build.gradle" }}
    - v1-dependencies-
- run: gradle dependencies
- save_cache:
    paths:
      - ~/.gradle
    key: v1-dependencies-{{ checksum "build.gradle" }}

This is not too helpful as it is basically a no-op.
gradle dependencies resolves and displays the dependency tree, but it does not download the actual artifacts, so there is nothing to cache and it just wastes build time.

You either need to save the cache after the actual build was run or use a different task. To my knowledge there is no “download everything” task, just a “use only cached stuff or fail” commandline switch.

But such a task can very easily be simulated by adding the following to the build script, so this should maybe be recommended for Gradle builds:

allprojects {
    task downloadDependencies() {
        description 'Download all dependencies to the Gradle cache'
        doLast {
            configurations.findAll { it.canBeResolved }.files
        }
    }
}

for a single-project Gradle build it can be simplified to

task downloadDependencies() {
    description 'Download all dependencies to the Gradle cache'
    doLast {
        configurations.findAll { it.canBeResolved }.files
    }
}

#2

A simple solution may be just to save_cache after your build rather than having an explicit download step.

- restore_cache:
    keys:
      - gradle-{{ checksum "build.gradle" }}
- run: gradle build
- save_cache:
     paths:
        - ~/.gradle
     key: gradle-{{ checksum "build.gradle" }}

dependencies will only get cached on a successful build, so builds might be slower when dependencies are changing, but it should speed up most standard cases…


#3

Actually you can add when: always to the save_cache step to have it always run, no matter whether the build was successful or not.

The problem is, that you then actually cache all Gradle metadata and that can even make your builds slower as all the files in the ~/.gradle folder need to be packed and uploaded and on the next build downloaded and unpacked.

Actually the most efficient and productive way is to add a downloadDependencies task as shown above, additionally enable the Gradle build cache by having org.gradle.caching = true in the gradle.properties file and then having a CircleCI config with three different caches with different cache keys for Gradle distribution, dependencies and build cache like:

- checkout

# restore saved caches
- restore_cache:
    keys:
      - gradle-wrapper-{{ checksum "gradle/wrapper/gradle-wrapper.properties" }}
  
- restore_cache:
    keys:
      # for multi-project builds where dependencies are not defined in one central file,
      # an additional run step before is necessary that concatenates affected files
      # into one file that can be checksummed
      - gradle-cache-{{ checksum "build.gradle" }}
      - gradle-cache

- restore_cache:
    keys:
      - gradle-build-caches-{{ .Revision }}
      - gradle-build-caches

- run:
    name: Restoring Gradle Build Caches
    command: |
      [ -d ~/gradle-build-caches ] &&
        [ -n "$(ls -A ~/gradle-build-caches)" ] &&
        rm -rf ~/.gradle/caches/build-cache-* &&
        mv ~/gradle-build-caches/* ~/.gradle/caches/ || true

# download and cache dependencies and Gradle
- run:
    name: Downloading Dependencies
    command: ./gradlew --max-workers 2 downloadDependencies

- save_cache:
    paths:
      - ~/.gradle/wrapper/
    key: gradle-wrapper-{{ checksum "gradle/wrapper/gradle-wrapper.properties" }}

- save_cache:
    paths:
      - ~/.gradle/caches/
    key: gradle-cache-{{ checksum "build.gradle" }}

# build everything needed for publication
- run:
    name: Building Project
    command: ./gradlew --max-workers 2 --continue build

# cache gradle build caches
- run:
    name: Collecting Gradle Build Caches
    command: |
      mkdir -p ~/gradle-build-caches
      [ -d ~/.gradle/caches ] &&
        [ -n "$(ls -Ad ~/.gradle/caches/build-cache-* 2>/dev/null)" ] &&
        mv ~/.gradle/caches/build-cache-* ~/gradle-build-caches || true
    when: always

- save_cache:
    paths:
      - ~/gradle-build-caches
    key: gradle-build-caches-{{ .Revision }}
    when: always

#4

I’m working with cordova/ionic. I’m not sure that there is an easy way to alter the build process. I haven’t seen much in terms of an adverse impact, but I do see your point about all the meta data.

I got a little more surgical and updated my save_cache paths to ~/.gradle/caches and ~/.gradle/wrapper.

I notice you’re checking the gradle-wrapper properties to invalidate the wrapper. Is that just to wipe out the extra content from the build when the required gradle version changes? In the case an incorrect version of gradle was cached a correct version of gradle would be downloaded, correct?


#5

Yes and no.
If the wrong wrapper version is cached, the correct is simply downloaded anyway, Gradle usually does the right things.
But the point is performance and also space optimization.

Performance:
If you change the Gradle version, but do not have the file with the Gradle version in the cache key, you will always download the old cache with the old Gradle version, unpack the old Gradle version and then download the new Gradle version, if the cache instead has the Gradle version as part of the cache key, for the first build the old cache will be downloaded, then the new Gradle version will be downloaded and a new cache with the new cache key is saved.
Actually while I write this I realize that the second key - gradle-wrapper is non-sense and will cause to have all Gradle versions ever used in the current cache, again a waste of space, download time and unpack time, that line should be removed.

Space:
If you have one cache with the caches and the downloaded Gradle distribution, then on each change of a dependency (or actually on each change of build.gradle, I have a separate dependencies.gradle only for the dependencies and for all projects in the 3-project-build for logical separation and also change minimisation), you will include a full Gradle distribution in your cache while this is totally unnecessary waste of space on the CircleCI data storage.


#6

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.