Maven dependency caching for dependencies fetched from test step for non-maven based project

[daluu]

Thought it may be useful to share this info to the community.

Scenario:

We have a Java-based packaged binary (JAR file) and this gets pulled via wget on every build as external tool/dependency in dependency step. This binary gets run in the test step and at runtime, it pulls in it’s needed dependencies (Java jars) into $HOME/.m2 (looks to be maven repo). But I notice that during each build (excluding 1st build), it still pulls dependencies rather than it being cached between CircleCI builds. Per your docs, it sounded like maven local repos are automatically cached. And just in case, I added $HOME/.m2/ to the cache_directories section of dependencies section. With or without the specific cache_directories entry, every build still pulls the dependencies and not read from cache. What am I doing incorrectly? Or is the cache only cached upon successful builds? Because my past builds have been failing due to known failures.

(NOTE: I haven’t since followed up on this yet as work on that repo has been shelved for the time being)

Response from Alexey at CircleCI:

The dependency cache is saved right after the dependencies step on each build—so if Maven actually downloads those files during the test step, it makes sense that they didn’t get cached. We still haven’t figured out how to make Maven cache all those dependencies, but one suggestion would be to do the following in your circle.yml:

dependencies:
override:
- mvn --fail-never dependency:go-offline || true

Could you please check if this actually pulls all the dependencies and stores them in ~/.m2?

[alexey]

This command should indeed download all the necessary files into ~/.m2 with the goal of creating a complete local repository on the machine where Maven is running. This StackOverflow answer has some more details about it.

[nohaapav]

Does this work ? I used similar approach but maven is still not using local storage.

This is my circle.yml

machine:
services:
- docker
java:
version: oraclejdk8

dependencies:
cache_directories:
- "~/.docker"
override:
- mvn --fail-never dependency:go-offline || true
- docker info

test:
override:
- mvn install -s settings.xml
- docker build -t nohaapav/airconomist-user-service .
post:
- mkdir -p $CIRCLE_TEST_REPORTS/junit/
- find . -type f -regex “.*/target/surefire-reports/.*xml” -exec cp {} $CIRCLE_TEST_REPORTS/junit/ ;

deployment:
hub:
branch: master
commands:
- docker login -e $DOCKER_EMAIL -u $DOCKER_USER -p $DOCKER_PASSWORD
- docker push nohaapav/airconomist-user-service

[rtyer]

There are outstanding issues with maven resolving dependencies as you would like. https://issues.apache.org/jira/browse/MDEP-82 being the pertinent issue (it has been around for quite a while, so I don’t expect we’ll ever see movement on it). It would really be idea if we could control when the cache is saved with a cache directive.

[j256]

I’ve found that overriding the dependency command and running the following worked better.

- mvn -U dependency:go-offline compile

I needed the -U because I was doing testing and needed deployed SNAPSHOTs. Adding compile downloaded the bulk of the other missing jars. The only jars that were then downloaded when I do mvn -U test are the ones associated with surefire – duh. Sure this compiles twice but that is relatively fast compared to anything else.

You can compare the downloads from the mvn -U test portion of this older build compared to this new build.

[rcrmtest]

Pls. help me out…

[rcrmtest]

working_directory: ~/__is it git repo ????__