Cross-posted on Stack Overflow: https://stackoverflow.com/questions/55282629/sbt-always-recompiles-full-project-in-ci-even-with-caching
I can’t seem to find a way to compile sources using scala/sbt in one workflow step and avoid full project recompilation on the next step.
I’ve looked at posts like How to cache SBT incremental compilation and it is not relevant to my setup, or at least not solving it.
My approach is basically this:
- attach workspace
/home/circleci/myorg - checkout code (to
/home/circleci/myorg/myproj) - compile the project (all compilation artifacts should reside at or below the git/checkout directory)
- persist
myorg/myproj,~/.sbt,~/.ivy2/cacheto the workspace
In the next workflow step (job):
- Restore workspace
- Move
.sbtand.ivy2/cacheback to the/home/circlecidir from the workspace - run
sbt test
However sbt test recompiles the full project every time. I am unable to determine why that’s the case. The workspace with all source code and resulting compiled .class files should all still exist in the workspace; nothing should appear to it to have changed.
Relevant circleci config:
---
version: 2
jobs:
# compile and cache compilation
test-compile:
working_directory: /home/circleci/myteam/myproj
docker:
- image: myorg/teika-myproj-base:sbt-1.2.8
steps:
# the directory to be persisted (cached/restored) to the next step
- attach_workspace:
at: /home/circleci/myteam
# git pull to /home/circleci/myteam/myproj
- checkout
- restore_cache:
# look for a pre-existing set of ~/.ivy2/cache, ~/.sbt dirs
# from a prior build
keys:
- sbt-artifacts-{{ checksum "project/build.properties"}}-{{ checksum "build.sbt" }}-{{ checksum "project/Dependencies.scala" }}-{{ checksum "project/plugins.sbt" }}-{{ .Branch }}
- restore_cache:
# look for pre-existing set of 'target' dirs from a prior build
keys:
- build-{{ checksum "project/build.properties"}}-{{ checksum "build.sbt" }}-{{ checksum "project/Dependencies.scala" }}-{{ checksum "project/plugins.sbt" }}-{{ .Branch }}
- run:
# the compile step
working_directory: /home/circleci/myteam/myproj
command: sbt test:compile
# per: https://www.scala-sbt.org/1.0/docs/Travis-CI-with-sbt.html
# Cleanup the cached directories to avoid unnecessary cache updates
- run:
working_directory: /home/circleci
command: |
rm -rf /home/circleci/.ivy2/.sbt.ivy.lock
find /home/circleci/.ivy2/cache -name "ivydata-*.properties" -print -delete
find /home/circleci/.sbt -name "*.lock" -print -delete
- save_cache:
# cache ~/.ivy2/cache and ~/.sbt for subsequent builds
key: sbt-artifacts-{{ checksum "project/build.properties"}}-{{ checksum "build.sbt" }}-{{ checksum "project/Dependencies.scala" }}-{{ checksum "project/plugins.sbt" }}-{{ .Branch }}-{{ .Revision }}
paths:
- /home/circleci/.ivy2/cache
- /home/circleci/.sbt
- save_cache:
# cache the `target` dirs for subsequenet builds
key: build-{{ checksum "project/build.properties"}}-{{ checksum "build.sbt" }}-{{ checksum "project/Dependencies.scala" }}-{{ checksum "project/plugins.sbt" }}-{{ .Branch }}-{{ .Revision }}
paths:
- /home/circleci/myteam/myproj/target
- /home/circleci/myteam/myproj/project/target
- /home/circleci/myteam/myproj/project/project/target
# in circle, a 'workflow' undergoes several jobs, this first one
# is 'compile', the next will run the tests (see next 'job' section
# 'test-run' below).
# 'persist to workspace' takes any files from this job and ensures
# they 'come with' the workspace to the next job in the workflow
- persist_to_workspace:
root: /home/circleci/myteam
# bring the git checkout, including all target dirs
paths:
- myproj
- persist_to_workspace:
root: /home/circleci
# bring the big stuff
paths:
- .ivy2/cache
- .sbt
# actually runs the tests compiled in the previous job
test-run:
environment:
SBT_OPTS: -XX:+UseConcMarkSweepGC -XX:+UnlockDiagnosticVMOptions -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -Duser.timezone=Etc/UTC -Duser.language=en -Duser.country=US
docker:
# run tests in the same image as before, but technically
# a different instance
- image: myorg/teika-myproj-base:sbt-1.2.8
steps:
# bring over all files 'persist_to_workspace' in the last job
- attach_workspace:
at: /home/circleci/myteam
# restore ~/.sbt and ~/.ivy2/cache via `mv` from the workspace
# back to the home dir
- run:
working_directory: /home/circleci/myteam
command: |
[[ ! -d /home/circleci/.ivy2 ]] && mkdir /home/circleci/.ivy2
for d in .ivy2/cache .sbt; do
[[ -d "/home/circleci/$d" ]] && rm -rf "/home/circleci/$d"
if [ -d "$d" ]; then
mv -v "$d" "/home/circleci/$d"
else
echo "$d does not exist" >&2
ls -la . >&2
exit 1
fi
done
- run:
# run the tests, already compiled
# note: recompiles everything every time!
working_directory: /home/circleci/myteam/myproj
command: sbt test
no_output_timeout: 3900s
workflows:
version: 2
build-and-test:
jobs:
- test-compile
- test-run:
requires:
- test-compile