Testing and Preparing Image in Parallel

Build Environment
,
1

One of our ClojureScript builds takes a long time, over 20 min. Significant periods of time are:

  • Getting dependencies (~1 min.)
    • boot deps
    • Deps are cached, but boot is slow
  • Migrating DBs (~1 min.)
    • Not much we can do about this, I think
  • Testing (~4 min.)
    • boot test-all
    • Actually pretty good as far as testing goes
  • Uberjar (~5 min. 30 s)
    • boot uberjar
  • Build and publish image to Quay.io (~5 min.)

AFAICS we don’t have more to gain by using caching.

Would it be reasonable to use parallelism to improve build times?

  • Separate tests and uberjar/building image
  • Publish image to Quay.io if tests pass

If so, is there an example of such dependency between parallel tasks that I can refer to?

This is CircleCI 1.0. I’m not familiar enough with CircleCI 2.0 to know whether it might offer some different optimizations.

2

Parallelism will only help with lowering the overall test time by splitting tests across containers.

Do you have a link to a build? I am curious why making the uberjar takes such a long time.

3

I looked at some of our older builds from our frontend (public) which also uses Clojurescript. It looks like we were averaging about 15 minutes https://circleci.com/gh/circleci/frontend/1401 on 1.0 and it also took about 6 minutes to compile our app.

Our more recent builds are pretty complex and use both 2.0 and workflows.

https://circleci.com/workflow-run/038b01bb-9164-4962-aeea-920759688ba8

However, we were able to get our build times down to about 5 minutes total.

I think the best way to optimize this going forward for you would be to switch to 2.0 at the least, and then continue to tinker with workflows. Using our app as an example is a great place to start: https://github.com/circleci/frontend/blob/master/.circleci/config.yml

4

I’ll take a look at CircleCI 2.0, thanks.

I haven’t investigated why boot uberjar takes so long. I’ll see if I can get more information.

5

I think thats actually “normal” for 1.0 based on the previous build history of our own app where the compilation step took 6 minutes. JS is single threaded so I am not sure how much optimization you can get :confused: