Running custom bash (test/validation) scripts in parallel

Build Environment
, , , ,
1

I’m trying to build Playframework using CircleCI and see if it will behaves more stable than our current CI. Play is a Java/Scala project, using Scala 2.11 and SBT 0.13.11 (which I’ve manually installed as suggested at the docs).

My doubt is about how to proper use the parallel feature to execute validation (not only tests, but code inspections too) steps in parallel. Here is what I tried until now:

  1. Adjust the parallelism to use 4x (Project Settings > Parallelism)
  2. Write my circle.yml file to add the parallel: true flag to each command that can execute in parallel
  3. Fire a build

Then, instead of executing the validation steps in parallel, Circle executed all the commands 4x (four builds executing all the steps). And then I decided to try like this:

  1. Adjust the parallelism to use 1x (Project Settings > Parallelism) (I was wild guessing that the parallel configuration would be inferred from circle.yml).
  2. Write my circle.yml file to add the parallel: true flag to each command that can execute in parallel
  3. Fire another build

Still no game. All the commands are executed sequentially, but this time once.

What I was expecting was to have the test/validation commands to be executed in parallel (4x).

ps.: I’m currently configuring the build for my own fork. After a green/parallel build and when everything works as expected, I can go for the official repository.

2

Unfortunately it’s not that easy. You have to specify the node index you want to run each script.

test:
  override:
    - case $CIRCLE_NODE_INDEX in 0) rake spec ;; 1) npm test ;; esac:
        parallel: true

You can see more advanced ways to handle this in the “Using environment variables” section of the parallelization docs. https://circleci.com/docs/parallel-manual-setup/

1 Like