Parallel test runs

Build Environment
1

Are services like database, memcached… shared among test workers when running in parallel mode? What kind of isolation should I expect between test nodes?

Cheers,
Tommaso

2

:+1: A better explanation what to expect from parallel mode would be nice!

Some parts I think that could also benefit from a more detailed explanation:

  • how/where parallel and non-parallel commands are executed
  • are folders (like $CIRCLE_ARTIFACTS) shared
  • is there any kind of synchonization (example: 3 commands, first and last are parallel. when is the second one started? after ALL nodes finished the first one? when then last one?)
5 Likes
3

Hi there,

Are services like database, memcached… shared among test workers when running in parallel mode? What kind of isolation should I expect between test nodes?

The services are not shared. Each container in a parallel build is completely isolated and runs its own services.

how/where parallel and non-parallel commands are executed

The only commands that are not executed in parallel are those in the test phase, which only run on container #0 unless explicitly specified as parallel using the parallel: modifier in a circle.yml file. Inferred test commands are also run in parallel.

are folders (like $CIRCLE_ARTIFACTS) shared

This folder is not shared. Files put into this directory are collected from each container at the end of the build and uploaded to our artifacts service.

is there any kind of synchonization (example: 3 commands, first and last are parallel. when is the second one started? after ALL nodes finished the first one? when then last one?)

Commands are run in lockstep across all containers. In your example, the second command would run after the first one has finished on all containers. The last command would start after the second command has finished on its container, even though the second command is not being run on all containers.

I hope this clarifies things for you! If this answers your questions, please mark this thread as solved by clicking on the checkmark below.

Cheers,
Frank

5 Likes
4

Can you clarify https://circleci.com/docs/parallel-manual-setup. Specifically things like

    - bundle exec rspec --format RspecJunitFormatter --out $CIRCLE_TEST_REPORTS/rspec.xml:
        parallel: true
        files:
          - spec/unit/sample.rb   # can be a direct path to file
          - spec/**/*.rb          # or a glob (ruby globs)

If I’m using a very custom command but generating my own junit results, can I still do files glob automatic balancing? What format do my junit results and filenames need to take?

Particularly, I’m looking at https://circleci.com/docs/parallel-manual-setup#manual-balancing but need something smarter since some of my custom subtests take 10 minutes and some take 1 minutes so I want to balance that by how long they actually took to execute.

5

You can use file globs and automatic test balancing with rspec so long as

  • the RspecJunitFormatter is used as shown in the docs
  • the custom command you are using accepts multiple test files as arguments (e.g. bundle exec rspec file1.rb file2.rb file3.rb)

This will balance the test files on each container by execution time so that each container takes a similar amount of time to run your tests.

1 Like
6

Thanks for the response. I worry I wasn’t very clear. See https://circleci.com/docs/parallel-manual-setup/#manual-balancing. I want to do exactly that, and something as customizable and generic as that, while splitting up by how long each individual item took on the past run.

Does this exist today, or is it something I will script myself?

7