How to get started with parallel builds


#1

I’m working on trying to convert our existing setup over to Beta 2.0. We currently run 8x parallelization in v1, splitting our RSpec suite across all of the containers. We also have three Node projects in the repo, and those test phases are currently added to the end of the parallel build.

I’m interested in trying out 2.0 because it’s my understanding that you can more directly control the individual containers. I’ve done a lot of reading in Discuss, but I can’t find specific resources that talk about different build/setup/cache strategies that are variable based on container.

For instance, I’m interested in making it so I can dedicate 6 containers to RSpec. On those containers, i want to skip all npm install commands, etc. I then want to dedicate 2 other containers for node related tests, where I’d skip all bundle install commands, postgres setup, etc.


#2

Currently that is achieved via if statements and checking the $CIRCLE_NODE_INDEX, much the same as 1.0. The major difference is the containers will not wait for each other; there are no blocking steps. Just specify what to run and where to run them, and they will run. You can absolutely run rspec in containers 0-5 and something else in 6-7.


#3

Is there a mechanism to do different setup per container - i.e. conditionals on the containerInfo?


#4

No, you would need a different section for that, or a different circle.yml entirely. Ultimately you will be able to run different sections of the config via the API; that might be helpful for your use case.


#5

I made a tool to split the test suite into containers: https://github.com/leonid-shevtsov/split_tests

It doesn’t support heterogenous containers, but you definitely could split rspec tests into 6 containers and npm into the other 2.

Another case when this is useful (in theory) is browser integration specs. They also require additional setup costs, such as booting the app server.


#6

This is awesome. It seems like I’ve been looking for it my entire life :P. One feedback is that I think most of the times people look for the naieve split by file count first; on a separate note, I wonder if more documentation on the using CircleCI’s API part is worthy, because I’m completely lost on why that part makes sense (or why the API is needed). Just my 5c. Again, awesome job!


#7

This tool looks amazing. Is it possible to use it with Cucumber tests in the same way?


#8

This is fantastic! But also dismaying! Because we built a similar tool and it got lost in the documentation process!

You can check out the CLI we built here: Managing Parallelism with the 2.0 Tests CLI.

Everyone: now you have a couple options, and thanks again @leonid-shevtsov for your work!


#9