CircleCI, parallel builds and Code Climate

Build Environment
,
1

EDIT: It is currently not possible to merge multiple coverage results generated by Code Climate. In a parallel build where more than one container reports the coverage to Code Climate, only the very first payload received for a certain build will be interpreted. Additional coverage reports sent in will be discarded. The only way to work around this is to use an alternative coverage solution.

To make sure that the coverage is reported correctly to Code Climate in your parallel builds, you’ll need to make sure that, after the coverage run has finished, the results are concentrated on a single machine (for example container 0), merged together and reported from there.

We suggest using the SSH between build containers feature to transfer the coverage info between machines. For example, the following will copy the ~/SampleProject/coverage.xml file to the container 0 from containers 1 to 5:

test:
  post:
    - "for i in $(seq 1 5) ; do scp node$i:~/SampleProject/coverage.xml coverage-node$i.xml ; done"

You can use Code Climate CLI to merge the coverage into a single file before reporting it back to Code Climate.

2 Likes
2

This might be a little more dynamic:

test:
  post:
    - "for i in $(seq 1 $CIRCLE_NODE_TOTAL) ; do scp node$i:~/SampleProject/coverage.xml coverage-node$i.xml ; done"
1 Like
3

Totally. Thanks!

4

Any pointers on how to do this? With Rails/Rcov its generating /coverage/.resultset.json

5

Sorry for the delay here. I got in touch with Code Climate about this, and looks like there is no way to merge multiple result files with the Code Climate CLI. I am very sorry for all the confusion this discussion has created so far.

The only thing we can offer right now is to use an alternative coverage solution.

6

For anyone willing to do some work, I believe that Indiegogo’s Simplecov merge solution should be adaptable to this problem. Indiegogo’s solution has 2 key steps:

  1. Copy test results into an artifacts folder and ensure the last node merges them.
  2. Parse the partial results and use Simplecov’s merge_resultset method(s) to handle the merge logic.

Code Climate’s ruby test reporter is based on Simplecov. In the code climate reporter specs, the formatter refers to the result as simplecov_result, so I think it’s likely that Code Climate reporter results are Simplecov compatible.

7

In the end I’ve used a modified version of: https://gist.github.com/evanwhalen/f74879e0549b67eb17bb

8

That gist has one problem where it doesn’t wait until all parallel builds are done. @matthewford how did you set it up so that you know all other containers are done with their coverage reports?

9

@drazisil how do you ensure that this would

a) only run on the primary node (node 0)
b) only run AFTER all other nodes have completed tests ?

10

@akshah123 @urkle The post commands in the test section will be executed on node0 only, once all the nodes are done with the override commands.