I am using Danger Ruby to automatically report test failures in my PRs.
TL;DR – Danger reads some JUnit files, looks for failing tests, and write a comment on Github showing which tests failed in a step. I cannot figure out how to read all JUnit across all my containers to produce a single comment with all failing specs
Longer version –
I find it very difficult to use Danger with CircleCI 2.0 workflows. Basically Danger scans the output JUnit file and makes a comment on the relevant PR on Github like this
the problem is, Ideally I would need to aggregate the JUnit files from all containers (I am using parallelism: >=2). Since there is no possibility to use a when: always on jobs, I cannot figure out a way to execute the Danger report job after all my containers have finishe the same job regardless of failure, and therefore I have resorted to a trick that duplicates the danger PR message from above using a different index, adding a step on each container. In my circleCI, I ended up adding Danger as a step on the SAME job than the one executing the tests with parallelism
# Config that produces %{parallelism} messages on Github
rspec:
parallelism: 2
executor: my-executor
steps:
- attach_workspace:
at: ~/workspace
- run: mkdir -p /tmp/test-results/rspec
- run:
name: run RSpec and retry once failing specs only if there are few of them (get rid of flakiness)
command: bash .circleci/rspec_with_retries.sh
- store_test_results:
path: /tmp/test-results/rspec
- store_artifacts:
path: /tmp/test-results/rspec
# Dangerfile-RSpec
- run:
name: Skip github fingerprint check that randomly fails
command: mkdir -p ~/.ssh/ && echo -e "Host github.com\n\tStrictHostKeyChecking no\n" > ~/.ssh/config
when: always
- run:
name: run danger for RSpec tests
command: >
bundle exec danger
--danger_id=danger-rspec-${CIRCLE_NODE_INDEX}
--dangerfile=Dangerfile-test-report
--fail-on-errors=true
when: always
The result being that, if some specs failed on both containers, I’ll end up with 2 Github comments like the one from my screenshot (I need to reference them with an ID to make sure the comments are deleted/refreshed with new commits that trigger new builds)
Since I am thinking on adding more containers to reduce build times, I will end up polluting too much my Github PR, so I’d like to fix the circleci config to have a single Danger notification. Any idea how I could do this ?
I originally wanted to trigger a new job dependent on my “rspec” job, but then I realized it is not possible to force the execution of a job if any of the required jobs failed. I have found 3 relevant issues opened regarding this*, but none seemed to provide enough information.
I would like to highlight that, my other requirement is that the “rspec” job should indeed be reported as “failed” because I use the specific job status as status check on my PRs, and I want to see a red cross on rspec when rspec fails. SO the solution that involves passing an environnement or whatever variable and always returning that job passes feels like a bad idea
*Always run a job when dependents jobs were failed
*How to run two jobs sequentially always running the second if the first fails
*Run job after dependent jobs, no matter if they fail or succeed