Aggregate JUnit tests results across severals containers including failing ones with Danger

I am using Danger Ruby to automatically report test failures in my PRs.

TL;DR – Danger reads some JUnit files, looks for failing tests, and write a comment on Github showing which tests failed in a step. I cannot figure out how to read all JUnit across all my containers to produce a single comment with all failing specs

Longer version –

I find it very difficult to use Danger with CircleCI 2.0 workflows. Basically Danger scans the output JUnit file and makes a comment on the relevant PR on Github like this

the problem is, Ideally I would need to aggregate the JUnit files from all containers (I am using parallelism: >=2). Since there is no possibility to use a when: always on jobs, I cannot figure out a way to execute the Danger report job after all my containers have finishe the same job regardless of failure, and therefore I have resorted to a trick that duplicates the danger PR message from above using a different index, adding a step on each container. In my circleCI, I ended up adding Danger as a step on the SAME job than the one executing the tests with parallelism

# Config that produces %{parallelism} messages on Github
    parallelism: 2
    executor: my-executor
      - attach_workspace:
          at: ~/workspace
      - run: mkdir -p /tmp/test-results/rspec
      - run:
          name: run RSpec and retry once failing specs only if there are few of them (get rid of flakiness)
          command: bash .circleci/
      - store_test_results:
          path: /tmp/test-results/rspec
      - store_artifacts:
          path: /tmp/test-results/rspec
      # Dangerfile-RSpec
      - run:
          name: Skip github fingerprint check that randomly fails
          command: mkdir -p ~/.ssh/ && echo -e "Host\n\tStrictHostKeyChecking no\n" > ~/.ssh/config
          when: always
      - run:
          name: run danger for RSpec tests
          command: >
            bundle exec danger
          when: always

The result being that, if some specs failed on both containers, I’ll end up with 2 Github comments like the one from my screenshot (I need to reference them with an ID to make sure the comments are deleted/refreshed with new commits that trigger new builds)

Since I am thinking on adding more containers to reduce build times, I will end up polluting too much my Github PR, so I’d like to fix the circleci config to have a single Danger notification. Any idea how I could do this ?

I originally wanted to trigger a new job dependent on my “rspec” job, but then I realized it is not possible to force the execution of a job if any of the required jobs failed. I have found 3 relevant issues opened regarding this*, but none seemed to provide enough information.

I would like to highlight that, my other requirement is that the “rspec” job should indeed be reported as “failed” because I use the specific job status as status check on my PRs, and I want to see a red cross on rspec when rspec fails. SO the solution that involves passing an environnement or whatever variable and always returning that job passes feels like a bad idea

*Always run a job when dependents jobs were failed
*How to run two jobs sequentially always running the second if the first fails
*Run job after dependent jobs, no matter if they fail or succeed

I’d be really interested in a solution to this. We run using 10 containers in parallel and would love the ability to have a Danger job that runs after all of them, aggregates their results, and reports failing tests back to GitHub.

We are looking for this solution where our artifact lies on various container according to parallelism. We have to visit each container to get the result file. Any other alternative?

I am still looking forward a way to deal with this. I was somehow hoping with the new features (Circleci Orbs, etc.) we’d have some progress this way, but I have not seen any new things that may be of use.

I have opened an issue on the Danger Ruby repo

To work outside the system, there is also and which can use GitHub webhook systems to do this outside of CI

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.