Report separate status checks for each test script to Github


#1

It would be extra useful to be able to display each test script or suite as a separate status check in Github. That is, for a project that runs two test scripts in circle.yml like so:

test:
  override:
    - run-first-test
    - run-second-test

It’d be great to see something like this in Github:

I spent some time fiddling around over the weekend and came up with a script that helps me do this:

test:
  override:
    - run-first-set-of-tests
    - run-second-set-of-tests; if [ $? -ne 0 ]; then ./send-github-status.js 'ci/circle:Suite 2' 'The second test suite failed' 'failure'; else ./send-github-status.js 'ci/circle:Suite 2' 'The second test suite passed' 'success'; fi

…but it’s quite a bear and creates lots of extra noise in the code. I could certainly streamline it a bit, but it’s still not great. It also requires me to include my Github user name and an access token as environment variables, which is not ideal—it’s not great for security and also means that the icon for the status is different—it’s my avatar on Github instead of Circle’s, as in the screenshot above.

Ideally, I’d love have have something like this give me the same result instead:

test:
  override:
    - run-first-test
      name: First Test
    - run-second-test
      name: Second Test

Some use cases:

  • It’s helpful to be able to show, at a glance, which part of the test suite(s) failed when looking at a PR.
  • If you have some solid tests and others that are flaky (say, some integration tests that rely on third-party infrastructure), this gives you a way to separate out the flaky results (they’d still show the PR or commit as failing overall, but it would be easier to see that it’s the flaky test that broke and thus is less likely to be a real issue).
  • In a monorepo that includes several projects, it’s very helpful to be able to separate out the tests by project. This also lets you easily see that a failure may have come from a different (but related) project than the one you were working in.

If different parts of the process (e.g. setup and build) fail before the tests, that could also be surfaced separately in the same way.


#2