We have a few browser tests that intermittently fail (presumably) due to javascript loading issues. Right now to focus our efforts on the biggest problems, we have to go through each failing build to count how often a particular test failed.
It would be great if CircleCI could automatically group and count our test failures over time so we don’t have to manually count up our failures.
I would add that a tool to pinpoint flaky tests would be very useful, and dovetails into this feature. It would basically need to keep track of which tests are failing over time, and also which tests “turn green” after only restarting the suite, and not committing new code.
We have integration tests that fail intermittently. It would be helpful to know the percentage of the time a particular test fails. We could use that information to develop some guidelines about when a test needs to be re-written to be more reliable. For example, if a test fails 25% of the time, then we’d have that as evidence to support removing it. Our current process is to occasionally skip flaky tests, but it’s more of a subjective decision.
Any updates on this? There’s a lot of value in test coverage, test-failures over time, etc., being on the Insights dashboard. In my mind, test coverage over time and test failures over time is a lot more interesting to plot than build clock time over time.
See other feature request: