Flaky test detection


#1

Simple: add a report to each project that lists tests that change their status when a build is restarted. If it goes from failed to passed, or if it goes from passed to failed, without any code changes, mark it as a flake. Keep track of how many times this spec “flakes” and report that in list sorted by the total number of flakeouts, so we can start a “burndown” on our test flakes.


#2

Even a simpler flaky test detection would do.
Just report test failure statistics that failed in the ‘master’ branch in the past month. In 90% of cases these would be the flaky ones.


#3