My company has struggled with a few Selenium tests being flaky for years.
It currently seems infeasible to debug all cases of flaky tests, so I want to make my workflows more tolerant of them without making flaky tests invisible.
What I wish I could do is retry the tests several times and continue deployment if they eventually pass, but get the test jobs and workflow to show “Warning” status in the pipelines view if any tests had to be retried.
Desired behavior
- Workflows can continue to
deployjob if all tests passed the first time or with retries - A workflow doesn’t show “failed” if it deployed successfully
- But I can see that something went wrong in the pipelines view if any tests had to retry (not just success across the board)
My workaround that my coworkers don’t like
I could have the test jobs fail if there were any retries, and have a confirm-tests job that always runs to examine the test artifacts, and pass as long as it sees that the tests passed after retries. And then I could make the deploy job depend on this confirm-tests job.
This way, test flakiness would still be visible in pipelines view (as “failed” test jobs) but deploy could still proceed.
But my coworkers didn’t like the idea of deploying in workflows that show “failed” status.
Are there any other ways I can deal with this?
I’m not very satisfied with the options CircleCI gives me to work with in this case…maybe there are features I’m unaware of?