I’ve been testing the rerun failed tests feature and noticed some unexpected behavior.
On the first test run, if there are failures, I can successfully rerun only the failed tests.
However, if a test fails again in the rerun, I am not able to rerun failed tests a second time.
Additionally, in this scenario, no test results are uploaded after the rerun.
I intentionally introduced a flaky test to simulate this, and the pattern is consistent. Is this the expected behavior? If so, is there a way to force a second rerun or ensure test results are uploaded after the first rerun attempt?
Hmm, on the second failure, is it a test failure? Or is it a different kind of failure? As long as it’s a test failure, then the test results should be getting uploaded. If it was a network timeout or something else, then that might cause the job to immediately stop and not upload test results.
Hmm yeah I think it’s because the job that is running the failed tests seems to be getting cancelled during the test execution? So the step that runs tests doesn’t finish its process of executing all the tests and writing the results to the XML results file.
Do you need that cancel step? Or can you adjust the timeout to be something higher?
I don’t think the cancel step is the cause of the issue because it’s set to run after 10 minutes, while my reruns are only taking around 4 minutes. I also added an extra log:
sleep $((60 * << parameters.minutes >>))
echo "Timeout reached: Cancelling the job..."
curl --request POST --url https://circleci.com/api/v2/project/gh/studio-corsair/parakeet/job/${CIRCLE_BUILD_NUM}/cancel?circle-token=${CIRCLE_CI_TOKEN}
This ensures that if the cancel step was responsible, it would appear in the logs before the job gets canceled. However, the rerun tests are still exiting early without any detailed errors, and I don’t see the timeout message in the logs.
It seems like something else might be causing the test execution to be interrupted.
