Maintaining test artefacts when rerunning failed tests

[shashwatx]

Our workflow triggers a job that launches a suite of functional tests.
For these tests, we use parallelism=20 and --split-by=timings.
If tests fail, we rerun only the failed tests. Note that the original launch and the rerun have different job-ids.

The next time the workflow is invoked, functional test job fetches the historical timings from the previous job-id i.e. rerun which only contains execution times for the failed tests. We lose timings for all tests that passed on the initial run (launch). This leads to suboptimal partitioning of tests in the current workflow.

In case of a Rerun Failed Only should the artefacts from the initial run (launch) and all intermediate reruns not be carried forward to the same location as the artefacts generated by the rerun?

[sebastian-lerner]

@shashwatx

nal test job fetches the historical timings from the previous job-id i.e. rerun which only contains

You can use a similar approach as the snippet in the bullet: “Jobs that upload code coverage reports:” from this section Rerun failed tests - CircleCI.

Instead of restoring the coverage reports like the example shows, you would save/restore the JUnit XML from the original job run. This will make it so that on the rerun, you are uploading the test results from the original job run which will contain all of the timings from all of the tests. So that when you next push code and trigger your functional tests to be run again, CircleCI will be using the full list of tests & respective timings.