We are having trouble splitting tests by timing information. It appears to always complain about missing timings file. And that is understandable as the ‘Downloading tests by timings’ step never appears in the workflow.
We have a following CI configuration:
workflows:
test:
.. triggered by tags push
jobs:
- ..some setup
- test-job
...
jobs:
test-job:
executor: our-executor
steps:
- ..mostly commands
- test-command
..parameters
...
commands:
test-command:
steps:
- ..setup steps
- ..suite splitting using 'circleci tests run' command like so:
cat ./tmp/all-tests.txt | circleci tests run --command="xargs echo > ./tmp/tests.txt" --split-by=timings --timings-type=filename
- run tests from the files
- store_test_results:
path: ./results
What works:
- Splitting by name works
- Rerunning only failed tests also works
- Insights work: all tests have correct relative file value and timings (It is a pain to navigate there as the workflow does not seem to be attached to any branch, since it’s triggered by a unique tags every time, so I found a way to view insights for
nullbranch and it’s ok) - Flakiness detection works
What doesn’t work:
- There is no ‘Downloading previous test results’ step in the job ever
- The splitting by timing always complains about the missing file because of the p.1
I’m asking here because there are too many components in this and I don’t have insights on how the hidden magic works. Please advise. Mainly the question is — why doesn’t the ‘Downloading previous test results’ step does not appear?
- Is it because the workflow is always triggered by a tag and never runs attached to a branch?
- Is it because the saving test results is in the command steps and not in the job steps?
- Is it because of the custom executor?
We’ll appreciate any help. Thank you!