'Downloading previous test results' step never appears and leads to always missing timings

Build Environment Running Tests
1

We are having trouble splitting tests by timing information. It appears to always complain about missing timings file. And that is understandable as the ‘Downloading tests by timings’ step never appears in the workflow.
We have a following CI configuration:

workflows:
  test:
    .. triggered by tags push
    jobs:
    - ..some setup
    - test-job
...
jobs:
  test-job:
    executor: our-executor
    steps:
    - ..mostly commands
    - test-command
      ..parameters
...
commands:
  test-command:
    steps:
    - ..setup steps
    - ..suite splitting using 'circleci tests run' command like so:
        cat ./tmp/all-tests.txt | circleci tests run --command="xargs echo > ./tmp/tests.txt" --split-by=timings --timings-type=filename
    - run tests from the files
    - store_test_results:
      path: ./results

What works:

  1. Splitting by name works
  2. Rerunning only failed tests also works
  3. Insights work: all tests have correct relative file value and timings (It is a pain to navigate there as the workflow does not seem to be attached to any branch, since it’s triggered by a unique tags every time, so I found a way to view insights for null branch and it’s ok)
  4. Flakiness detection works

What doesn’t work:

  1. There is no ‘Downloading previous test results’ step in the job ever
  2. The splitting by timing always complains about the missing file because of the p.1

I’m asking here because there are too many components in this and I don’t have insights on how the hidden magic works. Please advise. Mainly the question is — why doesn’t the ‘Downloading previous test results’ step does not appear?

  • Is it because the workflow is always triggered by a tag and never runs attached to a branch?
  • Is it because the saving test results is in the command steps and not in the job steps?
  • Is it because of the custom executor?

We’ll appreciate any help. Thank you!

2

Hi @sergey-kruk-cobalt ,

When the store_test_results runs, do you see confirmation in the step details that the file was successfully uploaded?

3

Hi @yannCI !

Yes, I see that it’s uploaded and I see the results in insights and if tests fail in tests tab.

4

@yannCI

UPDATE: I can see now that this ‘Download previous test results’ step appears in workflows triggered by a push to a branch. But never in a workflow triggered by a push of a tag. The jobs in these two workflows use exactly the same command and the same executor.

I guess it’s a bug or a feature request? When pushing a tag try downloading results from a job with the same name?

5

It would be very handy to get some kind of workflow documented for test splitting.

  1. If I’m on a particular branch (main or other) and I’ve never run a build nor stored results for that branch, my current understanding is that split-by-timing won’t work - does it fall back to split-by-name in this case?

  2. If main is my main branch and I’m on an offshoot branch from main, do I not get the benefit of split-by-timing on my new branch even if main has previously stored results? I’ve recently seen that I get no timings found when I manually trigger a pipeline on a new branch (I’ll create a PR shortly and test it via the normal build route)

  3. If I’m on a branch that has been run and there are stored results, how do I know they have been pulled into the job? @sergey-kruk-cobalt mentions the Download previous test results step - is this what I’m looking for?

  4. If I push a new commit to that branch which also adds new tests then of course there is no timing data for those new tests. Do they get split-by-name by default and the rest of the tests are split-by-timing? Or does everything get split by name in that case?

For me, I think the important point is number 2 as that would make offshoot branches for test splitting redundant - well for timing-based splitting at least.