Posting this here incase others experience the same issue, as I couldn’t find an existing issue about this or a support article.
I noticed recently that none of our tests (rspec) were being split correctly by previous test result timings. The config we were using was from the CircleCI docs:
- run: mkdir ~/rspec
- run:
command: |
SPEC_FILES=$(circleci tests glob "spec/**/*_spec.rb")
SPEC_CMD="xargs bundle exec rspec --format progress --format RspecJunitFormatter -o ~/rspec/rspec.xml"
echo $SPEC_FILES | circleci tests run --command="$SPEC_CMD" --verbose --split-by=timings
- store_test_results:
path: ~/rspec
- store_artifacts:
path: ~/rspec
I checked the stored artifacts and the rspec.xml files were being created and contained all the right information. But when the next test run started, we got a ton of “No timing found for” log messages for all files.
So I put a check run step in to check if the results were there, and surprise! Despite the “Downloading previous test results” step running successfully, by the time it got to “circleci tests run” command, the test results were no longer there.
The culprit: Some time ago, we speed up our test environment startup overhead by switching to using workspaces. An initial job will download the source code, download the gems, npm packages, etc, then persist the workspace. All jobs thereafter attach that workspace and are ready to go.
The problem is, the results are not present in the attached workspace and despite the persist workspace call being at the top of my job steps, attaching the workspace happens AFTER the previous results are download, and as a result, wipes out the directory where the results were downloaded to.
I have not been able to find any documentation on how to make previous results download AFTER the workspace is attached. It seems that if CircleCI detects a “store_test_results” step, it downloads the previous results before any user defined job steps.
The solution: Instead of storing in the path that the workspace backups, I have instead moved it to the /tmp directory. So instead of “~/rspec”, I now use “/tmp/rspec-results”. This appears, at least for now, to have resolved the issue. When a test runs, we are seeing far less “No timings found for” errors.