Getting "No timing found" errors using test splits by timing

Build Environment Running Tests
1

we’re running 20864 tests in our pipeline using timing-based test splits (circleci tests split --split-by=timings).

We keep seeing “No timing found for X” messages for some of our tests, despite confirming that they are running and the test results with timing are included in the test XML. You can also see under the TESTS tab for out workflow that all 20864 tests are running.

Looking at the results.json file confirms that some of our tests are not being recorded there. I counted only 20812 tests reported in the file which would explain the “No timing found” errors.

Are we hitting some kind of undocumented limitation? The file size of this JSON is in fact pretty large at 6MB.

2

@jasonwhipp Could you share the relevant portion of your config? Are you saving the JUINIT XML format test results to the save_test_results step?

3

Yes we are, when we execute the tests we use

–format RspecJunitFormatter
–out /tmp/test-results/rspec.xml

store_test_results:
path: /tmp/test-results

4

What does the output of the save test results step look like? Has it been successfully saved?
Provide as much information as possible about your current situation. Screenshots and code will help

The results json file you mentioned, is that saved to an artifact? How does it compare to the XML output?

5

We logged into our test container and SCP’d the “circle-test-results/results.json” file back for analysis. This results.json file has 20812 entries.

Circleci itself reports that 20864 tests ran.

For each test that reported “No timing found for”, we noticed that it is missing from results.json

No tests are missing from the test XML and circleci reports all of the tests as running successfully.

Therefor we concluded circle-test-results/results.json is missing 52 tests, but we don’t know why.