Test timings can't be found

I’m using RSpec in parallel and splitting by test timings. Recently the timings have stooped working (I’m not sure how far back, but at least the last week. I get output like the following:
TESTFILES=$(circleci tests glob “./spec/**/*_spec.rb” | circleci tests split --split-by=timings --timings-type=classname)
echo $TESTFILES
bundle exec rspec
–format RspecJunitFormatter
–out ./coverage/xml/rspec/rspec.xml
–format progress
– $TESTFILES

No timing found for "spec/api/api_constraints_spec.rb"
No timing found for "spec/application_spec.rb"
...

I have checked the stored test result XML (stored as an artifact for debugging) and they are indeed there, but CircleCI seems to not be finding them. The only thing that I can see that might be causing this is that the paths in the XML file start with ./, which is not present in the warning messages (nor the output of circleci tests glob).

Has something changed that I’ve missed?

1 Like

I’m seeing the same thing on an Elixir app. Timings/parallelization was working fine until a few weeks ago.

What version of CI are you using? We’re using v2 (not v2.0) and not workflows, and I think that might have something to do with it.

We’re using v2.1 with workflows.

1 Like

We are experiencing the same issue.

Do anyone has a solution for this No timing found issue?
We are experiencing the same issue since last month and for the info, we use version 2.1 with workflow, after investigation and using this validator got a lot of error. We did not change our formatting and it was working fine until last month approx. Dec 6, 2019 is the last time we check after that we got that issue.

We are also trying to figure out why we are getting

No timing found for "tests/integration/reckonhosted/DemoTestSet1Test.php"
No timing found for "tests/integration/reckonhosted/DemoTestSet1_1Test.php"

The command that we use for splitting the tests is the following (debugging one):

#!/bin/bash -eo pipefail
circleci tests split --split-by=timings --timings-type=classname --show-counts tests/integration/_circleci_tests_list.txt | xargs -n 1 echo

I am trying to experiment by removing the ‘–timings-type=classname’, since we are passing the list of path/filename. Guessing that maybe that’s the issue, but not sure.

Nope, this did not help with No timing issue.

We are using
config version: 2.1
workflows:
version: 2

Alright, we actually found the issue from the JUnit report that Cucumber generated by default. As we can see from the doc here the XML as following:


<?xml version="1.0" encoding="UTF-8"?>
<testsuite failures="0" errors="0" skipped="2" tests="2" time="0.05" name="Pending step">
  <testcase classname="Pending step" name="Pending" time="0.05">
    <skipped/>
      <system-out/>
    <system-err/>
  </testcase>
  <testcase classname="Pending step" name="Undefined" time="0.05">
    <skipped/>
    <system-out/>
    <system-err/>
  </testcase>
    <system-out>
      <![CDATA[]]>
    </system-out>
    <system-err>
      <![CDATA[]]>
    </system-err>
</testsuite> 

Using the validator that I mention in the previous comment. We found that some of the tags causing this issue on the circle-ci such as <![CDATA[]]> , <system-out/> , etc.

So we created custom JUnit that makes CircleCI happy about and it works. For the reference of how the XML format should be, please see this StackOverflow threads something like this:

<testsuites>
  <testsuite>
    <properties>
      <property> ... </property>
    </properties>

    <testcase> ... </testcase>
  </testsuite>
</testsuites>

Hope this can be helpful for those have similar issue like us.

I can confirm that RSpec and Karma is producing output similar to your Cucumber output. This sounds like CircleCI isn’t supporting the JUnit XML format properly (rather than three different test suites not supporting the format correctly).

Having the same problem.
Timings from our E2E that generate a junit xml report are not being picked up =/

Any news on this CircleCI!?

1 Like

For us in the end the issue was that CircleCI was expecting relative paths to tests in the xml report, but our report had absolute paths (kudos to Split by timing parallelism does not seem to balance out).

The message that the CircleCi was giving for splitting:

So it looks that internally they store something like
tests/integration/NameOfTest.php

In our xml report it was the absolute path (/root/project/…):

Once we cleaned up the report using the command below to remove the absolute part before storing it as test result/artifact, the timings started to be autodetected:
command: sed -i 's/\/root\/project\///g' tests/_output/integration/result.xml

Here how our test splitting started to look like after:

Note: We are not using flag --timings-type=classname and feeding the list of tests from the file. So the command looks something like this:
command: circleci tests split --split-by=timings --show-counts tests/integration/_circleci_tests_list.txt | xargs -n 1 echo