Test timings can't be found

[typhoon2099]

I’m using RSpec in parallel and splitting by test timings. Recently the timings have stooped working (I’m not sure how far back, but at least the last week. I get output like the following:
TESTFILES=$(circleci tests glob “./spec/**/*_spec.rb” | circleci tests split --split-by=timings --timings-type=classname)
echo $TESTFILES
bundle exec rspec
–format RspecJunitFormatter
–out ./coverage/xml/rspec/rspec.xml
–format progress
– $TESTFILES

No timing found for "spec/api/api_constraints_spec.rb"
No timing found for "spec/application_spec.rb"
...

I have checked the stored test result XML (stored as an artifact for debugging) and they are indeed there, but CircleCI seems to not be finding them. The only thing that I can see that might be causing this is that the paths in the XML file start with ./, which is not present in the warning messages (nor the output of circleci tests glob).

Has something changed that I’ve missed?

[danadaldoss]

I’m seeing the same thing on an Elixir app. Timings/parallelization was working fine until a few weeks ago.

What version of CI are you using? We’re using v2 (not v2.0) and not workflows, and I think that might have something to do with it.

[typhoon2099]

We’re using v2.1 with workflows.

[IceCreamYou]

We are experiencing the same issue.

[mcbuddy]

Do anyone has a solution for this No timing found issue?
We are experiencing the same issue since last month and for the info, we use version 2.1 with workflow, after investigation and using this validator got a lot of error. We did not change our formatting and it was working fine until last month approx. Dec 6, 2019 is the last time we check after that we got that issue.

[mlukjanska25f]

We are also trying to figure out why we are getting

No timing found for "tests/integration/reckonhosted/DemoTestSet1Test.php"
No timing found for "tests/integration/reckonhosted/DemoTestSet1_1Test.php"

The command that we use for splitting the tests is the following (debugging one):

#!/bin/bash -eo pipefail
circleci tests split --split-by=timings --timings-type=classname --show-counts tests/integration/_circleci_tests_list.txt | xargs -n 1 echo

I am trying to experiment by removing the ‘–timings-type=classname’, since we are passing the list of path/filename. Guessing that maybe that’s the issue, but not sure.

Nope, this did not help with No timing issue.

We are using
config version: 2.1
workflows:
version: 2

[mcbuddy]

Alright, we actually found the issue from the JUnit report that Cucumber generated by default. As we can see from the doc here the XML as following:

<?xml version="1.0" encoding="UTF-8"?>
<testsuite failures="0" errors="0" skipped="2" tests="2" time="0.05" name="Pending step">
  <testcase classname="Pending step" name="Pending" time="0.05">
    <skipped/>
      <system-out/>
    <system-err/>
  </testcase>
  <testcase classname="Pending step" name="Undefined" time="0.05">
    <skipped/>
    <system-out/>
    <system-err/>
  </testcase>
    <system-out>
      <![CDATA[]]>
    </system-out>
    <system-err>
      <![CDATA[]]>
    </system-err>
</testsuite> 

Using the validator that I mention in the previous comment. We found that some of the tags causing this issue on the circle-ci such as <![CDATA[]]> , <system-out/> , etc.

So we created custom JUnit that makes CircleCI happy about and it works. For the reference of how the XML format should be, please see this StackOverflow threads something like this:

<testsuites>
  <testsuite>
    <properties>
      <property> ... </property>
    </properties>

    <testcase> ... </testcase>
  </testsuite>
</testsuites>

Hope this can be helpful for those have similar issue like us.

[typhoon2099]

I can confirm that RSpec and Karma is producing output similar to your Cucumber output. This sounds like CircleCI isn’t supporting the JUnit XML format properly (rather than three different test suites not supporting the format correctly).

[BernardoSilva]

Having the same problem.
Timings from our E2E that generate a junit xml report are not being picked up =/

[darrikonn]

Any news on this CircleCI!?

[mlukjanska25f]

For us in the end the issue was that CircleCI was expecting relative paths to tests in the xml report, but our report had absolute paths (kudos to Split by timing parallelism does not seem to balance out).

The message that the CircleCi was giving for splitting:

So it looks that internally they store something like
tests/integration/NameOfTest.php

In our xml report it was the absolute path (/root/project/…):

Once we cleaned up the report using the command below to remove the absolute part before storing it as test result/artifact, the timings started to be autodetected:
command: sed -i 's/\/root\/project\///g' tests/_output/integration/result.xml

Here how our test splitting started to look like after:

Note: We are not using flag --timings-type=classname and feeding the list of tests from the file. So the command looks something like this:
command: circleci tests split --split-by=timings --show-counts tests/integration/_circleci_tests_list.txt | xargs -n 1 echo

[typhoon2099]

The paths in my spec files aren’t absolute, they’re relative to the project root. The only think I can see that may be preventing the timings from being used is the fact that the paths start with ./.

[typhoon2099]

I’ve just tried stripping off the ./ using sed -i 's/\.\///g' ./coverage/xml/rspec/rspec.xml and still get the “No timing found” messages all over the place. I’m running RSpec using:

TESTFILES=$(circleci tests glob "./spec/**/*_spec.rb" | circleci tests split --split-by=timings --timings-type=classname)
bundle exec rspec \
  --format RspecJunitFormatter \
  --out ./coverage/xml/rspec/rspec.xml \
  --format progress \
  -- $TESTFILES

[jaysonesmith]

I ran into a couple of these and it turns out one file was completely commented out and another was a helper file that happened to end in _spec.rb. Might be worth looking into the files that are being reported for similar stuff.

If that doesn’t help, you can check the stored test_results by SSH’ing into one of the containers and checking the file there.

cat /.circleci-task-data/circle-test-results/results.json | jq '.tests[] | .file' | grep {file}

[jaysonesmith]

While I’ve got things working for rspec and cypress, I’m still running into issues with Cucumber.

[billy1kaplan]

Did anyone figure out how to get this working for rspec?

At a random point in the middle of December (with no circleci changes on our end as far as we can tell), every run of our circleci pipeline has the No timing found for… message for every file in the test suite.

[aharbick]

@jaysonesmith’s answer helped me figure out that --timings-type=classname just doesn’t work.

circleci@707b3358e4a1:~/bw_rails$ echo "test/api/api_test.rb" | circleci tests split --timings-file /.circleci-task-data/circle-test-results/results.json --split-by=timings --timings-type=classname
No timing found for "test/api/api_test.rb"
test/api/api_test.rb
circleci@707b3358e4a1:~/bw_rails$ echo "test/api/api_test.rb" | circleci tests split --timings-file /.circleci-task-data/circle-test-results/results.json --split-by=timings
Autodetected filename timings.
test/api/api_test.rb

I was uploading the results correctly… but classname timings type was messing things up.

[mike]

To add to @aharbick’s comment about --timings-type=classname.

--timings-type=classname expects a list of class names not file names to be passed in. So if you are using circleci tests glob you will likely never want to use --timings-type=classname.

To make the point more clearly. circleci tests split simply takes in a list of strings and then tries to match them to the values that were originally in the test results file. So to give an example testcase line from a jUnit XML file

<testcase classname="MyClass" name="should access project data" file="tests/TestMyClass.java"/>

circleci tests split will simply look at the value in classname if you pass in --timings-type=classname. This is why this will never work if you pass in a list of file names. To use --timings-type=classname you want to have a method of creating a list of class names and not actually use the circleci tests glob command. The vast majority of use-cases will only make use of --timings-type=filename which is also the default if you leave it off.

Hope that helps clear up some of the confusion.