Error Splitting Test Timings in RSpec and Ruby - Autodetect no matching filename or classname

Build Environment Running Tests
,
1

Hello,

I am having problems setting up parallel runs with my RSpec build. I can successfully find and run the tests, however I am unable to match the timing data.

I get an error of this form from the CircleCI tests split command, that suggests the filenames do not match – however based upon the error the example file from timings does appear in the example input file list:

Error autodetecting timing type, falling back to weighting by name. Autodetect no matching filename or classname.  If file names are used, double check paths for absolute vs relative.
Example input file: "spec/some/other_test_spec.rb.rb spec/some/test_spec.rb spec/some/yet_other_test_spec.rb ..."
Example file from timings: "spec/some/test_spec.rb"

I’ve swapped in some shorter names for the example, you can see that the “spec/some/test_spec.rb” appears second in the list of example input files.

Here is the section from my job that runs tests and saves the test results for future runs.

- run:
    name: Run tests
    environment:
      ELASTICSEARCH_HOST: 127.0.0.1:9200
      RAILS_ENV: test
    command: |
      mkdir /tmp/test-results
      TEST_FILES="$(circleci tests glob "spec/**/*_spec.rb" | circleci tests split --split-by=timings)"
      bundle exec rspec --format progress \
                    --format RspecJunitFormatter \
                    --out /tmp/test-results/rspec.xml \
                    --fail-fast \
                    -- ${TEST_FILES}
- store_test_results:
    path: /tmp/test-results
- store_artifacts:
    path: /tmp/test-results
    destination: test-results

In my job definition, I have set parallelism: 2 to ensure that we try to run multiple concurrently. I have run this build a few times to ensure that test data was generated after I added the parallelism directive.

One hypothesis I have is that the JUnitFormatter results file has filenames that start with a leading . (period). e.g.

<testcase classname="spec.some.test_spec" 
  name="TestSpec#json does some things" 
  file="./spec/some/test_spec.rb" time="0.118335"></testcase>

Would this be a possible reason for the debug output that shows matching filenames, but the message that the filenames did not match?

I have verified that the “Downloading previous test results” does find previous test data:

Downloading previous test results from https://circleci.com/gh/my-account/my-repo/27071
Total size downloaded: 1.9 MiB

I have further SSH’d into the container while running and looked at the stored test results in $CIRCLE_INTERNAL_TASK_DATA/circle-test-results.

I looked at the JSON in that directory and found a few examples which included the exact filename as listed - note the filename appears multiple times as there are multiple examples in the spec file.

{
  "tests": [
...
    {
      "classname": "spec.some.test_spec",
      "file": "spec/some/test_spec.rb",
      "name": "TestSpec#json does some things",
      "result": "success",
      "run_time": 0.015734,
      "message": null,
      "source": "rspec",
      "source_type": "rspec"
    },
    {
      "classname": "spec.some.test_spec",
      "file": "spec/some/test_spec.rb",
      "name": "TestSpec#json does some other things",
      "result": "success",
      "run_time": 0.015502,
      "message": null,
      "source": "rspec",
      "source_type": "rspec"
    },

Thank you in advance for your advice!

Dan

2

For future readers, I solved this:

  1. I had put quotes around the TEST_FILES variable which turned it into a string.The single line in the “example input file” made me re-examine this command since other examples of output showed multiple lines.
  2. I had some unnecessary command that I hadn’t seen where we were translating line feeds - tr ‘\n’ ’ '. This was not in my example above and I believe is not necessary.

The period in the filename in the JUnit XML format is fine and does not cause any problems.

I hope this is helpful to others.