Example of splitting by test name for rspec rails

Build Environment Running Tests
, ,
16

I spent a day and a half frustrated by circleci trying to make this work. Test splitting by anything other than test names is really not documented well enough.

I finally got it to work after coming to the conclusion that circleci tests run reads test names as if they are strings with no whitespaces. If you pass it a list of test descriptions like “Admin: Unauthorized: logging in fails”, it will interpret this test description as five different test names.

Instead, it’s much simpler to pass a “test name” in the format /path/to/spec/file.rb:$LINE_NUMBER. This doesn’t actually identify a test, since updating a spec file will shift tests around. But most PRs change very few specs, so most timings are stable in most PRs. Supplying a reasonable default time is ok.

  • we’re throwing out rspec_junit_formatter
  • we’re using the native json formatter for rspec
  • I asked an LLM to write a script to convert the json output to the JUnit XML format. Crucially, name isn’t the test description, but is /path/to/spec/file.rb:LINE_NUMBER

Then, you can prepare a list of examples in the file:line_number format to pass to circleci test run.

So, my .circleci config file contains something like this, to prepare a list of feature specs (the slow specs we want to distribute evenly)

      - run:
          name: 'Extract test examples'
          command: |
            bundle exec rspec --dry-run --format json -o /tmp/feature_specs.json spec/features/
            cat /tmp/feature_specs.json | bin/rspec_json_to_test_names > /tmp/feature_specs.txt
            cat /tmp/feature_specs.txt

followed by a step like this, to read in the specs

      - run:
          name: Run tests
          command: |
            cat /tmp/feature_specs.txt |\
            circleci tests run \
              --command="xargs bundle exec rspec -fd -fj -o /tmp/rspec.json" \
              --verbose \
              --split-by=timings \
              --timings-type=name

Followed by a step that converts the output in /tmp/rspec.json to an XML that can be used by circleCI in future test runs.

Here’s the utilities used, for inspiration. I haven’t reviewed them beyond checking that they work with real specs.

1 Like