How does CircleCI match classnames to test metadata?

Build Environment Running Tests
1

I have a Pytest suite configured to use --split-by-timings and it works OK, but ideally I’d have the --timings-type=classname flag enabled too. However, I cannot figure out what heuristics Circle uses to match the classnames passed to circle tests split to the JUnit XML files from previous tests.

First note that modules are specified in the XML files using Python-style module specifiers e.g. path.to.some.test_feature.FeatureTestCase. Pytest has its own style of specifier e.g. path/to/some/test_feature.py::FeatureTestCase.

I have tried:

  • passing Pytest-style specifiers to circle test split
  • passing Python-style specifiers to circle test split, then converting them back before passing to Pytest
  • tweaking classnames to match the classname XML attribute (points to the test case class)
  • tweaking classnames to match the classname and name XML attribute joined (points to individual test methods)

but every time I get the dreaded No timing found for "..." error. Without the --timings-type=classname flag this isn’t an issue, so I’m pretty sure the XML is in the correct shape. I just can’t figure out why Circle isn’t matching these two strings together.

2

:wave: Hi @tpict, and welcome to the CircleCI community!

Functionally the circleci tests split command takes in a list of strings (like filename or classname), splits them accordingly (by timing or filesize, etc…, whatever you choose), then passes back out the string values isolated to each container.

If you don’t specify a timing type, and if the test splitter is unable to auto-detect it, the it falls back to weighting by filename.

Now if you specify --timings-type=classname , the test splitter will expect a list of classnames.

Therefore, you’ll need to ensure that:

  1. you pass a glob of classnames (rather than filenames) to the circleci tests split command.
  2. classnames in said glob exactly match the corresponding strings displayed in the classname field of the tests results.

Let me know if this helps.

1 Like