My test-reports are saved using --junit-xml with junit_family=xunit1 (when I had junit_family=xunit2, circleci wasn’t able to find test timings by filename or by classname; I tried both --split-by=timings and --split-by=timings --timings-type=classname to no avail).
With these settings, it is at least detecting the timings by filenames but it says No timing found for almost half of the tests which is really strange because I can see them being reported in both the TESTS and ARTIFACTS sections like all the other tests.
Autodetected filename timings.
No timing found for “my_module/feature_a//test_feature_a.py”
No timing found for “my_module/feature_b/tests/test_feature_b.py”
… (list truncated for readability)
It may be because of lack of autodetection of timings, out of my 5 parallel containers, my slowest one runs for 8 minutes and my fastest one runs for 1 minutes. This is very frustrating and CircleCI documentation is sorely lacking in this regard.
I’d really appreciate if someone has any insights on this.
When it comes to test splitting, there can be many factors involved where it isn’t able to find the timings. It could be they are present in the file, but not formatted correctly. It could also be that some of the test results are not being stored correctly or in the correct path since you mentioned it’s working for some of your tests.
With the above in mind troubleshooting these is usually easier when we can look at an example build link to take a look at the test results, your setup, etc.
If you are comfortable sharing a link to your project/build here – feel free to post it and I can look closer. Alternatively, you can also just submit a support ticket and note this discuss post (and provide an example build link) and we can investigate further, that way you don’t have to share the link publicly.
With all the above said, and I am not sure your exact project setup, but I do know there was some additional data massaging needed for Django tests to work with test splitting. I wrote a support article here with those details:
To resolve until the bug is fixed, you can do this:
cp -f custom_config.ini pytest.ini
pytest
This will run pytest without the -c option and the junit-xml will have appropriate file-paths and circleci tests split will correctly find timing information in them.