Hey all,
I am trying to get --split-by=timings to work on an Elixir project but I keep getting the following error:
$ TESTFILES=$(circleci tests glob "test/**/*_test.exs" | circleci tests split --split-by=timings --show-counts)
Read 359 lines of autodetect(s)
Error autodetecting timing type, falling back to weighting by name. Autodetect no matching filename or classname. If file names are used, double check paths for absolute vs relative.
Example input file: "test/channels/filter_header_channel_test.exs"
Example file from timings: ""
Bucket 0: assigning 90 autodetect(s), total weight 90
When I SSH into a container, here’s what the very end of .circleci-task-data/circle-test-results/results.json look like:
{"classname":"Elixir.Foo.ControllerTest","file":null,"name":"test some stuff","result":"success","run_time":0.5231,"message":null,"source":"unknown","source_type":"unknown"}
{"classname":"Elixir.FooProject.SomeTest","file":null,"name":"test some stuff","result":"success","run_time":0.0038,"message":null,"source":"unknown","source_type":"unknown"}],
"files":["results.xml","results.xml","results.xml","results.xml"],"exceptions":[]}
Note that all of the test cases in that file have "file":null.
(“files” seems to be an aggregation from the 4 containers I’m running (parallelism: 4)
I have compared notes with someone running a Ruby project, and the JSON in his test container looks like:
{
"tests": [
{"classname": "...", "file": "ACTUAL/FILE/NAME", "name": "...", "result": "...", "run_time": "...", "message": "...", "source": "...", "source_type": "..."},
...
]
}
And, as noted in the sample, all of his test cases have a file name.
Now, I assume that results.json is created out of the JUnit formatter results that get saved as part of the store_test_results step in the build. Looking into the XML that gets stored after a build, and comparing notes again with the Ruby project, it appears that for Ruby/Rspec projects, RspecJunitFormatter outputs XML that includes a file attribute, whereas the XML outputted from Elixir’s JUnit Formatter (GitHub - victorolinasc/junit-formatter: A JUnit XML report exporter for Elixir's ExUnit) does NOT have that field. Compare:
rspec.xml
<testsuite>
<testcase classname="..." name="..." file="..." time="..." />
</testsuite>
VS.
exunit.xml
<testsuite>
<testcase classname="Elixir.FooProject.SomeQueryTest" name="test where stuff happens" time="0.1038"/>
</testsuite>
Okay, so one includes file names, the other does not. CircleCI’s split function has a --timings-type flag that can be passed-in to tell the splitter to either look for filename or classname. However, adding --timings-type=classname to the command does not fix the issue, instead, it causes a new error:
Read 359 lines of classname(s)
No timing found for "test/channels/test1.exs"
No timing found for "test/channels/test2.exs"
...
No timing found for "test/views/test200.exs"
No timing found for "test/views/test201.exs"
Bucket 0: assigning 90 classname(s), total weight 2799000
Which leads me to believe that CircleCI’s split utility requires a file name in the JUnit XML, which currently is not available in Elixir’s junit-formatter.
Can someone confirm this or not? Am I missing something in my configuration?
Config.yml
For thoroughness, here are the relevant parts of my config file:
version: 2.1
jobs:
build:
docker:
- image: circleci/elixir:1.8.2-browsers
environment:
MIX_ENV: test
- image: redis:4.0.14
- image: circleci/postgres:10.10-alpine-postgis
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD:
working_directory: /home/circleci/my_project
parallelism: 4
steps:
- checkout
... caching stuff ...
- run: TESTFILES=$(circleci tests glob "test/**/*_test.exs" | circleci tests split --split-
by=timings --timings-type=classname --show-counts)
- run: mix test ${TESTFILES} --trace
- store_test_results:
path: test-results/exunit
Other knowns:
- Aside from timings not working, build run successfully with parallelization and files are split by filename.
- In our project, timings were actually working until December 9th, 2019, when they suddenly stopped. We had not implemented JUnit and were not using workflows.
- As of CircleCI v2.0, projects are expected to output their own JUnit XML.
- Splitting by timing data should work regardless of using workflows or not.
Additionally, I would draw your attention to a comment made in the RspecJunitFormatter repo:
Specifically the comment:
I think both file and filename are non-standard. Junit is a pretty lean, and isn’t really standardised at all, it’s just what the original junit tool used to produce. But many tools consuming rspec junit formatter output now rely on the file attribute so we cannot remove it now. I’m afraid this might be a problem xunit needs to fix — to ignore unknown attributes, or something.