Better Test Reporting


#1

It looks like currently the test reporting capabilities are quite limited, i.e it only shows the number of passed / failed tests, but not the names and status of indivual tests and test cases.

For example, I have some rspec tests and output the results in junit format here:
https://circleci.com/gh/tknerr/linus-kitchen/37

The test results are shown only as “Your build ran 27 tests in unknown with 0 failures”

However, that’s all I got. There is no way to drill down into individual test / testcase names, or is there?

Screenshots of the actual rspec output vs the information shown on circleci here in this tweet:
https://twitter.com/tknerr_de/status/672000001556484097


#2

I believe its based on what the test reporter writes into the junit file. I’m not sure ive ever had details on success, but for failures ive gotten something like this.

https://circleci.com/gh/drazisil/p2pool/236

I’ll look into vagrant’s settings (that is the test runner, right?) and see what i can find when i get a chance.


#3

Well, at least the test names are present in the junit.xml file and could be presented as green / success test cases under the tests tab:
https://circle-artifacts.com/gh/drazisil/p2pool/236/artifacts/0/tmp/circle-junit.jv2iYQR/junit/results.xml

Nope, vagrant is not the test runner. Actually the tests are run with rspec inside a Vagrant VM, using the junit output formatter. At the end I then copy the junit report file to $CIRCLE_TEST_REPORTS:
https://github.com/tknerr/linus-kitchen/blob/master/circle.yml#L19


#4

Also the junit test report from my build contains all the necessary information (i.e. test case names):
https://circle-artifacts.com/gh/tknerr/linus-kitchen/37/artifacts/0/tmp/circle-junit.NYwJKIm/junit-report.xml


#5

We show only the brief version of the information if all the tests have passed. We will show the complete error message and the name of each failing test case if there are any failing tests.


#6

@alexey is this intended to stay like this or is better reporting of (successful) test results on the roadmap?

Imho especially for BDD-style tests with RSpec, Cucumber, etc it would be really beneficial to render the “executable specification” that is been run


#7

For now we are not planning to show detailed information for the tests that passed. I do get your point about executable specification, I’ll talk to our product team about that. Thanks.


#8

FYI as a workaround I was playing around with generating an html report and keeping that as a build artifact, e.g. here:
https://circle-artifacts.com/gh/tknerr/linus-kitchen/43/artifacts/0/tmp/circle-artifacts.2VraPeB/test-report.html

This was a good start, but then I noticed that the artefact URLs are not predictable due to the missing branch name and random chars => see discussion / feature request in #931


#9