Test-results-store directory structure


#1

Checking understanding, and reporting a possible UI/UX improvement:

Our tests are emitting an XUnit-formatted file to the path /home/ubuntu/test-reports/junit/action-ava.xml. Then we are storing these results with the configuration:

      - type: test-results-store
        path: /home/ubuntu/test-reports

However, on the UI we still see the dialog:

Is this expected? Is the format of our test results incorrect? Below is a snippet of them…

<?xml version="1.0"?>
<testsuites>
  <testsuite tests="1" failures="0" errors="0" name="universal › modules › welcome › ducks › welcomeDuck-tests › initial state">
    <testcase name="#1 universal › modules › welcome › ducks › welcomeDuck-tests › initial state"/>
  </testsuite>
  <testsuite tests="1" failures="0" errors="0" name="universal › modules › welcome › ducks › welcomeDuck-tests › setWelcomeTeam() updates teamId, teamMemberId">
    <testcase name="#2 universal › modules › welcome › ducks › welcomeDuck-tests › setWelcomeTeam() updates teamId, teamMemberId"/>
  </testsuite>
  <testsuite tests="1" failures="0" errors="0" name="universal › modules › welcome › ducks › welcomeDuck-tests › nextPage() increments">
    <testcase name="#3 universal › modules › welcome › ducks › welcomeDuck-tests › nextPage() increments"/>
  </testsuite>
</testsuites>

How to view test reports?
#2

Are you able to change the XUnit formatting to JUnit?


#3

@rohara thank you for the swift reply.

Aren’t they equivalent formats?

Our test runner, ava, can emit tap as its generic format. I use tap-xunit to transform the stream.

I’m open to suggestions for an alternative way of feeding test data into CircleCI!


#4

They’re extremely similar.

The artifacts service on 2.0 accepted this file ok:

<?xml version="1.0" encoding="UTF-8"?>
<testsuite name="rspec" tests="727" failures="0" errors="0" time="417.957921" timestamp="2017-01-27T14:53:40+00:00">
  <!-- Randomized with seed 58995 -->
  <properties/>
  <testcase classname="spec.helpers.breadcrumbs.breadcrumbs_helper_spec" name="Breadcrumbs::BreadcrumbsHelper returns a breadcrumbs array" file="./spec/helpers/breadcrumbs/breadcrumbs_helper_spec.rb" time="1.797002"/>
...
</testsuite>

#5

At a glance, is there anything from my example XML that would prevent ingestion?

If so, I’d be happy to fork the tap-xunit and add a dialect flag to make it work with CircleCI…


#6

The system looks for time so that’s a good place to start


#7

I am noticing that test results are no longer being displayed, though have been generated and uploaded.

Example:
https://circleci.com/gh/culturekings/afterpay/201

I run the tests and output junit to a directory. then I cat the junut to the screen to prove file exists. I am still seeing the “Setup test summary” button.


#8

@jared-fraser can you also upload your test results with the artifact-store? I think this might be the wrong service handling that feature. I’d like to confirm before reporting it.


#9

Hi @Eric

Please see this build with artifacts being stored as well as tests.

https://circleci.com/gh/culturekings/afterpay/203#artifacts/containers/0


#10

Thanks for sharing the link Jared. This actually wasn’t an artifacts upload issue–but the artifact upload helped me check the xml results. It looks like you have the same XUnit format as Jordan.

This looks like an incompatible format with JUnit. After doing some searching around, I found an JUnit schema definition here. Among other things, time is a required field, and that’s not present in your xml output.

@jordanh maybe you can use that schema definition to update your formatter? I looked around the Ava formatter links and didn’t find any for Junit.


#11

Thanks for the schema. It seems that there doesn’t seem to be an official schema definition and alot of libraries use their own. I found maven-surefire.xsd via junit-team github

Unfortunately it seems these testing libraries I use validate against a much more relaxed schema definition and would require me to submit pull requests against their libraries to get it changed.

Did Circle CI start validating against this schema recently? As previous builds using the same testing libraries used to generate results example build (private repo)


#12

Looking into it in our application – it seems that our test runner avajs actively has decided not to provide test timing information. So, even if I had the info I couldn’t transform it into the proper spec.

I feel like I missing a nice feature in CircleCI – what would I normally see? Each test case with pass/fail status and timing information?


#13

When it worked for us it gave us the following information in the test section. The failure message was nicely presented and quickly highlight what went wrong with the build.

Pass:

Your build ran 1836 tests in phpcs, phpunit with 0 failures
Slowest test: testname (took 1.03 seconds).

Failure:

phpcs 2 failures
Your build ran 849 tests with 2 failures
Symfony2.Arrays.MultiLineArrayComma.Invalid at filename (15:16)
Add a comma after each item in a multi-line array
Symfony2.Commenting.FunctionComment.Missing at filename (13:12)
Missing function doc comment


#14

Thanks for sharing that link. We’ve always referred to this format as the Junit format, and some other tools do as well. If the JUnit team says the Surefire xml schema definition is the one they use, I think that should be treated as the source of truth.

CircleCI hasn’t started validating against this schema recently, but I went and compared your two build links a bit closer. I see that your phpspec xml file actually does look like it meets the JUnit XSD definition above. I’ll ask around about this.

In the mean time, can you see if re-running the older build (build number 4937) results in test results getting picked up or not?


#15

4937 rebuilt.

here is the rebuild of that build.


#16

Okay, that looks like a bug. I’ll report this to the team. Thanks!


#17

cheers let me know if you need more information


#18

It looks like this has stopped working. My test reports are getting uploaded and showing up in “Artifacts” after a build, but it does not show the test summary at the top anymore. It was working until a few days ago.


#19

I opened a bug ticket. Thank you!


#20

Here are the relevant lines:

The test results are written to /tmp/test-results. I want to a) view the test summary at the top, and b) see the test-results and coverage files in the artifacts tab. This accomplishes (b) but not (a).

  - type: test-results-store
    path: /tmp/test-results
  - type: artifacts-store
    path: /tmp/test-results
    destination: test-results
  - type: artifacts-store
    path: coverage
    destination: coverage