Test summary didnt show anything, despite store_test_results was succesfull

hi there,

i tried to upload my junit xml test report into test summary in circle ci, the xml report is indeed getting uploaded to circleci succesfully

!/bin/bash -eo pipefail mkdir -p ~/test-results/junit/ find . -type f -regex “.*/build/test->results/.*xml” -exec cp {} ~/test-results/junit/ ;

CircleCI received exit code 0

Archiving the following test results

  • /home/circleci/test-results/junit/TEST-com.daya.shared.taha.SampelclazzTest.xml
  • /home/circleci/test-results/junit/TEST-com.daya.taha.ContextInnUnitTest.xml

Uploaded

there should be 2 files get uploaded, TEST-com.daya.shared.taha.SampelclazzTest.xml and TEST-com.daya.taha.ContextInnUnitTest.xml
but nothing shown at the test tab, only the total of xml report was added but the xml test itself didnt shown

i followed this guideline by circle ci

here is the config i used

  - run:
      name: Download Dependencies
      command: ./gradlew androidDependencies
  - save_cache:
      paths:
        - ~/.gradle
      key: jars-{{ checksum "build.gradle" }}-{{ checksum  "app/build.gradle" }}

  - run:
      name: unit test on sonarqube
      command: ./gradlew sonarqube

  - store_artifacts:
      path: build/reports/jacoco/jacocoRootReportDebug/jacocoRootReportDebug.xml

  - run:
      name: Save test results
      command: |
        mkdir -p ~/test-results/junit/
        find . -type f -regex ".*/build/test-results/.*xml" -exec cp {} ~/test-results/junit/ \;
      when: always
  - store_test_results:
      path: ~/test-results
  - store_artifacts:
      path: ~/test-results/junit         

  - run:
      name: clean previous build
      command: ./gradlew clean

  - run:
      name: ui test on browserstack
      command: ./gradlew executeDebugTestsOnBrowserstack

  - store_artifacts:
      path: app/build/outputs/apk/debug/app-debug.apk

      # See https://circleci.com/docs/2.0/deployment-integrations/ for deploy examples

**note : both report was generated from different module, originally SampelclazzTest reside in shared module, and the ContextInnUnitTest in app module, the structure of project look like this

  • app
  • shared

I have a similar issue, confirmed the test results are being populated appropriately and am able to see the results when using store_artifacts. Nothing appears in the Tests tab

    steps:
      - run:
          name: Make artifact directories
          command: |
            mkdir -p /test-results/
      - run:
          name: Run Unit Tests
          command: |
            /opt/Unity/Editor/Unity - quit -batchmode -nographics -silent-crashes -logFile /dev/stdout -projectPath ./Source/ -runTests -testResults /test-results/results.xml
          working_directory: ~/project/
      - store_artifacts:
          path: /test-results/results.xml
          destination: /test-results
      - store_test_results:
          path: /test-results

1 Like

image
All tests appear to be uploaded appropriately

Hello,
Though the topic is quite old similar issues are common. I’m sending an update so that people from the future could troubleshoot their issues when encountered.
CircleCI test results tab is a crap - it won’t inform you about progress or issues while loading the test result files.
First of all - always upload the output .xml results file as artifact.
If you had the file - check its content. CircleCI can understand more than 1 format but won’t warn if it cannot parse yours.
Make sure you point to a directory with the test results - not the actual file. If you provide the file - it would say that it passed and show no output. Similar problems could occur if the file cannot be read properly. CircleCI images run as a normal user and your file might just lack correct credentials.
If you are sure that the file was uploaded correctly - and checked the artifact - try restarting job with SSH access and (assuming you have CircleCI image) try to access CircleCI from command line.
If that is still failing - try to create a dummy job which injects your test results file only and use a file which is known to work. Then in several re-runs add/modify content to get the output you have - you might find what is failing in the CircleCI parser.
If the CircleCI cannot show a file that is working for some other project (downloaded from it) then it is most likely something wrong with your build script (config.yml)…
Though it is never easy with CircleCI… I wish you good luck with the troubleshooting.

3 Likes

So, after 9 months and no solution for this issue, I was disappointed with CircleCI :slight_smile:

1 Like

After some hours dealing with same problem (but using Maven instead of Gradle) and based on Test summary troubleshooting” article where is detailed the things that your test results should include, I realized the problem is that neither surefire nor failsafe include the file attribute in the testcase tag.

Looks like all file, name, time and classname attributes must be present in testcase for CircleCI to be able to handle the test results.

What I did for CircleCi to be able to parse the test results was to include a step, before the store_test_results, for removing the noNamespaceSchemaLocation attribute from the testsuite tag (as the schema does not allow the file attribute for testcase tag) and add the file attribute to the testcase tags based on the content of the classname attribute, like so:

      # Make xml test reports CircleCI compliant
      # #########################
      - run:
          name: Make xml test reports CircleCI compliant
          command: |
            sudo apt-get update && sudo apt-get install -y xmlstarlet
            FILES="reports/**/*"
            for file in $FILES
            do
              echo "Processing $file file..."
              # Remove surefiere and failsafe namespaces
              xmlstarlet ed --inplace -d '//testsuite/@xsi:noNamespaceSchemaLocation' $file
              # Add attribute "file" to testcase nodes
              xmlstarlet ed --inplace -i "//testcase" -t attr -n file -v "" $file
              # Update attribute "file" based on "testcase" attribute
              xmlstarlet ed --inplace -u "//testcase/@file" -x "concat(concat('src/test/java/', translate(./../@classname,'.','/')), '.java')" $file
            done

Please, note that in my code all test results are saved in a reports sub-folder and that the step is run in an ubuntu-2004:202201-02 image where the xmlstarlet package is installed.

Hopes this helps someone else but it would be better if this problem is fixed in Circle CI so the file attribute is not mandatory and is inferred from classname (in case it is needed for something).

1 Like

Hi friends! I battled with this same issue today and found a solution for me. For some reason, CircleCI is incredibly picky about the test results being in not just a subfolder— but a subfolder of a subfolder.

This final structure worked for me:

./junit/rspec/rspec.xml

And this config:

  - store_artifacts:
      path: junit

  - store_test_results:
      path: junit

It would not work if rspec.xml was just in the junit folder. It had to be in another folder. WTF? Anyways I hope it helps you.

1 Like

I just realized, if we first open the job view, we will see as if it has problems uploading on test summary and artifact tabs.

I have nodeJS mocha test and my config is as guided on the docs page:

      - store_test_results:
          path: test-results          

      - store_artifacts:
          path: test-results

Apparently after it’s done executing, re-accessing the page will show the test summary and the artifact! What a bug.

Hi,
I tried used approach described by @rubensa but still circle won’t pick up my results.
I think my results file might be missing some parts - it comes from cgreen c/c++ unit test framework and looks like this:

<testsuite name="tests-timezone">
<testcase classname="tests/timezone" name="apply_timezone_offset" time="0.00100" file="tests/timezone.c"> </testcase>
<testcase classname="tests/timezone" name="get_timezone_offset" time="0.00100" file="tests/timezone.c"> </testcase>
<testcase classname="tests/timezone" name="set_timezone_offset" time="0.00100" file="tests/timezone.c"> </testcase>
</testsuite>

Can anyone share a example Junit style results file that circle parses correctly?

Now I’m using JUnit 5 and I updated just a bit the script I’m using. This is how it shows now:

# Setup unit test results to show them in Circle CI
# ########################
- run:
    name: Setup unit test results
    command: |
      mkdir -p reports/surefire-reports/
      find . -type f -regex ".*/target/surefire-reports/TEST-*.*xml" -exec cp {} reports/surefire-reports/ \;
# Make xml test reports CircleCI compliant
# #########################
- run:
    name: Make xml test reports CircleCI compliant
    command: |
      sudo apt-get update && sudo apt-get install -y xmlstarlet
      FILES="reports/**/*"
      for file in $FILES
      do
        echo "Processing $file file..."
        # Remove surefiere and failsafe namespaces
        xmlstarlet ed --inplace -d '//testsuite/@xsi:noNamespaceSchemaLocation' $file
        # Add attribute "timestamp" "testsuite" node
        xmlstarlet ed --inplace -i "//testsuite" -t attr -n timestamp -v "$(date -r $file +'%Y-%m-%dT%H:%M:%S')" $file
        # Add attribute "hostname" "testsuite" node
        xmlstarlet ed --inplace -i "//testsuite" -t attr -n hostname -v "$(hostname)" $file
        # Add attribute "file" "testsuite" node
        xmlstarlet ed --inplace -i "//testsuite" -t attr -n file -v "" $file
        # Update attribute "file" based on "testsuite" attribute "name"
        xmlstarlet ed --inplace -u "//testsuite/@file" -x "concat(concat('src/test/java/', translate(./../@name,'.','/')), '.java')" $file
        # Add attribute "file" to "testcase" nodes
        xmlstarlet ed --inplace -i "//testcase" -t attr -n file -v "" $file
        # Update attribute "file" based on "testcase" attribute "classname"
        xmlstarlet ed --inplace -u "//testcase/@file" -x "concat(concat('src/test/java/', translate(./../@classname,'.','/')), '.java')" $file
      done
# Show test results in Circle CI
- store_test_results:
    path: reports

One thing to take into account… If the tests goes well, in the TEST tab, you only see a message like:

Great Job

19 tests are passing. Visit Insights to increase your speed and confidence.

and, if you go to Tests Insights page, you need to be patient to see the results as:

Data delayed up to 24 hours.

NOTE: I found some interesting info about “Common JUnit XML Format & Examples” at GitHub - testmoapp/junitxml: JUnit XML file format & JUnit XML examples. Including format specification, description & conventions.