We run our Android instrumentation tests with flank and Firebase Test Lab. All the tests are run in two different devices, one in landscape mode and the other one in portrait mode. After the tests are executed, we get a report that includes two test suites, one for each device in which the tests were executed. The format is the following:
<?xml version=‘1.0’ encoding=‘UTF-8’ ?>
<testsuites>
<testsuite name=“Nexus7_clone_16_9-23-en-landscape” tests=“542" failures=“0” flakes=“0" errors=“0” skipped=“0" time=“3485.067” timestamp=“2023-12-05T13:21:08" hostname=“localhost”>
<testcase name=“---” classname=“---” time=“7.937"> </testcase>
…
</testsuite>
<testsuite name=“Nexus4-21-en-portrait” tests=“542" failures=“1” flakes=“0" errors=“0” skipped=“0" time=“2987.038” timestamp=“2023-12-05T13:21:11" hostname=“localhost”>
<testcase name=“---” classname=“---” time=“5.606"> <failure> …. </failure> </testcase>
…
</testsuite>
</testsuites>
We upload the report with the store_test_results command. When a test fails in both suites, there is no issue and the “TESTS” tab on Circle Ci shows the failing test.
There are no issues either when all tests pass in both suites.
The problem comes when all tests are passing on the first suite but one test is failing in the second suite. In that case the “TESTS” tab shows that all tests are passing and doesn’t show the offending test in the second suite.
Is CircleCI only taking into account one of the suites? We are not using parallelism within Circle CI because the tests are run externally, is that the reason why the “testsuite” elements in the XML report are not taken into account?