We are running ui-tests with Playwright in CircleCi. In our playwright config we have multiple projects defined, so that the same tests sometimes run with multiple different configurations. Also we use sharding for parallelization of our tests.
This whole setup used to work well, but since recently the Testing Overview in the Tests tab does not show the correct number of executed and failed tests any more. I believe this has something to do with the recent release of the new feature “Successful rerun view” “Successful rerun” view available on Tests tab in job page - CircleCI Changelog.
To me it looks like:
- Tests running multiple times with different configurations are considered to be reruns of the same test.
- The overview does not collect the results from all parallel executions.
It would be great if this could get fixed soon.
Hi @Henrieke, thank you for reporting this - our team is currently working on a fix for this and we will keep you posted!
@Henrieke Hello! The issues around how the tests are displayed on the Tests tab should now be resolved. Would you mind confirming it on your side?
The number of failed tests and their list in the tab seem to be correct now. The number of passed tests and the total still don’t match the number in my Playwright report.
Also I still see the issue that, when all tests pass on multiple shards, the total test number seems to be counting only one shard.
@Henrieke Would it be possible for you to share a link to the job that you are seeing this problem? It would help us get a more clear understanding of what you are experiencing.
Also, what are the numbers that you are seeing on the Test tab vs the numbers on the Playwright report?