Create coverage report format spec


#1

It would be great if circle had a standard format for reporting coverage run statistics. Teamcity has such a format, and it can be used by a coverage tool to report common coverage metrics, which will then be shown at the top of each build without having to open and read a report.


#2

Would it be correct to assume that SimpleCov is de-facto standard for coverage reports?


#3

I’m not sure I would say that? I’ve never heard of simpleCov because I don’t use ruby. I thought lcov was a more standard format for tools to use, but I could be mistaken about that as well. Really any format would work though, as long as different coverage tools could easily implement a reporter for that format. Right now I am using the istanbul tool for node to do coverage reporting.


#4

Got it, thanks for sharing that.

What exactly would you like to achieve by having a single standard? Would you like to merge multiple coverage reports from different test runners into a single report?


#5

No, I’m interested in circle supporting some kind of standard (or more that one, but one seems easiest) so that it can show coverage details more intelligently, like it does for test results. I’d ideally like it if you could view the coverage stats of every build by just going to that builds page and not having to dig through artifacts to find a report.


#6

:+1: Thanks!


#7

I have a related request to this one. That we have a way of tracking code coverage metrics over time associated with a build. A common format would be a very good start to this.

This feature request generalizes to arbitrary per-build metrics that people want to record and then plot on the build insights page.


#8

In the spirit of “do one thing, do it well” I think the current approach of generating a coverage file in your tests and committing it in the deployment phase via API to a dedicated provider like e.g. Codecov is an intriguingly simple solution, imo [/2 ct].


#9

I think there’s a lot of value in showing trends in coverage over time, which a CI tool is maybe uniquely in a position to do.
See my feature request


which is basically a request for an existing useful Jenkins feature.


#10

The original idea from @scottaj seems like something both easy to implement and provide a way for users to integrate with any tool out there.

For instance it can be as simple as defining a special file in artifacts folder (json, yml or plain text doesn’t matter) which must contain coverage details in a specific format, e.g.:

coverage.json
{
“coverage”: “95.5”,
“delta”: “-2.3”,
“html_report”: “path/to/index.html”
}

From CircleCI side it would just need to inspect this file to populate a build with coverage information. The rest can be done by users and eventually extracted in some sort of integrations for JUnit or LCOV or any other tool available.


#11