I’m not sure if this is a bug or something we started doing wrong recently, but our code coverage artifact from SimpleCov has been listing incorrect coverage recently. I noticed last week (August 4 or so), but it might have been going on longer.
I have tests that I know hit certain lines of code, but the HTML file produced by SimpleCov when those tests run on CircleCI shows those lines as not being hit. In fact, it makes it look like some of those tests haven’t even run; for one file in particular the coverage report shows lines being hit by only 2 tests, when there are 6-8 tests that should be hitting certain lines.
When running the tests locally the generated coverage file shows the correct coverage on those lines and files, and I’ve also tried debugging my builds via SSH, and saw the proper output there too. I also checked the “Collect Artifacts” output from the build, and that seemed to have executed properly, at least as far as I could tell.
Is it possible that the artifacts aren’t being written or stored properly when the builds are run? Or could there be something else I’m missing or should test? I’m stumped as to why this is happening now, as it seemed to work fine before (though maybe I just didn’t notice), and we haven’t changed anything on our end related to our artifacts or builds that I’m aware of.