We have a C++ computer vision system for which we run some regression tests on videos (see here for more info). This is, again, a quite open ended question.
Some of the tests are simple “pass/fail”, but there are also metrics like “false positive/negative percentage on video X” etc. which are more nuanced, and which we’d like to collect for each test run. E.g. we want to have a nice plot showing us how we have improved false negative rate from 4.24242% to 1.12345% over the last two months.
I am pretty sure we are not the only ones having this kind of wish/problem, however I could find almost nothing about this topic. Maybe I am missing a niche or don’t know the terms to search for?
Anyways, options I have looked at so far:
- Add a job to upload stuff to a GSheet -> This would be very nice as it easily allows creating plots etc. from the raw data, and we don’t have to host anything oureslves. However, GSheets API (primarily auth) is a nightmare.
- Hack something to output my data as JUnit XML -> can show some details in GUI, but no analysis/plots of what I want. Very simple, but very unflexible.
- Host our own database to store all the results in, and query however we like -> umm that sounds like a little too much work
I looked at the CCI integrations page (https://circleci.com/integrations/) and did not really find anything either.
If anyone knows more about this, that would be great!
Thank you all!