Storing and analyzing advanced test metadata

Hi all!

We have a C++ computer vision system for which we run some regression tests on videos (see here for more info). This is, again, a quite open ended question.


Some of the tests are simple “pass/fail”, but there are also metrics like “false positive/negative percentage on video X” etc. which are more nuanced, and which we’d like to collect for each test run. E.g. we want to have a nice plot showing us how we have improved false negative rate from 4.24242% to 1.12345% over the last two months.

I am pretty sure we are not the only ones having this kind of wish/problem, however I could find almost nothing about this topic. Maybe I am missing a niche or don’t know the terms to search for?

Anyways, options I have looked at so far:

  • Add a job to upload stuff to a GSheet -> This would be very nice as it easily allows creating plots etc. from the raw data, and we don’t have to host anything oureslves. However, GSheets API (primarily auth) is a nightmare.
  • Hack something to output my data as JUnit XML -> can show some details in GUI, but no analysis/plots of what I want. Very simple, but very unflexible.
  • Host our own database to store all the results in, and query however we like -> umm that sounds like a little too much work

I looked at the CCI integrations page (https://circleci.com/integrations/) and did not really find anything either.

If anyone knows more about this, that would be great!

Thank you all!

For future reference, this has now been solved using GSheets and Google service accounts, which make API auth relatively easy. You can follow this (slightly outdated Google API console screenshots) tutorial which works well enough.

We pass the priv key JSON file to the Python script which does the upload via a base64 encoded ENV var for easy handling of newlines etc.

1 Like