I am trying to calculate engineering metrics for our enterprise product team based on the report published by circleci here. I have a few questions when I tried to calculate the metrics for us. They are detailed as below
-
Stability
a. Do we consider mainline to be stable only if all tests (acceptance, regression, performance and client simulation tests) pass or after a CI build (that runs junits and integration tests) is green? Do we consider the time taken for the tests runs as part of stability or deploy time metric?
b. Is it the time spend in fixing red builds? If so, do we calculate a percentage of this time vs 24 hours? How do we include the number of builds in the calculation?
c. For our application we have around 5-10 CI builds during the day. The build takes around 1 hour to execute as it runs around 25k junits and integration tests. - Deploy time - We have a release process before which the builds are certified for production deployment. There are a few manual steps performed after all tests are green. Do we include this time in addition to the downtime taken for client upgrades?
- Deploy frequency - We deploy once every month for each of our customers. This frequency is part of the client agreement and not determined based on any engineering metrics. If we were to figure out the deploy frequency based on engineering teams performance, how should we be calculating it?
I would appreciate to hear thoughts on these queries.
Thanks!
