We were switching over our services to a new CC and Circle CI was missed in this process. However, this wasn’t immediately noticed since the GitHub integration shows messaging very similar to the success notification even if builds were skipped due to non-payment.
Our tests are very fast so the only thing that’s different from the success messaging is that it says “1 neutral check” instead of “1 successful check”. Since we generally don’t use Circle directly unless there are build failures, it wasn’t immediately obvious to us that anything was wrong. This assumed success lead us to deploying code that wasn’t tested by CI.
We completely recognize it’s on us to not get complacent with the messaging we see, but at any rate I do believe that this (as well as any other failure mode that results in skipping without the user’s input) should result in a failed run.