I was attempting to automatically re-run failed builds our
develop branch. A green CircleCI build is necessary before merging into
develop, so all
develop builds should be able to produce a green build eventually, regardless of stability. We’ve successfully set this up after a couple failed iterations.
We have an errant feature branch that contains code from one of these failed iterations. We have three issues working together to create a hairy situation for us:
- It fails quickly (less than 30s)
- It has 6 executors
- Each executor re-enqueues itself upon failure
The result is a 6x amplification of enqueued builds each time the build inevitably fails. At this point, I have a build queue depth of something like 5000.
Luckliy I’ve sent so many API calls in the last hour in this infinite loop and racing to cancel them via the API over command line that I’ve rate-limited myself. Or at least that’s what I assume by all the 404 “Project not found” messages I’m now receiving. This has halted the amplification issue for now. However, our CI pipeline is currently completely blocked.
- How can I cancel ALL builds?
- How can I prevent myself from enqueueing a build that is already in the queue?