Cancel Builds En Masse

[robert-baldwin]

Hey there,

I was attempting to automatically re-run failed builds our develop branch. A green CircleCI build is necessary before merging into develop, so all develop builds should be able to produce a green build eventually, regardless of stability. We’ve successfully set this up after a couple failed iterations.

We have an errant feature branch that contains code from one of these failed iterations. We have three issues working together to create a hairy situation for us:

1. It fails quickly (less than 30s)
2. It has 6 executors
3. Each executor re-enqueues itself upon failure

The result is a 6x amplification of enqueued builds each time the build inevitably fails. At this point, I have a build queue depth of something like 5000.

Luckliy I’ve sent so many API calls in the last hour in this infinite loop and racing to cancel them via the API over command line that I’ve rate-limited myself. Or at least that’s what I assume by all the 404 “Project not found” messages I’m now receiving. This has halted the amplification issue for now. However, our CI pipeline is currently completely blocked.

Questions:

1. How can I cancel ALL builds?
2. How can I prevent myself from enqueueing a build that is already in the queue?

[drazisil]

Hi Rob. Our support supervisor reached out to you via our ticketing system and handled this for you.

[robert-baldwin]

Hey Drazisil,

Thanks for your reply. Where can I find the message? I don’t see it in my email.

Best,
Rob

[robert-baldwin]

Hey Drazisil,

I found an email notifying me that my API token was revoked. This helped stop the issue temporarily, so thank you.

We would like to enable the functionality to retry a single executor in the event that it fails for a specific branch. Ideally, this would retry a limited number of times before stopping. What guidance can you give re: existing Circle CI features to achieve this behavior?

Once that’s done, how can we go about re-enabling my API token?