I am a believer in that any test code that fails should fail and a human should have to check it before it is approved. But 90% of the jobs in our CircleCI are not in fact tests but setup. Typically downloading and installation of dependencies. If these fail, it is not a problem with our deployment.
So I am looking for an option to set on a step as “setup” or similar which means that if it fails then the job is restarted automatically (With appropriate backoff and caps). If it passes in the end, then the job is formally considered a success.
This would save a lot of time as I notice that quite a few dependency installs fail due to outages even in the biggest most stable repositories.
closest ive found is retrying the install command inside the step itself. a short loop with a sleep between attempts gives you the backoff, and the step only fails if every try fails.
I think the tricky part is deciding which failures should actually trigger a retry.
A temporary network timeout while downloading dependencies is very different from a missing package version or a broken configuration. Retrying the first one makes sense, but retrying the second one just wastes time.
Rachel’s shell loop suggestion is a reasonable workaround, though I feel like having a retry policy directly on a step would be cleaner. Maybe something with a maximum number of attempts and configurable backoff.
One thing I’d really like to see is the retry count staying visible in the job output, even when the step eventually succeeds. Otherwise you could end up with green pipelines while dependency downloads are quietly failing several times every day.
I’d also prefer retrying only the affected step rather than restarting the entire job. Seems less wasteful, especially when earlier steps have already completed successfully.
Mrlaura: I think that you see my vision with perfect clarity. This is what I would want, yes.