Hi all, we’ve been using CircleCI on my team for about a year now, having switched from Jenkins. One of the things we understood about CircleCI was that it was both a CI and CD tool.
There are a number of requirements for reliable delivery, but one of the core requirements for most automated deployments is strictly ordered deployments.
This means that if you push changes A and then B (ordered that way in git history), then B will never deploy before A. There are nuances to this, some systems may coalesce A and B into the same deployment, some may define critical regions such that A and B can run in parallel for most of the build but get strictly ordered for the parts with side effects, and so on, but the core premise is that you can’t deploy in the wrong order.
This is critical for anything that deploys schema changes or does data migrations, but is typically a very useful property to have in all circumstances. It’s essentially table-stakes for a continuous delivery product.
Jenkins, for all its faults, does a pretty good job at this, and it was never something that required any thought from us. In ~6 years of operating Jenkins, this was never an issue we encountered.
Unfortunately CircleCI doesn’t have this. We were pointed towards the Queue Orb, which has helped us with the transition to CircleCI, but with our team growing and deployment velocity increasing, the race conditions in the orb and lack of flexibility around critical sections in deployments, we’re starting to have more and more issues of deployments going out in the wrong order. Also because the queueing is implemented in build, we pay build minutes for the time our builds queue, which is starting to become a significant proportion of our build time bill.
We’ve attempted to speak to our account manager about this, but after a few emails back and forth he went silent on us, and hasn’t responded to an email in months. The answer I always get from CircleCI support is a link to their “ideas” board, which isn’t a great sign for critical product features.
So I was wondering, how do others solve this?
How do you ensure your deployments go out in-order?
How do you enable queueing builds?
Is it possible to get around queueing while being billed for build minutes?
Are there alternative services that get this right?