Serialize workflow jobs

deployment
workflow

#1

I started trying out workflows, but I guess the same question could apply to job steps.

One of my workflow jobs (previously a build step) is for deployment. When we start multiple builds concurrently on the same branch and if both of them get to the deployment step roughly at the same time one of them will inevitably fail due a currently running deploy (we deploy ECS services through AWS CloudFormation and it will complain for any actions on the Stack unless it’s on a stable state). Is there any way to serialize those steps? So that if a deployment is already running for that branch the next one is blocked until the previous finishes.

It would be even better if there was a way to “queue” artifacts while that job is running and only run the step for the last one that was successfully built. I believe AWS CodePipeline does this.
This means we could trigger several builds at the same time and make sure only the most recently built artifact (docker image in our case) is deployed.

I went through the docs and I couldn’t find anything, but I believe it would be very useful and it will definitely increase the reliability of our builds.


#2

I have a similar desire: I want to do a build job followed by a fan-out to many test jobs that may run in parallel, but I need each of those tests to be running at most once at a time (because each of them hits a 3rd-party service under a test-specific account and saturates its rate-limit). (If I had three test jobs A, B, and C, it would be fine for those three to run in parallel, but it would not be fine for two A jobs from different commits to run at the same time.)


#3

Such a feature would be super helpful… it would be nice if each ‘deploy’ job was able to run to completion before a new job (of the same type) would start.


#4

The Circle team posted a reasonable solution back in the day, pre 2.0 and workflows https://discuss.circleci.com/t/serializing-deployments/153. I made the necessary adaptations to get it working w/ workflow jobs, and we’ve been happily using this since moving to workflows.

The original code in the repo they linked hasn’t been updated, but the gist with my working modifications is here.


#5

That is super helpful. I’ll add that as soon as I can find some time to go back to this. Thanks @amcelwee

Still, I with this was something that CircleCI was doing natively. It even crossed my mind switching to AWS CodePipeline (and probably CodeBuild) just to get that out of the box.


#6

That solution does mean your containers will be running while waiting for their turn. Not too bad for a deploy job, but it does mean my fan-out case would probably be a lot more expensive because of all of the containers waiting for their turn. (In my use-case, I’d be fanning out to around 100 specific jobs that each need to be serialized.)


#7

@AgentME - “Fan out to 100 specific jobs that each need to be serialized” sounds like exactly what workflows does via build chaining of the requires key. Maybe I’m not following your use case, though.


#8

The 100 different jobs can run parallel with each other, but two of the same job (such as from different commits) can’t run at the same time as each other.


#9

The script supports serializing on the combination of tag, branch, and job name, but you don’t have to include the branch name as a dimension to lock on. Just lock on the job name, and then for a given commit, you’ll get concurrency up to your circle build container count, and any individual job will be serialized across commits.


#10

This topic was automatically closed 41 days after the last reply. New replies are no longer allowed.