How to add an automated rollback to a CircleCI workflow?

Deployments
, ,
1

I would like to configure a deployment workflow that runs integration tests after a code deploy job. If the integration test job fails, then I want it to automatically run a code rollback job. I’m not asking here how to write the actual code deploy and rollback jobs, that’s a separate issue; what I’m asking is how I can configure a workflow to do that.

I searched and didn’t find a lot of posts about this. I did find this 2018 post from @krhubert :

They asked pretty much the same question I’m asking, but nobody really answered it. @ThiagoBarradas suggested a workflow that has an approval step after the integration test, and a rollback job that depends on the approval: Auto rollback with circleci 2.0 · GitHub

However, there are two issues with this suggestion:

  1. I think it allows you to run the rollback only if the integration test succeeds, not if it fails.
  2. It’s not automatic, it requires a person to notice the failure and click the approval. I want an automatic rollback to known good code.

Anyone have any solutions for either (or, ideally, both) of the problems above?

2

I’ m also searching for the same solution. @cos please let me know if you find any, i’ll let you know if i found any solution.

3

As far as I can tell, there is no way to configure a job to run if and only if another job fails.

However, if I put both the code deploy and the tests in the same job, there is a way to have a step within a job run if any previous step in the same job failed, using the “when” attribute, documented here: https://circleci.com/docs/2.0/configuration-reference/#run

In other words, I have a job like this:

steps:
  - checkout
  - *deploy-code
  - *test-environment
  - *rollback-code

The rollback-code step has a “when: on_fail” for its run command. That means that if the deploy and test steps succeed, the rollback step will not run. However, if either the deploy fails, or the deploy succeeds but then the tests fail, then the rollback step will run. If the deploy fails, the test step is skipped; only the rollback runs.

This is not ideal, because I want to use the same rollback step as a separate job, and put it behind an approve step. That would give people the ability to manually roll back when they choose to, even after a successful deploy and test. So I basically need to configure both a rollback step and a rollback job and use them in different contexts, which is a confusing and error-prone configuration. It also makes the flow less clear to users.

CircleCI really needs to add better flow control logic for workflows, I think. But this workaround can work in the meantime.

4

@cos i found a way for rollback strategy with CircleCI API calling with the help of CircleCI support center. Instead of defining deploy and rollback in same job and workflow, i have created two workflows(deploy, rollback) like below.

version: 2.1

parameters:
rollback_job:
type: boolean
default: false

jobs:
build:
docker:
- image: cimg/base:2020.01
steps:

  - run: echo "build job"
  - run: 
      name: run rollback job
      command: |
          curl --location --request POST 'https://circleci.com/api/v2/project/:vcs/:org/:repo/pipeline' \
          --header 'Content-Type: application/json' \
          -u "${CIRCLE_TOKEN}:" \
          -d '{
                        "parameters": {
                          "rollback_job": true
                        }
                      }'
 
      when: on_fail

test:
docker:
- image: cimg/base:2020.01
steps:
- run: echo “test job”
- run:
name: run rollback job
command: |
curl --location --request POST ‘https://circleci.com/api/v2/project/:vcs/:org/:repo/pipeline’
–header ‘Content-Type: application/json’
-u “${CIRCLE_TOKEN}:”
-d ‘{
“parameters”: {
“rollback_job”: true
}
}’

      when: on_fail

deploy:
docker:
- image: cimg/base:2020.01
steps:
- run: echo “deploy”
- run:
name: run rollback job
command: |
curl --location --request POST ‘https://circleci.com/api/v2/project/:vcs/:org/:repo/pipeline’
–header ‘Content-Type: application/json’
-u “${CIRCLE_TOKEN}:”
-d ‘{
“parameters”: {
“rollback_job”: true
}
}’

      when: on_fail

rollback:
docker:
- image: cimg/base:2020.01
steps:
- run: echo “running rollback job”

workflows:
build-test:
jobs:
- build
- test:
requires:
- build
- deploy:
requires:
- test
rollback_workflow:
when: << pipeline.parameters.rollback_job >>
jobs:
- rollback

I think, it solves our problem.