Rollback via rerun deploy job

Deployments
1

I currently use CircleCI for building docker images, pushing to ECR and deploying to ECS.

I want to be able to manually rollback a deploy if any issues are detected on production.

The way I can think of is to rerun the previous deploy job so that we can redeploy previous ECR image. I currently find a button to “Rerun Job with SSH” which allows me to rerun a previous deploy job.

I want to make sure this is the right way to rollback. And I want to ask whether there is a way to rerun a job without SSH since I don’t need the SSH connection for this case.

2

Hi @wangxiaoyou1993 welcome to the CircleCI community! At this time there are two ways to manually re-run a previously run job.

The first being the Rerun Job with SSH feature you’ve identified, which works best for debugging purposes by accessing a spun up container and manually running commands. While there is an option to rerunning a workflow from failed / start, I don’t recommend this for a rollback unless your config.yml is set up with logic to handle this.

Another way is by adding an approval step to your workflow, where anyone with push access can manually approve a job to run. You can read more about approval jobs here:
https://circleci.com/docs/2.0/workflows/#holding-a-workflow-for-a-manual-approval

As I mentioned earlier, you can also have your config.yml include logic to handle rollbacks. This could include having a build, test, deploy and rollback workflow, where the rollback job checks for any issues with the deployment and will rollback as needed. You can even utilize the when attribute and our v2 API to trigger a parametrized pipeline with the rollback job as a separate workflow.

Let us know how these options would work for you!

3

@wangxiaoyou1993 Actually i have got the same issue like rollback strategy. Here is the final solution what i found after taking support from CircleCI help center.

version: 2.1

parameters:
rollback_job:
type: boolean
default: false

jobs:
build:
docker:
- image: cimg/base:2020.01
steps:
- run: echo “build job”
- run:
name: run rollback job
command: |
curl --location --request POST ‘https://circleci.com/api/v2/project/:vcs/:org/:repo/pipeline’
–header ‘Content-Type: application/json’
-u “${CIRCLE_TOKEN}:”
-d ‘{
“parameters”: {
“rollback_job”: true
}
}’

      when: on_fail

test:
docker:
- image: cimg/base:2020.01
steps:
- run: echo “test job”
- run:
name: run rollback job
command: |
curl --location --request POST ‘https://circleci.com/api/v2/project/:vcs/:org/:repo/pipeline’
–header ‘Content-Type: application/json’
-u “${CIRCLE_TOKEN}:”
-d ‘{
“parameters”: {
“rollback_job”: true
}
}’

      when: on_fail

deploy:
docker:
- image: cimg/base:2020.01
steps:
- run: echo “deploy”
- run:
name: run rollback job
command: |
curl --location --request POST ‘https://circleci.com/api/v2/project/:vcs/:org/:repo/pipeline’
–header ‘Content-Type: application/json’
-u “${CIRCLE_TOKEN}:”
-d ‘{
“parameters”: {
“rollback_job”: true
}
}’

      when: on_fail

rollback:
docker:
- image: cimg/base:2020.01
steps:
- run: echo “running rollback job”

workflows:
build-test:
jobs:
- build
- test:
requires:
- build
- deploy:
requires:
- test
rollback_workflow:
when: << pipeline.parameters.rollback_job >>
jobs:
- rollback

I hope, it helps you :-)! Good luck.