[Workaround] Run jobs sequentially regardless of the outcome of the required jobs

Run jobs sequentially regardless of the outcome of the required jobs

Using the requires key you can configure a sequential job execution within a given workflow. However, the limitation is that the dependent job will only run if the required job finishes successfully.

The method outlined in this post offers a workaround for the above limitation, and will allow you to configure a job to only start once jobs have succeeded or failed.

How it works

  • Use CircleCI API v2 to retrieve the status of the required jobs.
  • “Loop” until none of the required jobs is in a running state.

Requirements

  1. The method relies on CircleCI API v2, so you will need to create a personal API token
  2. The method uses jq, which is installed on CircleCI images; if you’re using a non-CircleCI image, you might need to install it.

Step 1 – Add the required environment variable to your project settings or to a context.

Go to either:

or

 
And add the following environment variable:

Variable Name Description
CCI_Token This will be a personal API token

Step 2 – Configure your workflow

In the below example:

  • waiter will check if any of the required jobs is running
  • dependent-job will only start once all required jobs have completed (success or failed).

Note: depending on the expected duration of the required jobs, you can specify a longer or shorter duration for the sleep command.

version: 2.1

jobs:
  upstream-job:
    docker: 
      - image: circleci/python
    steps:
      - run: echo "Hello I am '$CIRCLE_JOB', the upstream job"

  required-job:
    docker: 
      - image: circleci/node
    steps:
      - run: echo "Hello I am '$CIRCLE_JOB', a required job"

  other-required-job:
    docker: 
      - image: circleci/python
    steps:
      - run: echo "Hello I am '$CIRCLE_JOB', another required job"

  dependent-job:
    docker: 
      - image: circleci/php
    steps:
      - run: echo "Hello I am '$CIRCLE_JOB', the dependent job"

  waiter:
    docker: 
      - image: circleci/node
    steps:
      - run: |
          while [[ $(curl --location --request GET "https://circleci.com/api/v2/workflow/$CIRCLE_WORKFLOW_ID/job" --header "Circle-Token: $CCI_TOKEN"| jq -r '.items[]|select(.name != "waiter")|.status' | grep -c "running") -gt 0 ]]
            do
              sleep 1
            done
      - run: echo "All required jobs have now completed"

workflows:
  my-workflow:
    jobs:
      - upstream-job
      - required-job:
          requires:
            - upstream-job
      - other-required-job:
          requires:
            - upstream-job
      - waiter
      - dependent-job:
          requires:
            - waiter

Caveats

  • This method adds another job to your workflow.
  • If any of the required jobs fails (or, if several or all of them fail) the dependent job will still run, however the workflow will be marked as failed.

Resources

Please see the related feature requests to add your vote and use-case: