Run jobs sequentially regardless of the outcome of the required jobs
Using the requires
key you can configure a sequential job execution within a given workflow. However, the limitation is that the dependent job will only run if the required job finishes successfully.
The method outlined in this post offers a workaround for the above limitation, and will allow you to configure a job to only start once jobs have succeeded or failed.
How it works
- Use CircleCI API v2 to retrieve the status of the required jobs.
- “Loop” until none of the required jobs is in a
running
state.
Requirements
- The method relies on CircleCI API v2, so you will need to create a personal API token
- The method uses
jq
, which is installed on CircleCI images; if you’re using a non-CircleCI image, you might need to install it.
Step 1 – Add the required environment variable to your project settings or to a context.
Go to either:
or
And add the following environment variable:
Variable Name | Description |
---|---|
CCI_Token | This will be a personal API token |
Step 2 – Configure your workflow
In the below example:
waiter
will check if any of the required jobs isrunning
dependent-job
will only start once all required jobs have completed (success
orfailed
).
Note: depending on the expected duration of the required jobs, you can specify a longer or shorter duration for the sleep
command.
version: 2.1
jobs:
upstream-job:
docker:
- image: circleci/python
steps:
- run: echo "Hello I am '$CIRCLE_JOB', the upstream job"
required-job:
docker:
- image: circleci/node
steps:
- run: echo "Hello I am '$CIRCLE_JOB', a required job"
other-required-job:
docker:
- image: circleci/python
steps:
- run: echo "Hello I am '$CIRCLE_JOB', another required job"
dependent-job:
docker:
- image: circleci/php
steps:
- run: echo "Hello I am '$CIRCLE_JOB', the dependent job"
waiter:
docker:
- image: circleci/node
steps:
- run: |
while [[ $(curl --location --request GET "https://circleci.com/api/v2/workflow/$CIRCLE_WORKFLOW_ID/job" --header "Circle-Token: $CCI_Token"| jq -r '.items[]|select(.name != "waiter")|.status' | grep -c "running") -gt 0 ]]
do
sleep 1
done
- run: echo "All required jobs have now completed"
workflows:
my-workflow:
jobs:
- upstream-job
- required-job:
requires:
- upstream-job
- other-required-job:
requires:
- upstream-job
- waiter
- dependent-job:
requires:
- waiter
Caveats
- This method adds another job to your workflow.
- If any of the required jobs fails (or, if several or all of them fail) the dependent job will still run, however the workflow will be marked as
failed
.
Resources
Please see the related feature requests to add your vote and use-case: