[Workaround] Run jobs sequentially regardless of the outcome of the required jobs

[yannCI]

Run jobs sequentially regardless of the outcome of the required jobs

Using the requires key you can configure a sequential job execution within a given workflow. However, the limitation is that the dependent job will only run if the required job finishes successfully.

The method outlined in this post offers a workaround for the above limitation, and will allow you to configure a job to only start once jobs have succeeded or failed.

How it works

• Use CircleCI API v2 to retrieve the status of the required jobs.
• “Loop” until none of the required jobs is in a running state.

Requirements

1. The method relies on CircleCI API v2, so you will need to create a personal API token
2. The method uses jq, which is installed on CircleCI images; if you’re using a non-CircleCI image, you might need to install it.

Step 1 – Add the required environment variable to your project settings or to a context.

Go to either:

• the project settings (Environment Variables)

or

• your organization settings (Contexts)

 
And add the following environment variable:

Variable Name	Description
CCI_Token	This will be a personal API token

Step 2 – Configure your workflow

In the below example:

• waiter will check if any of the required jobs is running
• dependent-job will only start once all required jobs have completed (success or failed).

Note: depending on the expected duration of the required jobs, you can specify a longer or shorter duration for the sleep command.

version: 2.1

jobs:
  upstream-job:
    docker: 
      - image: circleci/python
    steps:
      - run: echo "Hello I am '$CIRCLE_JOB', the upstream job"

  required-job:
    docker: 
      - image: circleci/node
    steps:
      - run: echo "Hello I am '$CIRCLE_JOB', a required job"

  other-required-job:
    docker: 
      - image: circleci/python
    steps:
      - run: echo "Hello I am '$CIRCLE_JOB', another required job"

  dependent-job:
    docker: 
      - image: circleci/php
    steps:
      - run: echo "Hello I am '$CIRCLE_JOB', the dependent job"

  waiter:
    docker: 
      - image: circleci/node
    steps:
      - run: |
          while [[ $(curl --location --request GET "https://circleci.com/api/v2/workflow/$CIRCLE_WORKFLOW_ID/job" --header "Circle-Token: $CCI_Token"| jq -r '.items[]|select(.name != "waiter")|.status' | grep -c "running") -gt 0 ]]
            do
              sleep 1
            done
      - run: echo "All required jobs have now completed"

workflows:
  my-workflow:
    jobs:
      - upstream-job
      - required-job:
          requires:
            - upstream-job
      - other-required-job:
          requires:
            - upstream-job
      - waiter
      - dependent-job:
          requires:
            - waiter

Caveats

• This method adds another job to your workflow.
• If any of the required jobs fails (or, if several or all of them fail) the dependent job will still run, however the workflow will be marked as failed.

Resources

Please see the related feature requests to add your vote and use-case:

• Allow job with dependencies to start even if some dependencies failed
• Option to allow failures in Fan-In/Out Workflow

[nicholaspshaw]

@yannCI has the implementation of this changed at all?

I need this exact functionality and have everything working as expected by overriding my command that can sometimes fail with a || true

This works as expected and I’m not trying to improve it using your method. From a high level things look okay but the waiter is executing and ending its job right away, when I check its output, it is:

jq: error (at <stdin>:2): Cannot iterate over null (null)
CircleCI received exit code 0

I assumed the URL or something had changed but when I do a simple curl, I’m getting the expected items array back, from the failure above I assumed it was empty. It seems this shouldn’t be the case, any pointers on debugging this?

[nicholaspshaw]

Seems my issue was where the CCI_TOKEN value was set.

I did this in my context as an ENV variable and I assumed this would be enough? I cannot find another way other than directly creating the environment variable in the waiter job. Is there another way @yannCI ? How exactly did you set it for your needs? I don’t really want my token in plaintext in our config

[Baptiste]

Hi Nicholas!
It might be a little late but just in case: I add the same issue as you (i.e. “Cannot iterate over null (null)”) and found out why. Iif, like I dit, you copy-pasted the code example, there is a small difference between the API token’s name as an env var and the way it’s used id the cose.

In the settings => CCI_Token

In the code (in the header of the curl command) => CCI_TOKEN (all caps)

Changing my code to CCI_Token made it work as expected.

Hope it’ll help!

[geeksam]

As I write this, it’s been five years and ten months since the first request for “Option to allow failures in Fan-In/Out Workflow” [apparently I can’t include links, but that’s at circleci DOT canny DOT io].

I’m trying to instrument a bloated test suite, so we can get some visibility into which parts we might get the most benefit from removing first. I’ve worked out how to persist files from each parallel job in the workspace and have a “fan-in” job compile the results from parallel jobs into one artifact at the end… but if any of our 60k(!) tests fail (NB: some of them are flaky) then that fan-in step doesn’t run.

Having to spin up a container and have it poll for 30-40 minutes while we run a LOT of other jobs in parallel seems like a crude workaround, but if that’s what works, I’ll hold my nose and code it. But…

If I use this technique, won’t it stop working when my personal API token expires? Is there a reason the V2 API doesn’t support project tokens?

[geeksam]

…well, turns out I don’t need an answer to that, because this workaround doesn’t appear to be helpful for my use case. Looks like a workspace is only available to downstream jobs via the requires feature, which means I can only ever collate/compile the stats I want on successful test runs.