Intro to Dynamic Config via Setup Workflows

Dynamic Config via Setup Workflows Introduction

Dynamic Config

Your pipelines are no longer limited to running workflows defined in the config file committed to your VCS repo. Through the use of Setup Workflows and a setup config you can dynamically generate a config file at run time to run workflows in a pipeline. To do this you start by creating a setup config.

Enable Setup Workflows in Project Advanced Settings

You can enable Setup Workflows on your project by going to your project advanced settings and toggling the Enable dynamic config using setup workflows option. After you’ve toggled the option on in the settings you can create a new kind of config called a setup config.

The Setup Config

The setup config is mostly the same as a normal CircleCI config.yml. When adding a Setup Workflow to your project the setup config will replace your current config.yml. You also need to make sure to add the new top level key: setup: true

The new key sits at the same indentation as the version key:

version: 2.1

setup: true 

jobs:
 ...

The Setup Workflow

The setup config only contains a single workflow - the Setup Workflow. This is a special workflow that always runs at the start of a pipeline. You can run several jobs. Make external API requests in those jobs. Invoke some commands. Checkout some repos. All the good times you normally have when setting up your ci. It’s a regular CircleCI workflow.

Continuation Key & Generated config

The difference with a Setup Workflow from a regular workflow is that we add a special key as an envar. The CIRCLE_CONTINUATION_KEY envar will let you continue the pipeline with a new config by passing the key and config file to the continuation API endpoint (https://circleci.com/api/v2/pipeline/continue).

Here’s what it looks like in the UI when you run a pipeline with a Setup Workflow:

You can see it’s a single pipeline with a Setup Workflow, and the workflow it continued on with. The setup workflow in this pipeline was used to trigger workflow1. The config file for workflow1 was created inside the Setup Workflow from a run step.

Setup Workflow live example

You can see here what a setup config looks like in the UI. Use the toggle at the top of the page to switch between the setup config, and the generated config. A Setup Workflow doesn’t need to continue with a new Workflow. It can decide there’s no work to be done and end the pipeline. You have up to six hours to continue the pipeline after the Setup Workflow starts.

ENDLESS

POSSIBILITIES

The purpose of the Setup Workflow is so you can generate the config that you want to run without being limited to what is committed in your repo. In fact, you could store a few different generated config files in your .circleci folder, and configure the Setup Workflow decide which one to use at the start of the pipeline. It doesn’t matter where the new config comes from, or how you create it. The setup workflow enables your pipeline to programmatically decide what it should do next. You have a wide range of freedom to make your pipelines dynamic with a Setup Workflow.

SERIOUSLY

ENDLESS

Unlock the power of advanced logic statements

You might not be aware, but this is a great time to remind you we offer advanced logic statements that when combined with conditional workflows and conditional steps give you extensive control over which workflows and steps run when the config is processed. But for a long time there was no way for your pipeline to be responsive and make adjustments to the values of these statements at run time.

To make use of the advanced logic you have access to pipeline variables. And you can define your own pipeline parameters to use within the logic statements as well.

But without Setup Workflows you can’t modify the parameter values when triggering pipelines without using the pipeline API. The Setup Workflow continuation endpoint also accepts pipeline parameters like the pipeline API endpoint. Allowing you to dynamically adjust the values of your pipeline parameters during a pipeline run, which then cascades down into the logic statements used throughout your generated config.

This allows you to maintain a single generated config file that has many different outcomes based on the state of the pipeline parameters being used in the config logic. The ability to programmatically decide what those parameter values should be before the config is processed gives you far more flexibility in deciding which workflows and steps are run any time a pipeline is triggered.

How to continue a Setup Workflow

The process to “continue” a pipeline from a Setup Workflow is pretty simple. You make an API request from within a Setup Workflow that has the two key ingredients mentioned previously:

  1. The “continuation key” which is a token automatically injected into Setup Workflows as the CIRCLE_CONTINUATION_KEY envar. This key is what connects any new workflows to the same pipeline as the Setup Workflow which triggered them. You pass it through to the API like you would any other API key.

  2. The generated config that you want to be processed to continue the pipeline. You can use command line tools, use a config from your repo, or even make a web request to pull a config. As long as you’re passing a valid CircleCI config.yml it will be processed and run as it would from pushing a commit.

  3. (optional) You can also pass pipeline parameters into this API request to further customize what is triggered based on the logic statements used within the config.

All of that gets sent to the continuation endpoint (https://circleci.com/api/v2/pipeline/continue) which will trigger new workflows within the pipeline.

The Continuation Orb

You can roll your own API request to continue a pipeline from within a Setup Workflow, or you can use the continuation orb.

The orb simplifies configuration of Setup Workflows and makes an example more readable. It’s not necessary to use it, but it’s helpful here. You can view the source of the continue command to see how the raw API request is made by the orb if you’d. like to hack your own. Here’s an example using the continuation orb :

version: 2.1

setup: true
 
orbs:
  continuation: circleci/continuation@0.1.2

jobs:
  setup:
    executor: continuation/default
    steps:
      - checkout
      - run:
          name: Generate config
          command: |
            cat > main.yml \<<- "EOF"
            version: 2.1
            jobs:
              job1:
                docker:
                  - image: cimg/base:2021.04
                steps:
                  - run: echo dynamic config!
            workflows:
             workflow1:
              jobs:
               - job1
            EOF
      - continuation/continue:
          configuration_path: main.yml

workflows:
  setup:
    jobs:
      - setup

Above you can see an example of a setup config with the continuation orb. The continuation orb provides a small and simple base executor environment to run the setup workflow. You can see it applied to the job as executor: continuation/default.

The Setup Workflow starts by running - checkout. This example doesn’t do anything with the checked out code. But it’s there to demonstrate you have access to all the existing CircleCI commands. In the example above we’re just creating a completely new basic CircleCI config file right from a run step and saving it as main.yml

The config:

           version: 2.1
            jobs:
              job1:
                docker:
                  - image: cimg/base:2021.04
                steps:
                  - run: echo dynamic config!
            workflows:
             workflow1:
              jobs:
               - job1

Here’s where the orb magic helps out. You’ll see that after I run the command to generate a new config I’m calling a command from the orb named continue.

      - continuation/continue:
          configuration_path: main.yml

All i’m doing with this continue command is giving it a path to the continue config file that I want to continue the pipeline with. Which is the config that I generated in thee run step of the setup config.

Why no mention of the “continuation key”? The orb grabs it from the environment and adds it to the curl request automatically. The continue command from the orb also has an option to pass pipeline parameters into it.

Use your imagination

Setup workflows is a feature that solves many problems, and opens up so many opportunities. It’s one of those features where you might not realize how useful it is until you encounter some obscure problem it might solve. Like enforcing your pipeline to always use the config file from a specific branch. It was also designed to help solve some common needs for developers.

Monorepos

The repository here shows an example of combining the path filtering orb with a monorepo. Where each module in the repo has it’s own unique CircleCI config. Only modules with changes will have their module level config merged into the generated config to trigger work on that module. This also demonstrates splitting up the config and transforming it into the complete generated config.

Try it out!

Dynamic Config via Setup Worfklows is available as a preview, and now you know what they are and how to use them! Do check out the continuation Orb so that you can quickly make a working setup config. And make sure to read about the current limitations and other technical info in our preview docs and discuss release.

5 Likes