Dynamic Config via Setup Workflows Introduction
Dynamic Config
Your pipelines are no longer limited to running workflows defined in the config file committed to your VCS repo. Through the use of Setup Workflows and a setup config you can dynamically generate a config file at run time to run workflows in a pipeline. To do this you start by creating a setup config.
Enable Setup Workflows in Project Advanced Settings
You can enable Setup Workflows on your project by going to your project advanced settings and toggling the Enable dynamic config using setup workflows option. After youâve toggled the option on in the settings you can create a new kind of config called a setup config.
The Setup Config
The setup config is mostly the same as a normal CircleCI config.yml. When adding a Setup Workflow to your project the setup config will replace your current config.yml. You also need to make sure to add the new top level key: setup: true
The new key sits at the same indentation as the version key:
version: 2.1
setup: true
jobs:
...
The Setup Workflow
The setup config only contains a single workflow - the Setup Workflow. This is a special workflow that always runs at the start of a pipeline. You can run several jobs. Make external API requests in those jobs. Invoke some commands. Checkout some repos. All the good times you normally have when setting up your ci. Itâs a regular CircleCI workflow.
Continuation Key & Generated config
The difference with a Setup Workflow from a regular workflow is that we add a special key as an envar. The CIRCLE_CONTINUATION_KEY envar will let you continue the pipeline with a new config by passing the key and config file to the continuation API endpoint (https://circleci.com/api/v2/pipeline/continue).
Hereâs what it looks like in the UI when you run a pipeline with a Setup Workflow:
You can see itâs a single pipeline with a Setup Workflow, and the workflow it continued on with. The setup workflow in this pipeline was used to trigger workflow1. The config file for workflow1 was created inside the Setup Workflow from a run step.
Setup Workflow live example
You can see here what a setup config looks like in the UI. Use the toggle at the top of the page to switch between the setup config, and the generated config. A Setup Workflow doesnât need to continue with a new Workflow. It can decide thereâs no work to be done and end the pipeline. You have up to six hours to continue the pipeline after the Setup Workflow starts.
ENDLESS
POSSIBILITIES
The purpose of the Setup Workflow is so you can generate the config that you want to run without being limited to what is committed in your repo. In fact, you could store a few different generated config files in your .circleci folder, and configure the Setup Workflow decide which one to use at the start of the pipeline. It doesnât matter where the new config comes from, or how you create it. The setup workflow enables your pipeline to programmatically decide what it should do next. You have a wide range of freedom to make your pipelines dynamic with a Setup Workflow.
SERIOUSLY
ENDLESS
Unlock the power of advanced logic statements
You might not be aware, but this is a great time to remind you we offer advanced logic statements that when combined with conditional workflows and conditional steps give you extensive control over which workflows and steps run when the config is processed. But for a long time there was no way for your pipeline to be responsive and make adjustments to the values of these statements at run time.
To make use of the advanced logic you have access to pipeline variables. And you can define your own pipeline parameters to use within the logic statements as well.
But without Setup Workflows you canât modify the parameter values when triggering pipelines without using the pipeline API. The Setup Workflow continuation endpoint also accepts pipeline parameters like the pipeline API endpoint. Allowing you to dynamically adjust the values of your pipeline parameters during a pipeline run, which then cascades down into the logic statements used throughout your generated config.
This allows you to maintain a single generated config file that has many different outcomes based on the state of the pipeline parameters being used in the config logic. The ability to programmatically decide what those parameter values should be before the config is processed gives you far more flexibility in deciding which workflows and steps are run any time a pipeline is triggered.
How to continue a Setup Workflow
The process to âcontinueâ a pipeline from a Setup Workflow is pretty simple. You make an API request from within a Setup Workflow that has the two key ingredients mentioned previously:
-
The âcontinuation keyâ which is a token automatically injected into Setup Workflows as the
CIRCLE_CONTINUATION_KEYenvar. This key is what connects any new workflows to the same pipeline as the Setup Workflow which triggered them. You pass it through to the API like you would any other API key. -
The generated config that you want to be processed to continue the pipeline. You can use command line tools, use a config from your repo, or even make a web request to pull a config. As long as youâre passing a valid CircleCI
config.ymlit will be processed and run as it would from pushing a commit. -
(optional) You can also pass pipeline parameters into this API request to further customize what is triggered based on the logic statements used within the config.
All of that gets sent to the continuation endpoint (https://circleci.com/api/v2/pipeline/continue) which will trigger new workflows within the pipeline.
The Continuation Orb
You can roll your own API request to continue a pipeline from within a Setup Workflow, or you can use the continuation orb.
The orb simplifies configuration of Setup Workflows and makes an example more readable. Itâs not necessary to use it, but itâs helpful here. You can view the source of the continue command to see how the raw API request is made by the orb if youâd. like to hack your own. Hereâs an example using the continuation orb :
version: 2.1
setup: true
orbs:
continuation: circleci/continuation@0.1.2
jobs:
setup:
executor: continuation/default
steps:
- checkout
- run:
name: Generate config
command: |
cat > main.yml \<<- "EOF"
version: 2.1
jobs:
job1:
docker:
- image: cimg/base:2021.04
steps:
- run: echo dynamic config!
workflows:
workflow1:
jobs:
- job1
EOF
- continuation/continue:
configuration_path: main.yml
workflows:
setup:
jobs:
- setup
Above you can see an example of a setup config with the continuation orb. The continuation orb provides a small and simple base executor environment to run the setup workflow. You can see it applied to the job as executor: continuation/default.
The Setup Workflow starts by running - checkout. This example doesnât do anything with the checked out code. But itâs there to demonstrate you have access to all the existing CircleCI commands. In the example above weâre just creating a completely new basic CircleCI config file right from a run step and saving it as main.yml
The config:
version: 2.1
jobs:
job1:
docker:
- image: cimg/base:2021.04
steps:
- run: echo dynamic config!
workflows:
workflow1:
jobs:
- job1
Hereâs where the orb magic helps out. Youâll see that after I run the command to generate a new config Iâm calling a command from the orb named continue.
- continuation/continue:
configuration_path: main.yml
All iâm doing with this continue command is giving it a path to the continue config file that I want to continue the pipeline with. Which is the config that I generated in thee run step of the setup config.
Why no mention of the âcontinuation keyâ? The orb grabs it from the environment and adds it to the curl request automatically. The continue command from the orb also has an option to pass pipeline parameters into it.
Use your imagination
Setup workflows is a feature that solves many problems, and opens up so many opportunities. Itâs one of those features where you might not realize how useful it is until you encounter some obscure problem it might solve. Like enforcing your pipeline to always use the config file from a specific branch. It was also designed to help solve some common needs for developers.
Monorepos
The repository here shows an example of combining the path filtering orb with a monorepo. Where each module in the repo has itâs own unique CircleCI config. Only modules with changes will have their module level config merged into the generated config to trigger work on that module. This also demonstrates splitting up the config and transforming it into the complete generated config.
Try it out!
Dynamic Config via Setup Worfklows is available as a preview, and now you know what they are and how to use them! Do check out the continuation Orb so that you can quickly make a working setup config. And make sure to read about the current limitations and other technical info in our preview docs and discuss release.