Setup Workflows Open Preview

[thekatertot]

Setup workflows are a new functionality we are introducing to help customers better handle their unique workflows.

What are setup workflows?

A setup workflow is a single workflow that can be used to “set up” a pipeline. It does this by using CircleCI jobs to generate or set a pipeline config and/or pipeline parameters. This enables two main use cases:

• Dynamic config: If a customer can create a script that generates a valid config at runtime, that can be used as the pipeline config
• Pre-build steps: If a customer needs to run any sort of pre-build step that does analysis to determine what to build at runtime.

One of the areas where we anticipate setup workflows being useful is in helping customers with monorepos. Different jobs can be used for solving various monorepo use cases like running git commands to analyze what files have been changed, combining multiple config files, and doing complex dependency analysis using existing build tools like Bazel. These different functions can be bundled into orbs that can be easily reused. We will be starting by releasing an orb for path filtering to set parameters based on which files have been modified in a push to filter which workflows run.

How do they work?

1. The .circleci/config.yml file will contain a single workflow with the “setup” parameter set to true. This is the setup workflow

2. (Optional) Some of the jobs within the setup workflow may generate pipeline parameters. This can be done by running user developed scripts that export these parameters as a JSON file

3. Some of the jobs within the setup workflow will generate the config. This can be done in three main ways

4. A user script within a job that programmatically generates the config from scratch

5. Pre-made config files could be placed within a file structure. If there are more than one config, a job might be required to collect/concatenate these files. If only a single file is required, then no jobs would be required.

6. A hybrid of both options.

7. At the end of the workflow, a job from the continue orb should be called. This job should take the parameters file, config file, and a continuation key generated as part of the setup workflow process as inputs and call a continuation API. This will compile the config and append it to the pipeline with the submitted parameters. (The current path filtering orb does this automatically)

8. The appended pipeline will continue running.

How do I use a setup workflow?

1. Enable the Setup Workflows setting to true (it is under “Advanced” in the project settings)
2. (Optional) Create a new branch for testing the setup workflow
3. (Optional) Duplicate your current config to another file.
4. Update your .circleci/config.yml to contain a setup workflow. This workflow will contain jobs that generate config/parameters
5. (Optional) If you are setting parameters in the config, make sure any generated config has the parameters defined.
6. Commit/push this branch and the setup workflows will start running
7. (Optional) If you created a branch for setup workflows, you can merge it into your default branches.

More technical documentation can be found here

Orb Docs and orb source

How do I set up path filtering?

1. Follow the processes for setting up a setup workflow.
2. In the .circleci/config.yml, use the path filtering orb
3. Use the path filter job and set up the mappings of regexes for modified files to pipeline parameters
4. (Optional) Make sure the config file being set has the pipeline parameters defined, either in the file that is selected or in the scripts to generate it
5. Make sure the generated config uses these pipeline parameters to filter what runs.

More technical documentation can be found here

Orb Docs and orb source

FAQs

I thought pipeline parameters could only be used with the API?

Normally this is true. But using setup workflows will allow you to set parameters when you trigger from both API or webhook (VCS push)

What is the “continuation” orb/job?

The setup workflow is used to generate parameters and config, but we need a way to signal when this work is completed and what to actually append to the pipeline. There is a Continue API that accepts the config as parameters and completes the append. The job/orb are a way of wrapping these efficiently so they are easy to use.

What is the difference between the “sandbox” and “circleci” orbs for both continuation and path filtering?

The “sandbox” versions were our experimental versions of the orbs used earlier in the preview. These orbs will no longer be receiving updates. The “circleci” orbs are where we are currently pushing all updates. If you are using code that uses the “sandbox” orbs, we recommend updating to the “circleci” orbs.

[mqchau]

Good news. I have been looking forwards to this since 2018.

[mattfnsidc]

Hello,

This new feature in version 2.1 has broken our configuration files that were pinned to 2 or 2.0. We use the setup key as a YAML anchor to DRY out our job configurations, which was accepted in v2 prior to v2.1 being released. It seems the v2.0 spec has been retroactively changed with the release of v2.1 to forbid the setup key. The error: Version 2.1 is required for Setup workflows. This happened in the middle of a workday, so one minute a build was passing, and the next it was failing, and required me to stop what I was doing to fix the problem. I would not expect this unless I opted in to the new version.

I’ve worked around the issue by renaming the key to x-setup for one job, but there remain many left to fix that I thought were safely pinned and protected against breaking changes. Is CircleCI going to make an imminent fix to avoid breaking changes to v2.0 workflows, or should we go ahead and update all of those jobs?

Thank you for your time!
Matt

[kbeamnsidc]

This breaks configs which happen to have a yaml node named setup. In the example below we have a shared yaml node.

version: 2

setup: &setup
  docker:
    - image: docker:17.05.0-ce-git
  working_directory: ~/fortytwopy-svc

jobs:
  build:
    <<: *setup
    steps:
    ...

[salil]

Hi Matt, we’re investigating to find the best way to address this issue. I should have an update on what we will be doing in the next day. However, I would recommend making this change regardless as it will be required in future versions of the config specification. Making these changes now will make it easier to transition to newer versions in the future. – Salil

[mattfnsidc]

Thank you for investigating, salil. We are making our best effort to stay abreast of ecosystem changes, but we do have dozens of previously-valid projects using CircleCI which now can’t build. We’d prefer for their builds to continue to work until we opt each project in to version 2.1 of the configuration format. Making the change now would be a good amount of unplanned time away from our scheduled feature work.

[salil]

Hi Matt, we’re working on a fix that should resolve this for customers using 2.0 config. I’ll post another update once that’s live.

[mattfnsidc]

Thank you, Salil!

[salil]

Hi Matt, we pushed a change to resolve this. You shouldn’t have any issues unless you try to use setup workflows, in which case it will require 2.1 config.

[mattfnsidc]

Wonderful, thank you for your support!

[jakeatoms]

setup is such a generic term that I can’t imagine the issues this will cause once rolled out. Normally, breaking changes would be gated by the version (ie, 2.1 → 3.0). I think being more specific here would be better with something like pipeline_setup.

[salil]

To clarify, we are not limiting the term setup in general, just reserving it as a top level key in the config.yml. Using setup in your parameters, commands, jobs, or workflows should be unaffected.