Technique for unified circleci config?

Build Environment
,
1

I believe the common approach toward ensuring a central (or main) circleci configuration is used for all pull request commits is to attempt to enforce only up to date branches (still relative to destination branch) can form pull requests.

Does anybody else have a clever way of essentially forcing the use of up-to-date configuration provided by a pinned or even a destination branch rather than from the base of the originating branch?

My goal here is to have a main configuration that is always preferred regardless of the originating branch.

1 Like
2

This sounds like a great use case for setup workflows. Simply put, it allows you to run a special workflow at the start of a pipeline run. In this special setup workflow you can do anything you want. It’s a normal CircleCI workflow. Run jobs. Make external API requests. Invoke some commands. Checkout some repos.

The magic here is that within this setup workflow you can continue the pipeline with a completely different config than the one checked into your repo. You can pull an existing config from a different branch, dynamically generate a completely new config, as well as pass along pipeline parameters to customize which logic statements are invoked during config compilation.

That completely different config used to continue the pipeline can come from anywhere. In your case, running a checkout and pulling it from a specific branch might be helpful. You could even use the GitHub API and pull it directly from there. You have a lot of options to control what happens when you decide the continue the pipeline beyond the setup workflow.

The process to “continue” a pipeline from a setup workflow is pretty simple. You make an API request from within a setup workflow that has two key ingredients.

  1. First a “continuation key” which is a token automatically injected into setup workflows with the CIRCLE_CONTINUATION_KEY envar. This connects any new workflows to the same pipeline of the setup workflow which invoked them.

  2. Then there’s the actual config that you want to be processed to continue the pipeline. You can also optionally pass pipeline parameters into this API request to further customize what is triggered based on the logic statements used within the config. So, you can roll your own API request to continue a pipeline from within setup workflow - or you can use the continuation orb.

Here’s an example using the continuation orb. The orb simplifies configuration of setup workflows and makes an example more readable. It’s not necessary to use but it’s helpful here.

version: 2.1

setup: true
 
orbs:
  continuation: circleci/continuation@0.1.2

jobs:
  setup:
    executor: continuation/default
    steps:
      - checkout
      - run:
          name: Generate config
          command: |
            cat > main.yml \<<- "EOF"
            version: 2.1
            jobs:
              job1:
                docker:
                  - image: cimg/base:2021.04
                steps:
                  - run: echo dynamic config!
            workflows:
             workflow1:
              jobs:
               - job1
            EOF
      - continuation/continue:
          configuration_path: main.yml

workflows:
  setup:
    jobs:
      - setup

So here we have an example of a setup-workflow. to use a setup workflow you need to enable the setup workflows preview option in your projects advanced settings. It’ll be at the bottom of the options list. Once that’s turned on you can add setup: true at the top level of your config. That will trigger CircleCI to create a setup workflow at the start of the pipeline. The continuation orb provides a small and simple base executor environment to run the setup workflow. You can see it in executor: continuation/default.

The setup workflow starts by running - checkout. This example doesn’t do anything with the checked out code. But in your case it could be followed by a git checkout main to access whatever config is in the main branch. In the example above we’re just creating a completely new basic CircleCI config and saving it as main.yml

The config:

           version: 2.1
            jobs:
              job1:
                docker:
                  - image: cimg/base:2021.04
                steps:
                  - run: echo dynamic config!
            workflows:
             workflow1:
              jobs:
               - job1

Here’s where a bit of orb magic helps out. You’ll see that after I run the command to generate a new config I’m calling a command from the orb named continue.

      - continuation/continue:
          configuration_path: main.yml

All i’m doing with this continue command is giving it a path to the config file that I want to continue the pipeline with. Which is a completely different config file from the one I started with, and it’s being generated right in the setup workflow job. Why no mention of the “continuation key”? The orb grabs it from the environment and adds it to the curl request automatically. It’s just running a curl command behind the scenes. The continue command from the orb also has an option to pass pipeline parameters into it. Like you would with the raw API request to trigger a pipeline.

You can see an example of a setup workflow config here in the UI. At the top of the config you can toggle between the setup workflow config, and the generated config used to continue the pipeline.

It’s kind of a new feature currently in open preview, so give it a shot and share any feedback you might have. Your situation is one of the many cases where setup workflows can come in handy.

1 Like