Reuse of workflows in `config.yml`

Build Environment
1

Following up on an earlier question on reuse of enum definitions (Reusing enum type definitions across multiple commands/jobs in a single `config.yml`), I have a more ambitious question on reusing workflow definitions.

The config.yml in question has a number of copies of the same workflow for building and deploying to a particular environment. These workflows have the same structure, differing only in the value used for the environment parameter in a few of the steps.

Are there any reccomended ways for reducing this sort of duplicate code? I know there the configuration SDK, which allows programmatic configuration, but it would require a fair amount of rework of our existing config.yml to convert it into that form.

Thanks in advance!

2

You may need to explain your config in more detail.

A workflow causes one or more jobs to be executed and these in turn can be expressed as one or more commands. It is possible to re-use both jobs and commands within a config.yml as they can accept parameters. As you have noted being able to drive a workflow with parameters is more complicated as you need to control/set such parameters externally via something like the API or the continuation ORB.

If you have all the work expressed at the workflow level your first step is going to be to refactor your code so the work is defined as a job instead. You can then pass distinct parameter values from the workflow to the job.

Depending on how much work would be needed to refactor your code you should also read up on the dynamic configuration/continuation ORB. This is an advanced feature of the CircleCI environment that allows a controlling config.yml file to be defined that will then call the script in your current config.yml with a set of parameters that are defined at runtime.

The result is that one workflow defines the values to be passed on to another workflow, without the need to use the API.

As an example of workflow/job structure, this is part of one of my config.yml files

workflows:
  Backend-dev1:
    when:
      or:
        - matches: { pattern: "^[0-9a-zA-Z]+.ci-dev1" , value:  << pipeline.git.tag >> }
    jobs:
      - build-runner-latest-from-branch:
          doppler_token: "dp.st.base_dev1.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
          <<: *tag_filters

  Backend-dev2:
    when:
      or:
        - matches: { pattern: "^[0-9a-zA-Z]+.ci-dev2" , value:  << pipeline.git.tag >> }
    jobs:
      - build-runner-latest-from-branch:
          doppler_token: "dp.st.base_dev2.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
          <<: *tag_filters

  Backend-production:
    when:
      or:
        - matches: { pattern: "^[0-9a-zA-Z]+.ci-prod" , value:  << pipeline.git.tag >> }
    jobs:
      - build-runner-latest-from-branch:
          doppler_token: "dp.st.base_production.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
          <<: *tag_filters

The workflow is branching based on the tag assigned to the changed repo branch and is used to decide if the target is dev1,dev2, or prod. Based on the choice a token is passed onto the build-runner-latest-from-branch job as a parameter which is where my code is placed (you will also see my use of anchors to keep the boilerplate code under control).

Side note - doppler is a third-party secret store where I hold rather long lists of parameters used for each configuration, otherwise my parameter list would be far to long and very repetitive.

1 Like
3

Thanks… I’ve thought about something similar, but had the concern that you lose some amount of parallelism. Our deployment workflows are structured in a way that lets us overlap execution of jobs and run several at the same time. We can factor these out to jobs easily, but at the expense of the parallelism offered by the workflows, no?

4

One complication is the terminology used within CircleCI, which can make things a little more complicated when exchanging messages. When it comes to CircleCI the term ‘parallelism’ refers to the number of parallel instances of a single job - think splitting a set of tests to allow them to execute in parallel. While ‘Concurrency’ is used to indicate the number of different jobs that a workflow will create in parallel, with each job having a different role within the workflow.

In general, if you are using workflows to create a large amount of concurrency it would be best to look at dynamic configuration as this allows you to pass parameters chosen at runtime to the workflow. You can then pass the parameters down to the different jobs without changing the structure of your workflow.

This is not something I have yet had to do as my workflows are currently still at the point of performing at an OK speed without any level of additional parallelism or concurrency.

1 Like