Nested Dynamic Config

Build Environment
,
1

We have a mono-repo, every project has a different dev branch but a common prod branch(main). New features get added to dev branches, once ready will merge to main for prod deployment.
If a common static config file is used, the main branch will keep on changing as independent projects change the main branch frequently, and merge conflicts on the config.yml file will become a pain.

Heard about dynamic config generation using the setup workflows, we can have different config file per branch that removes the merge conflict as well as provides an abstraction to other projects <>.yml since each project has its own config file.
But Inside a project, we need to trigger workflows based on files for folder changes. Any idea on how to do this from generated config ?
I tried using swissknife/trigger-workflows-for-all-modified in the generated config , but I believe internally it’s using APIs to trigger workflows. I believe the orb always invokes workflows in config.yml at the root. So it tries to invoke workflows from generated config and checks in config.yml at the root which results in an error.

Can anyone please provide me any idea or suggestions on how to handle this scenario?

@roopakv @fernfernfern

2

Hi, yeah there should be a solution here. Have you seen the path filtering orb?

version: '2.1'
orbs:
  path-filtering: circleci/path-filtering@0.0.1
workflows:
  generate-config:
    jobs:
      - path-filtering/filter:
          base-revision: main
          config-path: .circleci/continue-config.yml
          mapping: |
            src/.* build-code true
            doc/.* build-docs true

I pulled that from the example. So basically you can have a generated config that uses a bunch of advanced logic. So in the example above the first mapping will fire if anything in the src/.* regex is changed. If it does then it passes pipeline parameter build-code a true value. You could imagine in the continue config there’s a workflow with an when attribute that’s set to false by default. But when path filtering orb detects a change in those folders then it changes it to true. And so a workflow for that path is kicked off from the continuation config.

version: '2.1'

parameters:
  build-code:
    type: boolean
    default: false
  build-docs:
    type: boolean
    default: false

workflows:
  build-code-workflow:
    when: << pipeline.parameters.build-code >>
    jobs:
      - build-code-job
  build-docs-workflow:
    when: << pipeline.parameters.build-docs >>
    jobs:
      - build-docs-job

So .circleci/continue-config.yml might look something like that. And so you can create all kinds of mapping and parameters to fire off different kinds of steps or workflows based on something changing in a directory. Does that work for you?

3

@bagio.johnson Have you checked out swissknife’s filter and continue.

What it seems like you are looking for, is to choose the config.yml based on the files modified instead?

What we do instead is have a giant yml which is generated from many small files.

If you think the pick the config rather than breaking up the yml works better for you, happy to find a way to make a new command / job for you on swissknife.

4

Yeah it seems like you want a file switch approach vs a combine small files to a large one approach.

We can definitely add this to swissknife. if you want to discuss exactly what will work, or drop an example of repo structure and what you need I can make a command.

5

First of all thanks to both of you for your quick replies.

Yes @fernfernfern , I have already seen this. What I tried to implement was that using dynamic config generator I will pick config based on branch(using continue orb in .circleci/config.yml) and then I tried using path filtering orb in my .circleci/continue-config.yml
where I mentioned the parameter config-path: circleci/continue-config.yml
This gave me

CIRCLE_CONTINUATION_KEY is required. Make sure setup workflows are enabled.’

.
I believe this is because I am using it on generated config rather than real config(nested dynamic config generation)

So I tried using swissknife then found out that there is no option specify the location of config file.

@roopakv I tried using filter and continue again got ‘CIRCLE_CONTINUATION_KEY is required. Make sure setup workflows are enabled.’

I believe the continuation token is not being passed from the generated config.

What I am trying to achieve is:
1. use dynamic config: → pick branch_config.yml
2. once control reaches branch_config.yml figure out which files/folders changed on changed branch using some mechanism(tried both swissknife and path-filtering)
3. trigger workflows based on changed files

6

your problem is that the first config picked is ALWAYS config.yml in the .circleci folder.

That can define what the next config is. If you want to jump on a zoom sometime shoot me a DM and we can talk thru it.

1 Like
7

Hello, want to follow up on this as I’m trying to achieve the same thing.

  1. I have my base-level config.yml file uses dynamic config (via continue orb) to generate a generated_config.yml file.
  2. I am then trying to run additional additional branching logic that will lead to a final continuation/continue on a 2nd generated config file.

On the 2nd continuation I am getting the “CIRCLE CONTINUATION KEY is required. Make sure setup workflows are enabled.” error message.

Can someone explain what worked above?