Trigger pipeline with API when using dynamic config

API Pipelines
1

We have a project that uses dynamic configs and the circleci/path-filtering@0.0.2 orb to do conditional builds in a monorepo. That works great.

Now we want to use the API to also trigger workflows from this project, with:

curl --location --request POST 'https://circleci.com/api/v2/project/github/org/repo/pipeline' \
--header 'Circle-Token: xxxxxxxxxxxxxxxxxxxxxxxxxxxx' \
--header 'Content-Type: application/json' \
--data-raw '{
    "parameters": {
        "my-pipeline-param": true
    }
}'

When I send this, I get a setup-pending response. Then in the CircleCI pipeline UI I get a build error of the form:

Unexpected argument(s): my-pipeline-param

Can these two features not be used together?

Here is a minimal example of how my CircleCI project is set up:

.circleci/config.yml

version: 2.1

setup: true

orbs:
  path-filtering: circleci/path-filtering@0.0.2

workflows:
  deploy-services:
    jobs:
    - path-filtering/filter:
        base-revision: main
        config-path: .circleci/deploy.yml
        mapping: |
          somedir/somefile.yml my-pipeline-param true

.circleci/deploy.yml

version: 2.1

parameters:
  my-pipeline-param:
    default: false
    type: boolean


workflows:
  my-workflow:
    jobs:
      - my-job
    when:
      equal: [ true, << pipeline.parameters.my-pipeline-param >> ]
2

Have you included my-pipeline-param in the parameters section of your config.yml? You need to define pipeline parameters in the config before you can use them. Maybe it isn’t defined? It would look like:

version: 2.1
parameters:
  my-pipeline-param:
    type: boolean
    default: false
3

It’s defined, but it’s not in config.yml. It’s defined in the config-path file that the path-filtering orb is using,

4

I’ve edited the original post to show how my project config is set up.

5

hmmm okay I think I see the issue. I think it’s the endpoint that you’re calling from within the setup workflow. In your initial example, you’re trying to kick off a pipeline using thehttps://circleci.com/api/v2/project/github/org/repo/pipeline endpoint. This would trigger a completely separate pipeline, rather than continuing the pipeline from your setup workflow. So I think it would be expecting the parameters you’ve defined to be in your setup config rather than the generated config.

The endpoint to continue a pipeline and trigger more workflows from within the setup workflow is different. It’s https://circleci.com/api/v2/pipeline/continue. Check out the API docs are here for the endpoint. You can see it has some input for the parameters that you have defined in your deploy.yml.

if you’re trying to continue the pipeline and call new workflows from the setup workflow you should use the continue API endpoint above. If you’re trying to launch a completely separate pipeline then you would use the API pipeline endpoint in your original post, but I think the parameters would need to be defined in your config.yml. does that make sense?

This post here may help break down what you need to do for continuing the pipeline. Check out the continuation orb example as that may make this easier for you and remove the need craft your own API calls.

6

That does make sense. But how do I get the continuation-key param required by the continuation API? From what I can tell that is injected into the env of my project when the setup workflow runs. But in this case I’m trying to use the API to trigger these workflows from other contexts.

7

I had to mess around with this a bit. I think I’m doing something wrong. Hopefully someone who understands this better can provide an improved answer. But this is what I got working. This lets you call the same pipeline from an existing pipeline, and then uses a different continuation config file to modify the result of the new pipeline.

So for example I have this setup config.yml:

version: 2.1

setup: true
 
orbs:
  continuation: circleci/continuation@0.1.2

parameters:
  setup-config-param:
    default: "THIS IS A SETUP CONFIG PARAM"
    type: string
  yml-file:
    default: "main.yml"
    type: string

jobs:
  not-setup:
    docker:
      - image: cimg/node:13.13
    steps:
      - run: echo << pipeline.parameters.setup-config-param >>
      - run: echo << pipeline.parameters.other >>

  setup:
    executor: continuation/default
    steps:
      - checkout
      - run: echo << pipeline.parameters.setup-config-param >>
      - continuation/continue:
          configuration_path: ".circleci/continue/<< pipeline.parameters.yml-file >>"

workflows:
  setup:
    jobs:
      - setup

And in this folder I have two continuation configs. main.yml and other.yml. You’ll notice that I have pipeline parameters defined in this config at the top. With a yml-file parameter that contains the string main.yml. This gets passed into the continuation config.

main.yml looks like this:

version: 2.1
jobs:
  job1:
    docker:
      - image: cimg/base:2021.04
    steps:
      - run: echo dynamic config!
      - run: echo ${CIRCLE_PROJECT_USERNAME} ${CIRCLE_PROJECT_REPONAME}
      - run: | 
              curl --location --request POST "https://circleci.com/api/v2/project/gh/${CIRCLE_PROJECT_USERNAME}/${CIRCLE_PROJECT_REPONAME}/pipeline" \
                    -u "${CIRCLE_TOKEN}:" \
                    --header 'Content-Type: application/json' \
                    --data-raw '{
                                  "parameters": {
                                    "setup-config-param": "this is a new string",
                                    "yml-file": "other.yml"
                                  }
                                }'
workflows:
  workflow1:
    jobs:
      - job1

It has your API call to trigger a completely new pipeline, and it modifies both parameters to use a different string for the test text and continue config file name. This way when the setup workflow runs it’s instructed to continue with other.yml instead of `main.yml.

This works. Below you can see job1 hit the API to start a new pipeline that ran otherjob

image
image931×547 40.3 KB

The one issue is that I needed to include the parameters from the config.yml file into other.yml:

version: 2.1

parameters:
  setup-config-param:
    default: "THIS IS A SETUP CONFIG PARAM"
    type: string
  yml-file:
    default: "main.yml"
    type: string

jobs:
  otherjob:
    docker:
      - image: cimg/base:2021.04
    steps:
      - run: echo OTHER JOB
workflows:
  workflow1:
    jobs:
      - otherjob

Otherwise, it threw an error about unexpected parameters. I may be doing something wrong. I’m probably overcomplicating this as well, but it works-ish.

8

Hi @dseravalli ,

I have came up with another way to go about this. Here are the config files I used:

config.yml

version: 2.1

setup: true

orbs:
  path-filtering: circleci/path-filtering@0.0.2
  continuation: circleci/continuation@0.1.2

parameters:
  my-pipeline-param-api:
    type: boolean
    default: false

workflows:
  deploy-services-filter:
    unless: << pipeline.parameters.my-pipeline-param-api >>
    jobs:
      - path-filtering/filter:
          base-revision: main
          config-path: .circleci/deploy.yml
          mapping: |
            somedir/somefile.yml my-pipeline-param true
  deploy-services-api:
    when: << pipeline.parameters.my-pipeline-param-api >>
    jobs:
      - continuation/continue:
          pre-steps:
            - checkout
          configuration_path: .circleci/deploy.yml
          parameters: '{"my-pipeline-param": true}'

deploy.yml

version: 2.1

parameters:
  my-pipeline-param:
    default: false
    type: boolean
  my-pipeline-param-api:
    default: false
    type: boolean
    
jobs:
  my-job:
    docker:
      - image: alpine
    steps:
      - run: echo "parameter was true"

workflows:
  my-workflow:
    jobs:
      - my-job
    when:
      equal: [ true, << pipeline.parameters.my-pipeline-param >> ]

A sample curl api request I used to test:

curl --location --request POST 'https://circleci.com/api/v2/project/github/ogii/test-both-continue/pipeline' \
--header "Circle-Token: $CIRCLECI_TOKEN" \
--header 'Content-Type: application/json' \
--data-raw '{
    "parameters": {
        "my-pipeline-param-api": true
    }
}'

where $CIRCLECI_TOKEN is my user api token.
This approach uses both of the orbs in the config.yml file.

There were a couple of things I had to adjust for when using both orbs:

  1. The parameter I am passing in (my-pipeline-param-api) had to be different than the name of the parameter I am passing in with the continuation/continue job.
  2. Both the parameter I am passing in with the API and the other parameter I am passing in with the orb jobs had to be declared in deploy.yml
  3. The checkout step is defined within the orb source for the continuation/continue job, but for some reason was not executing. I added it as a pre-step so that deploy.yml is available in the set-up container.

Please let us know if you were able to have any success using the approach Fernando or I suggested.

9

Hey @aaronclark I have tried this method and it does not work in this state. When using the example you provided, CircleCi throws the error Max number of workflows exceeded.

This currently appears to be a limitation of the dynamic configs Dynamic Configs - Max workflows

10

Hi @pmdebaets
I tested this again on my end, and multiple workflows are able to be defined within a set up config, but only one can be executed each time. The documentation example does not include a conditional statement, so both workflows would always be executed, leading to the error being thrown. I’ll update the article to include this clarification.

I create a sample open project with the above configs that you can reference here:
https://app.circleci.com/pipelines/github/ogii/test-dynamic-config

Please take a look and let me know if there is anything else I can help with.

Thank you!

1 Like