Dynamically set parameter values of matrix

Build Environment
,
1

Hi,

My use case is this:
I run a script to generate an array of versions. I want these values to be used as parameters in a matrix so that I can run my test job for each one. I do not know the values before running the pipeline, the script generates them dynamically and they can change anytime as they are an external dependecy.

Let’s say ‘build-matrix’ is the job that runs the script and generates the version and ‘test’ is the job I want to run as a parameterised matrix.

I haven’t figured out how this is possible, as I assume the matrix parameters are expected to be set when the container starts, and the script in my other job (build-matrix) hasn’t run yet.

Is there any way to do this? An example mock snippet:

....
jobs:
  build-matrix:
    steps:
      - checkout
            - run:
          name: get versions
          command: | 
            TARGET_VERSIONS=$(script-to-generate-versions.sh)
            echo "export TARGET_VERSIONS='$TARGET_VERSIONS'" >> "$BASH_ENV"
  test:
    parameters:
      my_version:
        type: string
    - steps:
       ....

workflows:
  test:
    jobs:
      - build-matrix
      - test:
          name: test-<< matrix.my_version >>
          matrix:
            parameters:
              engine_version: $TARGET_VERSIONS
          requires:
            - build-matrix

....

We have achieved the same behaviour in Github workflows by setting the output of the first job in “outputs” and then using that in the strategy matrix.

1 Like
2

Sorry for the delay in a response @ipap

If I’m understanding what you’re wanting to do, you can do this with Matrix jobs, making sure to call the parameters and instantiate them as needed. Here are some things to use as examples and should get you down the line here:

3

Hi @jerdog , I managed to do this by using a dynamic config. I first run a script to generate the values I need and then pass them as env variables to a “generate-config” bash script. Then I expand those values in the matrix.

The tricky part is that the continuation config accepts just a json object, whereas I wanted to pass something like this: [2.3 , 5.6, 5.7, 8.0] .
A helpful video was this:

where they left the parameters as an empty object: parameters: {} and they passed the variable needed as a en env var to the bash script.

4

Yep that makes a lot of sense! Do you think you could share your config (sanitised from secrets, etc.) to give others an example?

5

Sure. It looks like this:

I have one “base” config: config.yml, where I used the continuation orb.

  1. I calculate the versions I need
  2. I save those in an env var and pass it into a bash script (can be any script)
  3. I use the continuation orb to specify the next config.
  • As you notice, I leave parameters with the default empty { } as my target versions are in the format of "[“v1.0.0”, “v1.2.1”, “v.1.3.1”] and parameters accepts only a JSON object. It didn’t make sense for me to make it a JSON object for this and it was easier to setup. (if you know a smarter way let me know!)
# this allows us to use CircleCI's dynamic configuration feature
setup: true

orbs:
  continuation: circleci/continuation@0.3.1

workflows:
  setup:
    jobs:
      - setup

jobs:
  setup:
    <<: *defaults
    resource_class: small
    steps:
      - checkout
      - run:
          name: calculate versions
          command: |
            # mock code to calculate TARGET_VERSIONS
            echo "export TARGET_VERSIONS='$TARGET_VERSIONS'" >> "$BASH_ENV"
      - run:
          name: Generate Pipeline continue_config.yml file
          command: |
            ./scripts/ci/generate_test_config.sh ${TARGET_VERSIONS}
      - continuation/continue:
          configuration_path: .circleci/continue_config.yml
          parameters: '{}'

Then in the continue_config.yml config I have everything I need, and in the place of the matrix, I have left an empty but it doesn’t matter what you put there as it will be replaced by the script:

workflows:
  test:
    jobs:
      - test:
          matrix:
            parameters:
              #              This value will be replaced by the generate_test_config.sh script in the CI
              target-version: [ ]
jobs:
  test:
    resource_class: medium
    parameters:
      target-version:
        type: string
    steps:
      ....

now in the bash script itself. you can do anything, I just used yq to edit the one line I needed in place:

#!/usr/bin/env bash
set -o errexit -o pipefail

target_versions="$TARGET_VERSIONS"

echo "Installing yq..."
curl -s -L -o yq "https://github.com/mikefarah/yq/releases/download/v4.31.2/yq_linux_amd64"
chmod +x ./yq
mkdir -p "${HOME}/.local/bin"
mv ./yq "${HOME}/.local/bin/"

# We use yq to replace the target-version key, this is the only edit in place we need
yq e ".workflows.test.jobs[0].test.matrix.parameters.\"target-version\" = $target_versions" -i .circleci/continue_config.yml

# (optionally install aws cli tool too and validate the config you generated)
# circleci config validate .circleci/continue_config.yml
6

@jerdog let me know if something doesn’t make sense, or can be refactored! Thanks

1 Like
7

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.