RFC: Matrix Jobs syntax

Build Environment
,
#1

We’ve seen some demand for the ability to express “matrix” builds in CircleCI config, and we’re working on implementing it. We’d like to share our latest design to get a sense for how it fits into your use cases.

Our implementation of this feature builds on top of parameterized jobs. A “matrix” here is essentially syntactic sugar for invoking a certain job many times with different combinations of parameters.

jobs:
  test:
    parameters:
      os:
        type: string
      node-version:
        type: string
    steps:
      - ...

workflows:
  build:
    jobs:
      - test:
          matrix:
            parameters:
              os:
                - linux
                - macos
              node-version:
                - 6
                - 8
                - 10

This would turn into a workflow with six jobs.

568×598 26.6 KB

There are several other facets of this syntax that are better explained in our more comprehensive config example: https://github.com/CircleCI-Public/pipeline-preview-docs/blob/master/docs/concepts/matrices.yml

Would this feature help you write maintainable configs? What alterations to the syntax would make it more intuitive / useful?

Thanks!

4 Likes
#2

At pytorch/pytorch we have a few extra things we’d need to make this more useful:

  1. We have multiple matrices; for example, there is one matrix for Linux, and another for Windows. If you can only define a single top-level matrix that would be very troublesome
  2. Our matrix jobs are actually multiple jobs with dependencies on one another. Can that be expressed in this syntax?
  3. We sometimes want to “replicate” the matrix into multiple workflows, e.g., a workflow for PR builds and a workflow for nightlies (but with tweaks on parameters)

Here is an example, simplified CircleCI file that we’d be interested in using matrices on: https://github.com/pytorch/vision/blob/master/.circleci/config.yml#L294 and it would be interesting to see how this can be expressed in the new syntax.

#3

It looks like the matrix would support more than 2 dimensions. Is that true? It would be useful.

1 Like
#4

I think this could be handled by invoking the same job multiple times with different matrices; for example:

# linux matrix
- test:
    os: linux
    matrix:
      parameters:
        ...
# windows matrix
- test:
    os: windows
    matrix:
      parameters:
        # different params
        ...

I don’t think either of these are handled by the current matrix syntax, but both are good ideas for extensions to consider in the future.

#5

Yes, any number of dimensions is supported. You can also do a “one dimensional” matrix which could be useful if you have a long list of similar jobs.

#6

This would be very useful if we were able to invert this current syntax and be able to specify multiple jobs within the matrix. I created a mockup of what this would look like here:

    jobs:
      # Special job name, everything in this block will be expanded
      - matrix:
        # Special name to designate matrix name
        - matrix_name: python_nightlies
        # Special name as well to name matrix parameters 
        - matrix_values:
          - python_version:
              - "3.5"
              - "3.6"
              - "3.7"
              - "3.8"
          - cuda_version:
              - "cpu"
              - "9.2"
              - "10.1"
              - "10.2"
        # Jobs would be anything that does not match our current special values
        - build:
            name: "build_python<< matrix_value.python_version >>_<< matrix_value.cuda_version >>"
            # Parameters could potentially be auto-filled based off of matrix_value
            # but it would also be good to be able to fill them ourselves
            python_version: "<< matrix_value.python_version >>"
            cuda_version: "<< matrix_value.cuda_version >>"
        - test:
            name: "test_python<<matrix_value.python_version>>_<< matrix_value.cuda_version >>"
            python_version: "<<matrix_value.python_version>>"
            cuda_version: "<< matrix_value.cuda_version >>"
            # Support for dependencies
            depends_on:  "build_python<< matrix_value.python_version >>_<< matrix_value.cuda_version >>"
        - upload:
            name: "upload_python<<matrix_value.python_version>>_<< matrix_value.cuda_version >>"
            python_version: "<<matrix_value.python_version>>"
            cuda_version: "<< matrix_value.cuda_version >>"
            # Support for a chain of dependencies
            depends_on:  "test_python<< matrix_value.python_version >>_<< matrix_value.cuda_version >>"
      - update_nightlies_html:
        # Only update html when all nightlies have finished uploading
        - requires: python_nightlies
Equivalent current config 
    jobs:
      - build:
          name: "build_python3.5_cpu"
          python_version: "3.5"
          cuda_version: "cpu"
      - build:
          name: "build_python3.5_9.2"
          python_version: "3.5"
          cuda_version: "9.2"
      - build:
          name: "build_python3.5_10.1"
          python_version: "3.5"
          cuda_version: "10.1"
      - build:
          name: "build_python3.5_10.2"
          python_version: "3.5"
          cuda_version: "10.2"
      - build:
          name: "build_python3.6_cpu"
          python_version: "3.6"
          cuda_version: "cpu"
      - build:
          name: "build_python3.6_9.2"
          python_version: "3.6"
          cuda_version: "9.2"
      - build:
          name: "build_python3.6_10.1"
          python_version: "3.6"
          cuda_version: "10.1"
      - build:
          name: "build_python3.6_10.2"
          python_version: "3.6"
          cuda_version: "10.2"
      - build:
          name: "build_python3.7_cpu"
          python_version: "3.7"
          cuda_version: "cpu"
      - build:
          name: "build_python3.7_9.2"
          python_version: "3.7"
          cuda_version: "9.2"
      - build:
          name: "build_python3.7_10.1"
          python_version: "3.7"
          cuda_version: "10.1"
      - build:
          name: "build_python3.7_10.2"
          python_version: "3.7"
          cuda_version: "10.2"
      - build:
          name: "build_python3.8_cpu"
          python_version: "3.8"
          cuda_version: "cpu"
      - build:
          name: "build_python3.8_9.2"
          python_version: "3.8"
          cuda_version: "9.2"
      - build:
          name: "build_python3.8_10.1"
          python_version: "3.8"
          cuda_version: "10.1"
      - build:
          name: "build_python3.8_10.2"
          python_version: "3.8"
          cuda_version: "10.2"
      - test:
          name: "test_python3.5_cpu"
          python_version: "3.5"
          cuda_version: "cpu"
          depends_on: "build_python3.5_cpu"
      - test:
          name: "test_python3.5_9.2"
          python_version: "3.5"
          cuda_version: "9.2"
          depends_on: "build_python3.5_9.2"
      - test:
          name: "test_python3.5_10.1"
          python_version: "3.5"
          cuda_version: "10.1"
          depends_on: "build_python3.5_10.1"
      - test:
          name: "test_python3.5_10.2"
          python_version: "3.5"
          cuda_version: "10.2"
          depends_on: "build_python3.5_10.2"
      - test:
          name: "test_python3.6_cpu"
          python_version: "3.6"
          cuda_version: "cpu"
          depends_on: "build_python3.6_cpu"
      - test:
          name: "test_python3.6_9.2"
          python_version: "3.6"
          cuda_version: "9.2"
          depends_on: "build_python3.6_9.2"
      - test:
          name: "test_python3.6_10.1"
          python_version: "3.6"
          cuda_version: "10.1"
          depends_on: "build_python3.6_10.1"
      - test:
          name: "test_python3.6_10.2"
          python_version: "3.6"
          cuda_version: "10.2"
          depends_on: "build_python3.6_10.2"
      - test:
          name: "test_python3.7_cpu"
          python_version: "3.7"
          cuda_version: "cpu"
          depends_on: "build_python3.7_cpu"
      - test:
          name: "test_python3.7_9.2"
          python_version: "3.7"
          cuda_version: "9.2"
          depends_on: "build_python3.7_9.2"
      - test:
          name: "test_python3.7_10.1"
          python_version: "3.7"
          cuda_version: "10.1"
          depends_on: "build_python3.7_10.1"
      - test:
          name: "test_python3.7_10.2"
          python_version: "3.7"
          cuda_version: "10.2"
          depends_on: "build_python3.7_10.2"
      - test:
          name: "test_python3.8_cpu"
          python_version: "3.8"
          cuda_version: "cpu"
          depends_on: "build_python3.8_cpu"
      - test:
          name: "test_python3.8_9.2"
          python_version: "3.8"
          cuda_version: "9.2"
          depends_on: "build_python3.8_9.2"
      - test:
          name: "test_python3.8_10.1"
          python_version: "3.8"
          cuda_version: "10.1"
          depends_on: "build_python3.8_10.1"
      - test:
          name: "test_python3.8_10.2"
          python_version: "3.8"
          cuda_version: "10.2"
          depends_on: "build_python3.8_10.2"
      - upload:
          name: "upload_python3.5_cpu"
          python_version: "3.5"
          cuda_version: "cpu"
          depends_on: "test_python3.5_cpu"
      - upload:
          name: "upload_python3.5_9.2"
          python_version: "3.5"
          cuda_version: "9.2"
          depends_on: "test_python3.5_9.2"
      - upload:
          name: "upload_python3.5_10.1"
          python_version: "3.5"
          cuda_version: "10.1"
          depends_on: "test_python3.5_10.1"
      - upload:
          name: "upload_python3.5_10.2"
          python_version: "3.5"
          cuda_version: "10.2"
          depends_on: "test_python3.5_10.2"
      - upload:
          name: "upload_python3.6_cpu"
          python_version: "3.6"
          cuda_version: "cpu"
          depends_on: "test_python3.6_cpu"
      - upload:
          name: "upload_python3.6_9.2"
          python_version: "3.6"
          cuda_version: "9.2"
          depends_on: "test_python3.6_9.2"
      - upload:
          name: "upload_python3.6_10.1"
          python_version: "3.6"
          cuda_version: "10.1"
          depends_on: "test_python3.6_10.1"
      - upload:
          name: "upload_python3.6_10.2"
          python_version: "3.6"
          cuda_version: "10.2"
          depends_on: "test_python3.6_10.2"
      - upload:
          name: "upload_python3.7_cpu"
          python_version: "3.7"
          cuda_version: "cpu"
          depends_on: "test_python3.7_cpu"
      - upload:
          name: "upload_python3.7_9.2"
          python_version: "3.7"
          cuda_version: "9.2"
          depends_on: "test_python3.7_9.2"
      - upload:
          name: "upload_python3.7_10.1"
          python_version: "3.7"
          cuda_version: "10.1"
          depends_on: "test_python3.7_10.1"
      - upload:
          name: "upload_python3.7_10.2"
          python_version: "3.7"
          cuda_version: "10.2"
          depends_on: "test_python3.7_10.2"
      - upload:
          name: "upload_python3.8_cpu"
          python_version: "3.8"
          cuda_version: "cpu"
          depends_on: "test_python3.8_cpu"
      - upload:
          name: "upload_python3.8_9.2"
          python_version: "3.8"
          cuda_version: "9.2"
          depends_on: "test_python3.8_9.2"
      - upload:
          name: "upload_python3.8_10.1"
          python_version: "3.8"
          cuda_version: "10.1"
          depends_on: "test_python3.8_10.1"
      - upload:
          name: "upload_python3.8_10.2"
          python_version: "3.8"
          cuda_version: "10.2"
          depends_on: "test_python3.8_10.2"
      - update_nightlies_html:
        # Only update html when all nightlies have finished uploading
          - requires:
            - upload_python3.5_cpu
            - upload_python3.5_9.2
            - upload_python3.5_10.1
            - upload_python3.5_10.2
            - upload_python3.6_cpu
            - upload_python3.6_9.2
            - upload_python3.6_10.1
            - upload_python3.6_10.2
            - upload_python3.7_cpu
            - upload_python3.7_9.2
            - upload_python3.7_10.1
            - upload_python3.7_10.2
            - upload_python3.8_cpu
            - upload_python3.8_9.2
            - upload_python3.8_10.1
            - upload_python3.8_10.2

This same principle could also be applied to entire workflows.

Aside from this the proposed matrix format as it sits seems fine for simplistic workflows but declines in utility once you get into entire pipelines that depend on a matrix of parameters.

#7

This might not be appropriate for the MVP, but one really cool feature that Azure has is the ability to generate the matrix dynamically. Essentially you run a step that spews out some JSON and then reference that output later.

This is documented here - https://docs.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml#multi-job-configuration (BTW, it was really hard for me to find this doc page again so don’t copy Azure when it comes to docs :wink:

Here’s an example:

    jobs:
      - job: GenerateMatrix
        displayName: Generate the matrix of Perl versions to build
        pool:
          vmImage: ubuntu-18.04
        steps:
          - template: templates/deploy/install-perl.yml
            parameters:
              perlbrew_root: $(PERLBREW_ROOT)
          - bash: |
              set -eo pipefail
              set -x
              $(PERLBREW_ROOT)/bin/perlbrew exec --with 5.30.1 perl ./deploy/bin/print-perls-matrix.pl | tee ./deploy-matrix
              perls=$( cat ./deploy-matrix )
              set +x
              echo "##vso[task.setVariable variable=perls;isOutput=true]$perls"
            name: matrixGenerator
            displayName: Generate perl version matrix

      - job: BuildOneImage
        displayName: Deploy the runtime-perl images
        pool:
          vmImage: ubuntu-18.04
        dependsOn: GenerateMatrix
        strategy:
          matrix: $[ dependencies.GenerateMatrix.outputs['matrixGenerator.perls'] ]
        steps:
          - template: templates/deploy/install-perl.yml
            parameters:
              perlbrew_root: $(PERLBREW_ROOT)
          - bash: |

To summarize what’s happening. The job “GenerateMatrix” contains one step which in turn has a few jobs. One of those jobs invokes a Perl script to generate a test matrix as a JSON object. Then that object is put in a variable that’s available to all future parts of the pipeline. The matrix itself is based on finding all the currently available stable Perl versions as well as the latest dev release. This means that whenever this runs it will be up to date with Perl’s release cycle, without me having to manually update version numbers anywhere.*

Then in the “BuildOneImage” job we set our matrix based on that output. The end result is we run the “BuildOneImage” job once for each version of Perl that was found in the “GenerateMatrix” job.

This is an incredibly powerful feature that, AFAICT, is unique to Azure Pipelines among SaaS products. Of course, I’m sure you could do this with things like Jenkins or TeamCity, which let you inject arbitrary code into the process, but there’s a reason I prefer SaaS tools for CI.

#8

Hi!

This is awesome to have this feature (we have long configurations in our project and most of it is basically manual matrix definition :slight_smile: - we were really waiting for this).

All in all it seems that we can improve our configurations with this feature, however, there are questions regarding how “require” works with matrixes. Only example I’ve found is here: https://github.com/CircleCI-Public/pipeline-preview-docs/blob/master/docs/concepts/matrices.yml#L56

Let’s say that we have workflow which at first builds for all architectures and then tests all architectures.
If we apply the proposal “as is”, then it would mean that we have two matrixes: “build_all” and “test_all”.

Here we have some questions:

  1. is it possible to require matrix from other matrix, i.e.:
    "build_all" matrix is required to finish before we run "test_all" matrix

  2. how to require specific job in require for the whole matrix?
    Let’s say that we have common preparation job “prepare_build_and_test”, then:
    "prepare_build_and_test" job is required to finish before we run "build_all" matrix

  3. requiring job from matrix1 with same parameters for other job from matrix2 to start:
    I.e., we would like to have opportunity to have flow like that:
    build_x86 -> test_x86
    build_armv7 -> test_armv7
    test_x86 & test_armv7 -> publish
    i.e. we would like to define matrixes and then build flow around the same parameters (for the flow common is parameter value, but jobs are from different matrixes)

Aleksandrs

1 Like
#9

Hi!

In addition to @aivanovs questions I have these:

  1. Would it possible to exclude some combinations in a matrix? Let’s say I want to build some things for different platforms, different architecture and different flavours. But I want to exclude from execution exactly one (or two) combination from this matrix.
  2. What is the maximum size of a matrix? How many jobs we could have?

Ilia.

#10

Thanks to everyone who took a look at this and added their feedback. Reading through these ideas, I noticed two big takeaways:

1: We want to generate job-to-job dependencies via a matrix.

Experienced configists already leveraging CircleCI workflows to orchestrate a chain of jobs want to still benefit from the syntactic sugar of matrices. To accommodate this, we’re adding a couple of new capabilities. First, you can template out the requires and name fields of any matrix job, effectively letting you replicate a whole flow of work:

  - build:
      matrix:
        parameters:
          os: [linux, macos]
          node_version: [6, 8, 10]
      name: build-<< matrix.os >>-<< matrix.node_version >>
  - test:
      matrix:
        parameters:
          os: [linux, macos]
          node_version: [6, 8, 10]
      name: test-<< matrix.os >>-<< matrix.node_version >>
      # Templated dependency:
      # "Each `test` job depends on its corresponding `build` job"
      requires:
        - build-<< matrix.os >>-<< matrix.node_version >>

Additionally, a single job can depend on a whole matrix of jobs, if you want to wait until every case has been thoroughly tested before doing something else.

  - test:
      matrix:
        ...
  - deploy:
      requires:
        # `deploy` won't start until every job in the `test` matrix passes
        - test

2: In general, we want to generate CircleCI config more dynamically.

We can keep adding new capabilities to our YAML syntax, but we’ll never be able to satisfy every use case people come up with (props to @autarch for the interesting Perl example!). How we can give more power and flexibility back to the developers when those complicated cases arise? That’s a separate discussion we’re having, and there are ideas already on the table such as Setup Jobs or investing further into our existing Pipeline Variables.

Conclusion

I’ve updated our detailed config examples file on GitHub to reflect the above changes, which I encourage you to read if you’re interested in the full suite of functionality we’re aiming for (for example, excluding specific combinations will be supported!).

I’m closing this thread for now, but I will resurface once we have a real implementation to share with you all.

Thanks again,
–Alex

#11
#12

Another quick matrix update: To prepare for our work on implementing the matrix jobs functionality, matrix is now a “reserved keyword” in CircleCI Config Version 2.1; this won’t affect you unless you already have a parameterized job with a parameter called matrix. If you happen to encounter any config compilation errors because of this, it can be addressed by renaming the parameter to something else.

Thanks!

#13

We have shipped matrix support. Read more about it here.

Happy building!

1 Like