RFC: Matrix Jobs syntax

We’ve seen some demand for the ability to express “matrix” builds in CircleCI config, and we’re working on implementing it. We’d like to share our latest design to get a sense for how it fits into your use cases.

Our implementation of this feature builds on top of parameterized jobs. A “matrix” here is essentially syntactic sugar for invoking a certain job many times with different combinations of parameters.

jobs:
  test:
    parameters:
      os:
        type: string
      node-version:
        type: string
    steps:
      - ...

workflows:
  build:
    jobs:
      - test:
          matrix:
            parameters:
              os:
                - linux
                - macos
              node-version:
                - 6
                - 8
                - 10

This would turn into a workflow with six jobs.

There are several other facets of this syntax that are better explained in our more comprehensive config example: https://github.com/CircleCI-Public/pipeline-preview-docs/blob/master/docs/concepts/matrices.yml

Would this feature help you write maintainable configs? What alterations to the syntax would make it more intuitive / useful?

Thanks!

2 Likes

At pytorch/pytorch we have a few extra things we’d need to make this more useful:

  1. We have multiple matrices; for example, there is one matrix for Linux, and another for Windows. If you can only define a single top-level matrix that would be very troublesome
  2. Our matrix jobs are actually multiple jobs with dependencies on one another. Can that be expressed in this syntax?
  3. We sometimes want to “replicate” the matrix into multiple workflows, e.g., a workflow for PR builds and a workflow for nightlies (but with tweaks on parameters)

Here is an example, simplified CircleCI file that we’d be interested in using matrices on: https://github.com/pytorch/vision/blob/master/.circleci/config.yml#L294 and it would be interesting to see how this can be expressed in the new syntax.

It looks like the matrix would support more than 2 dimensions. Is that true? It would be useful.

I think this could be handled by invoking the same job multiple times with different matrices; for example:

# linux matrix
- test:
    os: linux
    matrix:
      parameters:
        ...
# windows matrix
- test:
    os: windows
    matrix:
      parameters:
        # different params
        ...

I don’t think either of these are handled by the current matrix syntax, but both are good ideas for extensions to consider in the future.

Yes, any number of dimensions is supported. You can also do a “one dimensional” matrix which could be useful if you have a long list of similar jobs.

This would be very useful if we were able to invert this current syntax and be able to specify multiple jobs within the matrix. I created a mockup of what this would look like here:

    jobs:
      # Special job name, everything in this block will be expanded
      - matrix:
        # Special name to designate matrix name
        - matrix_name: python_nightlies
        # Special name as well to name matrix parameters 
        - matrix_values:
          - python_version:
              - "3.5"
              - "3.6"
              - "3.7"
              - "3.8"
          - cuda_version:
              - "cpu"
              - "9.2"
              - "10.1"
              - "10.2"
        # Jobs would be anything that does not match our current special values
        - build:
            name: "build_python<< matrix_value.python_version >>_<< matrix_value.cuda_version >>"
            # Parameters could potentially be auto-filled based off of matrix_value
            # but it would also be good to be able to fill them ourselves
            python_version: "<< matrix_value.python_version >>"
            cuda_version: "<< matrix_value.cuda_version >>"
        - test:
            name: "test_python<<matrix_value.python_version>>_<< matrix_value.cuda_version >>"
            python_version: "<<matrix_value.python_version>>"
            cuda_version: "<< matrix_value.cuda_version >>"
            # Support for dependencies
            depends_on:  "build_python<< matrix_value.python_version >>_<< matrix_value.cuda_version >>"
        - upload:
            name: "upload_python<<matrix_value.python_version>>_<< matrix_value.cuda_version >>"
            python_version: "<<matrix_value.python_version>>"
            cuda_version: "<< matrix_value.cuda_version >>"
            # Support for a chain of dependencies
            depends_on:  "test_python<< matrix_value.python_version >>_<< matrix_value.cuda_version >>"
      - update_nightlies_html:
        # Only update html when all nightlies have finished uploading
        - requires: python_nightlies
Equivalent current config
    jobs:
      - build:
          name: "build_python3.5_cpu"
          python_version: "3.5"
          cuda_version: "cpu"
      - build:
          name: "build_python3.5_9.2"
          python_version: "3.5"
          cuda_version: "9.2"
      - build:
          name: "build_python3.5_10.1"
          python_version: "3.5"
          cuda_version: "10.1"
      - build:
          name: "build_python3.5_10.2"
          python_version: "3.5"
          cuda_version: "10.2"
      - build:
          name: "build_python3.6_cpu"
          python_version: "3.6"
          cuda_version: "cpu"
      - build:
          name: "build_python3.6_9.2"
          python_version: "3.6"
          cuda_version: "9.2"
      - build:
          name: "build_python3.6_10.1"
          python_version: "3.6"
          cuda_version: "10.1"
      - build:
          name: "build_python3.6_10.2"
          python_version: "3.6"
          cuda_version: "10.2"
      - build:
          name: "build_python3.7_cpu"
          python_version: "3.7"
          cuda_version: "cpu"
      - build:
          name: "build_python3.7_9.2"
          python_version: "3.7"
          cuda_version: "9.2"
      - build:
          name: "build_python3.7_10.1"
          python_version: "3.7"
          cuda_version: "10.1"
      - build:
          name: "build_python3.7_10.2"
          python_version: "3.7"
          cuda_version: "10.2"
      - build:
          name: "build_python3.8_cpu"
          python_version: "3.8"
          cuda_version: "cpu"
      - build:
          name: "build_python3.8_9.2"
          python_version: "3.8"
          cuda_version: "9.2"
      - build:
          name: "build_python3.8_10.1"
          python_version: "3.8"
          cuda_version: "10.1"
      - build:
          name: "build_python3.8_10.2"
          python_version: "3.8"
          cuda_version: "10.2"
      - test:
          name: "test_python3.5_cpu"
          python_version: "3.5"
          cuda_version: "cpu"
          depends_on: "build_python3.5_cpu"
      - test:
          name: "test_python3.5_9.2"
          python_version: "3.5"
          cuda_version: "9.2"
          depends_on: "build_python3.5_9.2"
      - test:
          name: "test_python3.5_10.1"
          python_version: "3.5"
          cuda_version: "10.1"
          depends_on: "build_python3.5_10.1"
      - test:
          name: "test_python3.5_10.2"
          python_version: "3.5"
          cuda_version: "10.2"
          depends_on: "build_python3.5_10.2"
      - test:
          name: "test_python3.6_cpu"
          python_version: "3.6"
          cuda_version: "cpu"
          depends_on: "build_python3.6_cpu"
      - test:
          name: "test_python3.6_9.2"
          python_version: "3.6"
          cuda_version: "9.2"
          depends_on: "build_python3.6_9.2"
      - test:
          name: "test_python3.6_10.1"
          python_version: "3.6"
          cuda_version: "10.1"
          depends_on: "build_python3.6_10.1"
      - test:
          name: "test_python3.6_10.2"
          python_version: "3.6"
          cuda_version: "10.2"
          depends_on: "build_python3.6_10.2"
      - test:
          name: "test_python3.7_cpu"
          python_version: "3.7"
          cuda_version: "cpu"
          depends_on: "build_python3.7_cpu"
      - test:
          name: "test_python3.7_9.2"
          python_version: "3.7"
          cuda_version: "9.2"
          depends_on: "build_python3.7_9.2"
      - test:
          name: "test_python3.7_10.1"
          python_version: "3.7"
          cuda_version: "10.1"
          depends_on: "build_python3.7_10.1"
      - test:
          name: "test_python3.7_10.2"
          python_version: "3.7"
          cuda_version: "10.2"
          depends_on: "build_python3.7_10.2"
      - test:
          name: "test_python3.8_cpu"
          python_version: "3.8"
          cuda_version: "cpu"
          depends_on: "build_python3.8_cpu"
      - test:
          name: "test_python3.8_9.2"
          python_version: "3.8"
          cuda_version: "9.2"
          depends_on: "build_python3.8_9.2"
      - test:
          name: "test_python3.8_10.1"
          python_version: "3.8"
          cuda_version: "10.1"
          depends_on: "build_python3.8_10.1"
      - test:
          name: "test_python3.8_10.2"
          python_version: "3.8"
          cuda_version: "10.2"
          depends_on: "build_python3.8_10.2"
      - upload:
          name: "upload_python3.5_cpu"
          python_version: "3.5"
          cuda_version: "cpu"
          depends_on: "test_python3.5_cpu"
      - upload:
          name: "upload_python3.5_9.2"
          python_version: "3.5"
          cuda_version: "9.2"
          depends_on: "test_python3.5_9.2"
      - upload:
          name: "upload_python3.5_10.1"
          python_version: "3.5"
          cuda_version: "10.1"
          depends_on: "test_python3.5_10.1"
      - upload:
          name: "upload_python3.5_10.2"
          python_version: "3.5"
          cuda_version: "10.2"
          depends_on: "test_python3.5_10.2"
      - upload:
          name: "upload_python3.6_cpu"
          python_version: "3.6"
          cuda_version: "cpu"
          depends_on: "test_python3.6_cpu"
      - upload:
          name: "upload_python3.6_9.2"
          python_version: "3.6"
          cuda_version: "9.2"
          depends_on: "test_python3.6_9.2"
      - upload:
          name: "upload_python3.6_10.1"
          python_version: "3.6"
          cuda_version: "10.1"
          depends_on: "test_python3.6_10.1"
      - upload:
          name: "upload_python3.6_10.2"
          python_version: "3.6"
          cuda_version: "10.2"
          depends_on: "test_python3.6_10.2"
      - upload:
          name: "upload_python3.7_cpu"
          python_version: "3.7"
          cuda_version: "cpu"
          depends_on: "test_python3.7_cpu"
      - upload:
          name: "upload_python3.7_9.2"
          python_version: "3.7"
          cuda_version: "9.2"
          depends_on: "test_python3.7_9.2"
      - upload:
          name: "upload_python3.7_10.1"
          python_version: "3.7"
          cuda_version: "10.1"
          depends_on: "test_python3.7_10.1"
      - upload:
          name: "upload_python3.7_10.2"
          python_version: "3.7"
          cuda_version: "10.2"
          depends_on: "test_python3.7_10.2"
      - upload:
          name: "upload_python3.8_cpu"
          python_version: "3.8"
          cuda_version: "cpu"
          depends_on: "test_python3.8_cpu"
      - upload:
          name: "upload_python3.8_9.2"
          python_version: "3.8"
          cuda_version: "9.2"
          depends_on: "test_python3.8_9.2"
      - upload:
          name: "upload_python3.8_10.1"
          python_version: "3.8"
          cuda_version: "10.1"
          depends_on: "test_python3.8_10.1"
      - upload:
          name: "upload_python3.8_10.2"
          python_version: "3.8"
          cuda_version: "10.2"
          depends_on: "test_python3.8_10.2"
      - update_nightlies_html:
        # Only update html when all nightlies have finished uploading
          - requires:
            - upload_python3.5_cpu
            - upload_python3.5_9.2
            - upload_python3.5_10.1
            - upload_python3.5_10.2
            - upload_python3.6_cpu
            - upload_python3.6_9.2
            - upload_python3.6_10.1
            - upload_python3.6_10.2
            - upload_python3.7_cpu
            - upload_python3.7_9.2
            - upload_python3.7_10.1
            - upload_python3.7_10.2
            - upload_python3.8_cpu
            - upload_python3.8_9.2
            - upload_python3.8_10.1
            - upload_python3.8_10.2

This same principle could also be applied to entire workflows.

Aside from this the proposed matrix format as it sits seems fine for simplistic workflows but declines in utility once you get into entire pipelines that depend on a matrix of parameters.