RFC: Matrix Jobs syntax

Build Environment
,
7

This might not be appropriate for the MVP, but one really cool feature that Azure has is the ability to generate the matrix dynamically. Essentially you run a step that spews out some JSON and then reference that output later.

This is documented here - https://docs.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml#multi-job-configuration (BTW, it was really hard for me to find this doc page again so don’t copy Azure when it comes to docs :wink:

Here’s an example:

    jobs:
      - job: GenerateMatrix
        displayName: Generate the matrix of Perl versions to build
        pool:
          vmImage: ubuntu-18.04
        steps:
          - template: templates/deploy/install-perl.yml
            parameters:
              perlbrew_root: $(PERLBREW_ROOT)
          - bash: |
              set -eo pipefail
              set -x
              $(PERLBREW_ROOT)/bin/perlbrew exec --with 5.30.1 perl ./deploy/bin/print-perls-matrix.pl | tee ./deploy-matrix
              perls=$( cat ./deploy-matrix )
              set +x
              echo "##vso[task.setVariable variable=perls;isOutput=true]$perls"
            name: matrixGenerator
            displayName: Generate perl version matrix

      - job: BuildOneImage
        displayName: Deploy the runtime-perl images
        pool:
          vmImage: ubuntu-18.04
        dependsOn: GenerateMatrix
        strategy:
          matrix: $[ dependencies.GenerateMatrix.outputs['matrixGenerator.perls'] ]
        steps:
          - template: templates/deploy/install-perl.yml
            parameters:
              perlbrew_root: $(PERLBREW_ROOT)
          - bash: |

To summarize what’s happening. The job “GenerateMatrix” contains one step which in turn has a few jobs. One of those jobs invokes a Perl script to generate a test matrix as a JSON object. Then that object is put in a variable that’s available to all future parts of the pipeline. The matrix itself is based on finding all the currently available stable Perl versions as well as the latest dev release. This means that whenever this runs it will be up to date with Perl’s release cycle, without me having to manually update version numbers anywhere.*

Then in the “BuildOneImage” job we set our matrix based on that output. The end result is we run the “BuildOneImage” job once for each version of Perl that was found in the “GenerateMatrix” job.

This is an incredibly powerful feature that, AFAICT, is unique to Azure Pipelines among SaaS products. Of course, I’m sure you could do this with things like Jenkins or TeamCity, which let you inject arbitrary code into the process, but there’s a reason I prefer SaaS tools for CI.

3 Likes