Help with matrix: unexpected arguments(s)

Build Environment
1

I seem to be hitting a wall I can’t figure out. I’m trying to add a matrix to a CircleCI workflow but it just doesn’t like me. My config (partially) looks like:

version: 2.1

executors:
  gfortran:
    docker:
      - image: gmao/ubuntu20-geos-env-mkl:v6.1.0-openmpi_4.0.5-gcc_10.2.0
        auth:
          username: $DOCKERHUB_USER
          password: $DOCKERHUB_AUTH_TOKEN
    environment:
      OMPI_ALLOW_RUN_AS_ROOT: 1
      OMPI_ALLOW_RUN_AS_ROOT_CONFIRM: 1
      OMPI_MCA_btl_vader_single_copy_mechanism: none
    resource_class: xlarge
  ifort:
    docker:
      - image: gmao/ubuntu20-geos-env:v6.1.0-intelmpi_2021.2.0-intel_2021.2.0
        auth:
          username: $DOCKERHUB_USER
          password: $DOCKERHUB_AUTH_TOKEN
    resource_class: xlarge

workflows:
  build-and-test:
    jobs:
      - build-MAPL:
          name: build-MAPL-<< matrix.executor >>
          matrix:
            parameters:
              executor: [gfortran, ifort]
          context:
            - docker-hub-creds
      - run-MAPL-tests:
          name: run-MAPL-tests-<< matrix.executor >>
          matrix:
            parameters:
              executor: [gfortran, ifort]
          context:
            - docker-hub-creds
          requires:
            - build-MAPL-<< matrix.executor >>

jobs:
  build-MAPL:
    executor: << matrix.executor >>
    working_directory: /root/project
    steps:
      - checkout:
          path: MAPL
      - run:
          name: "Versions etc"
          command: mpirun --version && << matrix.executor >> --version && echo $BASEDIR && pwd && ls && echo "$(nproc)"
...
  run-MAPL-tests:
    executor: << matrix.executor >>
    working_directory: /root/project
    steps:
      - attach_workspace:
          at: workspace
...

But CircleCI isn’t happy:

❯ circleci config validate .circleci/config.yml
Error: Error calling workflow: 'build-and-test'
Error calling job: 'build-MAPL'
Unexpected argument(s): executor

I was following along echoing the example code in the docs:

workflows:
  workflow:
    jobs:
      - build:
          name: build-v<< matrix.version >>
          matrix:
            parameters:
              version: ["0.1", "0.2"]
      - deploy:
          name: deploy-v<< matrix.version >>
          matrix:
            parameters:
              version: ["0.1", "0.2"]
          requires:
            - build-v<< matrix.version >>

Can you help me figure out my mistake?

2

Replying to myself. A couple of issues. One, apparently for requires: that “space” after the colon is…required. Without it, the YAML is wonky.

Then, later on in the jobs: section, I had to add:

jobs:
  build-MAPL:
    parameters:
      compiler:
        type: string

and then refer to parameters.compiler:

    executor: << parameters.compiler >>

Yaml is fun.

3

Hi @mathomp4,

Thanks for bringing up this interesting question; it led me to look into a parameter type I rarely get a chance to use, namely the executor type.

There are a couple of things I’d like to point out in your initial config.yml:

  1. The error you initially mentioned was occurring because the parameter named executor specified in the matrix jobs had not previously been defined.
    As you’ve figured out, you need to declare said parameter in the jobs respective declaration.

  2. The line executor: << matrix.executor >> would also generate an Unknown variable(s) error because the job declaration should be independent of how it’ll be used downstream.

  3. The command << matrix.executor >> --version would fail for 2 reasons:

    • As above, the use of << matrix.executor >> would lead to an Unknown variable(s) error
    • Even with the use of an appropriate parameter, the command << matrix.executor >> --version (or let’s say << parameters.compiler >> --version) would fail; both gfortran and ifort are defined as executors, so the command would be interpolated with the whole block defining gfortran and ifort which wouldn’t make sense, and cause a command not found error.

  4. As I initially hinted at, you can use the executor type for your jobs’ parameter. Please find below a configuration suggestion:

version: 2.1

executors:
  gfortran:
    docker:
      - image: gmao/ubuntu20-geos-env-mkl:v6.1.0-openmpi_4.0.5-gcc_10.2.0
        auth:
          username: $DOCKERHUB_USER
          password: $DOCKERHUB_AUTH_TOKEN
    environment:
      OMPI_ALLOW_RUN_AS_ROOT: 1
      OMPI_ALLOW_RUN_AS_ROOT_CONFIRM: 1
      OMPI_MCA_btl_vader_single_copy_mechanism: none
    resource_class: xlarge

  ifort:
    docker:
      - image: gmao/ubuntu20-geos-env:v6.1.0-intelmpi_2021.2.0-intel_2021.2.0
        auth:
          username: $DOCKERHUB_USER
          password: $DOCKERHUB_AUTH_TOKEN
    resource_class: xlarge

workflows:
  build-and-test:
    jobs:
      - build-MAPL:
          name: build-MAPL-<< matrix.compiler >>
          matrix:
            parameters:
              compiler: [gfortran, ifort]
          context:
            - docker-hub-creds
      - run-MAPL-tests:
          name: run-MAPL-tests-<< matrix.compiler >>
          matrix:
            parameters:
              compiler: [gfortran, ifort]
          context:
            - docker-hub-creds
          requires:
            - build-MAPL-<< matrix.compiler >>

jobs:
  build-MAPL:
    parameters:
      compiler:
        type: executor
    executor: << parameters.compiler >>
    working_directory: /root/project
    steps:
      - checkout:
          path: MAPL
      - run:
          name: "Versions etc"
          command: mpirun --version && echo $BASEDIR && pwd && ls && echo "$(nproc)"
...
  run-MAPL-tests:
    parameters:
      compiler:
        type: executor
    executor: << parameters.compiler >>
    working_directory: /root/project
    steps:
      - attach_workspace:
          at: workspace
...

 
Regarding your remark about the space after requires:, I can confirm this is actually not required, but it could be that you fixed an incorrect indentation at the same time; YAML is quite sensitive to indentation :confused:

Let me know if any of the above helps.

4

@yannCI Actually I made it a string because it was the “easy” way I could think of telling CMake what compiler to use:

... -DCMAKE_Fortran_COMPILER=<< parameters.compiler >> ...

But I now realize I could just set an environment variable in the executor and key off of that.

1 Like
5

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.