Do I have to duplicate jobs in config if the same job will be used in multiple places in a workflow?

Build Environment
1

Say I have two kinds of jobs in my standard workflow, a job that builds something, and a job that tests the thing that was just built. The build job has a step that takes a parameter, which affects what it builds. But the test job takes no parameters, it just reads an ID from a file in the workspace (which was persisted at the end of the build job), and tests that build which is identified by that ID.

In my config.yml I might have something like this for the build:

  build_someting_a: &buildsomething_a
    run:
      shell: /bin/bash +ex -uo pipefail
      command: |-
        bin/build_something.sh param-a

  build_someting_b: &buildsomething_c
    run:
      shell: /bin/bash +ex -uo pipefail
      command: |-
        bin/build_something.sh param-c

 tests:
      - run:
          shell: /bin/bash +ex -uo pipefail
          command: |-
            source BUILD_ID_FILE
            bin/run-tests.sh

cleanup:
  [...]

...
jobs:
 build_a:
    <<: *defaults
    steps:
      - checkout
      - *build_something_a
      - *persist

 build_b:
    <<: *defaults
    steps:
      - checkout
      - *build_something_b
      - *persist

 tests:
    <<: *defaults
    steps:
      - checkout
      - *tests

...

workflows:
  version: 2

  build_things:
    jobs:
      - build_a:
          <<: *filter

      - build_b:
          <<: *filter

What I’d like is for the workflow to run tests after each build, and then run a single cleanup after all of the tests jobs have completed. Something kind of like this, though this exactly as-is won’t work:

  - tests:
      <<: *filter
      requires:
        - build_a

  - tests:
      <<: *filter
      requires:
        - build_b

  - cleanup:
      <<: *filter
      requires:
        - [all tests]

Is there a way to do this?

  • Include the same job in different places in a workflow, with different dependencies, and have it read the persisted file from the job it depends on?
  • Have one job require all of the above, even if they have the same name?

Yes, I realize that in this toy example, you might be tempted to suggest that the test not be a separate job, just another step in the build job. In this contrived example, yes, that’s the reasonable thing to do, but that’s not the question I’m asking; I just made this minimal example to illustrate my more general question.

What I have been doing is duplicating all of the job configs, so that instead of tests I have tests_a, tests_b, etc, all of them with identical bodies, just different names. But imagine this with a) more builds than just _a and _b, and b) a lot more steps in each job, not just a single script. It adds up to a lot of duplication, makes it harder to make small updates since every change has to be made in all the copies, and creates a possibility of mistakenly failing to keep them consistent with each other and not noticing. So I’d like to collapse all that duplication, and define a job just once, even if it’s going to be used in multiple places in a workflow.

Can I do something like that?

1 Like
2

Hi @cos,

Have you considered reusable commands?

It allows you to define a sequence of steps as a map to be executed in a job; enabling you to reuse a single command definition across multiple jobs.

3

I had been using yaml anchors, but I will try reusable commands, it might be a bit nicer for some uses.

However, it still seems to be intended for situations where you have slightly different jobs that have a lot in common, and you need to use parameters to express the small differences between two mostly-similar jobs.

What I’m asking about is multiple instances of exactly identical jobs, so that I can stop littering my config with jobs named jobname-1, jobname-2, jobname-3, etc. which all have EXACTLY the same couple of lines - basically invoking the same yaml anchor or the same reusable command, without any parameters. I’d like to instead define the job just once, and then give it different instance-names in each workflow if it is used in that workflow multiple times.

4

Hi @cos,

I’m reviving this thread as I’m realizing matrix jobs might be the solution here.

Note: CircleCI Server v3.x supports matrix jobs

Here’s a config suggestion:

jobs:
  build:
    parameters:
      argument:
        type: string
        
    <<: *defaults
    steps:
      - checkout
      - run:
          shell: /bin/bash +ex -uo pipefail
          command: |-
              bin/build_something.sh << parameter.argument >>
      - *persist
  
  tests:
    parameters:
      argument:
        type: string

    <<: *defaults
    steps:
      - checkout
      - run:
          shell: /bin/bash +ex -uo pipefail
          command: |-
                source BUILD_ID_FILE
                bin/run-tests.sh

  cleanup:
  [...]

workflows:
  workflow:
    jobs:
      - build:
          name: Build << matrix.argument >>
          matrix:
            parameters:
              argument: ["arg_a", "arg_b"]
              
              
      - test:
          name: Test << matrix.argument >>
          matrix:
            parameters:
              argument: ["arg_a", "arg_b"]
          requires:
            - Build << matrix.argument >>
            
      - cleanup:
          requires:
            - test

 
Let me know if this works for you.

5

Thanks. CircleCI Server was still stuck on 2.x when I wrote this. However, CircleCI Server 3 finally arrived a couple of months ago (after I think at least two years of waiting!) and we just upgraded to it a couple of weeks ago. So now I can look at CircleCI 3.x features if issues like this come up in the future.

1 Like