On_fail for commands in steps?

Build Environment Running Tests
1

All,

Some of my CI has been having intermittent build failures that I think are due to a combination of parallel make and limited memory. Now I could move to a larger resource class, but that would double my cost for about a 10% speed improvement by my testing (well, not counting the fact I might have to press the “re-run tests” again).

So, I thought about using when: on_fail for my setup to re-run a build in case of failure (since then it might work).

Currently I like to use commands and my config looks like:

  build-GEOSgcm:
    parameters:
      compiler:
        type: string
    executor: << parameters.compiler >>
    working_directory: /root/project
    steps:
      - checkout_fixture:
          repo: GEOSgcm
      - mepoclone:
          repo: GEOSgcm
      - mepodevelop:
          repo: GEOSgcm
      - checkout_mapl_branch:
          repo: GEOSgcm
      - cmake:
          repo: GEOSgcm
          compiler: << parameters.compiler >>
      - buildinstall:
          repo: GEOSgcm
      - compress_artifacts
      - store_artifacts:
          path: /logfiles

And I thought, great I’ll just make a new command and do:

...
      - buildinstall:
          repo: GEOSgcm
      - retrybuild:
          repo: GEOSgcm
          when: on_fail
      - compress_artifacts
      - store_artifacts:
          path: /logfiles

But that failed:

❯ circleci config validate
Error: Error calling workflow: 'build-and-test'
Error calling job: 'build-GEOSgcm'
Error calling command: 'retrybuild'
Unexpected argument(s): when

as I figured once I looked at the docs.

So for now, in my buildinstall command, I have an extra run: step with a when: on_fail trigger:

  buildinstall:
    description: "Build and install"
    parameters:
      repo:
        type: string
        default: ""
    steps:
      - run:
          name: "Build and install"
          command: |
            cd ${CIRCLE_WORKING_DIRECTORY}/workspace/build-<< parameters.repo >>
            make -j"$(nproc)" install |& tee /logfiles/make.log
      - run:
          name: "Retry build and install after failure"
          command: |
            cd ${CIRCLE_WORKING_DIRECTORY}/workspace/build-<< parameters.repo >>
            make -j"$(nproc)" install |& tee /logfiles/make-retry.log
          when: on_fail

That validates and, well, now I’m waiting to see if the failure is helped by this.

But I had a question: is there a way to do what I wanted with the command in steps syntax? Or is just that steps have no when?

Thanks,
Matt

PS: Soon I’ll be thinking about using orbs since the commands in that fragment you see up there are used in a few repos. I think orbs would let me share these commands easily…right? :slight_smile:

2

Hi @mathomp4 ,

Thank you for sharing your question on our forum!

I would have to see your full config.yml to understand why your first attempt was not working, but I created the following sample config file that shows how to use when inside of a command:

version: 2.1

commands:
  retrybuild:
    description: Retry a step when it fails
    parameters:
      repo:
        type: string
        default: ""
      when:
        type: string

    steps:
      - run: 
          command: |
            echo "This ran because the previous step failed"
            echo "<<parameters.when>>"
          name: Rerun step due to failure
          when: <<parameters.when>>

jobs:
  build:
    docker:
      - image: cimg/base:2021.07
    steps:
      - checkout
      - run: echo "test"
      - run: exit 1
      - retrybuild:
          repo: "TEST"
          when: "on_fail"

workflows:
  workflow1:
    jobs:
      - build

I am defining a when parameter (this could be named anything) inside of the retrybuild command, which then I am passing into the similarly-named when key. I did not set a default value for this command, so config validation will fail if no value is passed in.

With when: "on_fail" , the retrybuild step will execute because the previous step is exiting with status 1.

This would be a great chance to use orbs. It would allow you to create reusable code, and cleanup your config.yml file as the extra code would be imported via the orb.

Our Slack orb actually is a good example of using conditional steps (with the notify command). You can click the following link to see the source:

Let me know if this clears things up, or if you have any further questions.

3

Ahh. Okay. You are passing down when I was hoping I could use when directly on the command in jobs/steps. I guess when: is only something that run can see, not something that a command step can see.

4

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.