Run all test steps even if one fails

circle.yml
2.0

#1

As part of testing, I use multiple test steps. Example tiny file:

version: 2
jobs:
  build:
    working_directory: ~/foo
    docker:
      - image: ruby:2.3.3
    steps:
      - checkout
      - run: bundle check --path=vendor/bundle || bundle install --path=vendor/bundle --jobs=4 --retry=3  # must pass to proceed
      - run: bin/rubocop # TestA: current: bails on fail
      - run: bin/rspec # TestB: expected: this should run in addition, even on failure of rubocop

As you can see, bundler must pass before I try to do anything else.

In Circle 1.0, the test override allowed all steps to succeed or fail independently, then it would fail the job as a whole at the end when there’s >0 failures. I’m looking for a way to have similar behavior with Circle 2.0.

Hints and tips?


Run multiple tests and report on all that fail
Run multiple tests and report on all that fail
#2

You can set +e to prevent ending the build on a failure, but that does mean you could get a green build by accident. This is what I mean:

set +e
bin/rubocop
bin/rspec

The build would only fail if bin/rspec fails, not /bin/rubocop.


#3

@CJBridges another option you can use is parallelization. You can run your build in two parallel containers and manually specify in your config what to run in each container:

version: 2
jobs:
  parallelism: 2
  build:
    working_directory: ~/foo
    docker:
      - image: ruby:2.3.3
    steps:
      - checkout
      - run: bundle check --path=vendor/bundle || bundle install --path=vendor/bundle --jobs=4 --retry=3  # must pass to proceed
      - run: |
          if [[ "${CIRCLE_NODE_INDEX}" == 0 ]]
          then
             bin/rubocop # TestA: current: bails on fail
          fi
      - run: 
          if [[ "${CIRCLE_NODE_INDEX}" == 1 ]]
          then
             bin/rspec # TestB: expected: this should run in addition, even on failure of rubocop
          fi

In this case:

  • bundler will be executed in both containers.
  • bin/rubocop in the first container only if bundler succeed
  • bin/rspec in the second container only if bundler succeed and no matter the bin/rubocop
  • the whole build will be successful only if both bin/rubocop and bin/rspec succeed

#4

So the two solutions aren’t ideal:
Regarding @rohara - I want to bomb out on failures properly. If I set +e, then I need to build my own handling for failures. Your approach (presuming you were suggesting putting this all into one shell script) also loses support for separating results from different build commands in separate UI containers.

Regarding @smaantci’s solution:
This works fine if I
a) want my build to take multiple containers (I don’t - my build takes 21s on Circle 2.0). In addition, for longer running builds, I want to leverage my containers for splitting test load, not working around error handling.
b) Want to limit myself to N test steps, where N is the number of containers I am using. A different build in question uses ~8 test steps.

Prototyped out something that works a lot like the old behavior:

- run: bundle check --path=vendor/bundle || bundle install --path=vendor/bundle --jobs=4 --retry=3
- run: bin/rubocop || export SOME_TEST_FAILED=true
- run: bin/rspec || export SOME_TEST_FAILED=true
- run: test -n "$SOME_TEST_FAILED" && false

It unfortunately only flags the final step as an error.

This really does need support from the Circle team to support my use case properly as I need to notify circle there was a failure, but still proceed.

The problem I’d like to solve is: As a CircleCI customer, I’d like the ability to configure a test block (a la steps) or attribute on a run step that a non-zero exit status does not prevent the build from continuing to run other test steps so that I can see if multiple failures have affected my build.

Sample .yml files:

version: 2
jobs:
  build:
    working_directory: ~/foo
    docker:
      - image: ruby:2.3.3
    steps:
      - checkout
      - run: bundle check --path=vendor/bundle || bundle install --path=vendor/bundle --jobs=4 --retry=3
      - test_steps: # can run and fail, but do not halt the build
        - run: bin/rubocop
        - run: bin/rspec
version: 2
jobs:
  build:
    working_directory: ~/foo
    docker:
      - image: ruby:2.3.3
    steps:
      - checkout
      - run: bundle check --path=vendor/bundle || bundle install --path=vendor/bundle --jobs=4 --retry=3  # must pass to proceed
      - run:
          halt_build_on_fail: false  # default true
          command: bin/rubocop
      - run:
          halt_build_on_fail: false  # default true
          command: bin/rspec

Support for multiple phases, with configurable error behaviour
#5

Just to be super explicit, the behavior I’m looking for is the one we’ve been using from Circle 1.0 here:

all test commands will run, even if one fails. This allows our test output to tell you about all the tests that fail, not just the first error.


#6

What about creating a wrapper shell script that doesn’t allow the error to propagate out?

bin/exec.sh

#!/usr/bin/env bash

# execute command
$@

# check status
STATUS=$?
if [ $STATUS == 0 ]; then
  echo "Command '$@' completed successfully"
else
  echo "Command '$@' exited with error status $STATUS"
fi
version: 2
jobs:
  build:
    working_directory: ~/foo
    docker:
      - image: ruby:2.3.3
    steps:
      - checkout
      - run: bundle check --path=vendor/bundle || bundle install --path=vendor/bundle --jobs=4 --retry=3  # must pass to proceed
      - run:
          command: bin/exec.sh bin/rubocop
      - run:
          command: bin/exec.sh bin/rubocop

Basic Database example (not ruby)
#7

@Chekote: This one is roughly equivalent in function to my $SOME_TEST_FAILED example unfortunately.

It fails at two places:

  1. The error condition is not propagated to CircleCI (and therefore the UI), so it cannot show a red box on the appropriate step when there’s a failure.
  2. When suppressed like this, the error condition does not cause the overall build to fail. I’m really interested in allowing N test blocks to run, 2 of them to fail, then report the entire build as failed. That was what my hack around $SOME_TEST_FAILED attempted to solve (imperfectly - we both fail at my point (1)

#8

I don’t think there’s much you can do about having the step flagged as failure without support from the CircleCI platform.

You should be able to resolve the non-failing build issue by having your script store a value that dictates whether or not any of the executions failed. I believe CircleCI 2.0 isolated each command, so you won’t be able to use environmental variables, but you could easily touch a file to indicate a failure of any kind.

Then you can just add a check as the final step to your build that checks for the existence of the file and fails the build.


#9

The example from above does actually work in practice (using environment variables):


#10

@CJBridges I think for now this is the best solution you can get. We might provide some built-in way of doing this in the future, but no ETA now.


#11

So the environmental variables do persist across run steps? Good to know! Thanks!


#12

Check the documentation for a new when flag that mostly resolves this problem.
https://circleci.com/docs/2.0/configuration-reference/#run

Set a build step to always to enable the “run even if the build has already failed” behavior.

Note that we still don’t have a mechanism to fail a build if an important prerequisite fails (e.g. bundle install), but generally that results in fast catastrophic failures in later steps anyway, so it’s probably not nearly as important.

@smaantci - can you please mark this as the “Answer” for this thread?


Parallel tests (cuc+rspec) w/failures exit early leaving less workers to finish
#13

when=always is great; but this doesn’t work for the persist_to_workspace step; nor is there a way to specify sequential jobs to always execute in workflows. Any thoughts on those usecases?
Edit: I guess the solution is to add a when=onfail line as the last step of the job. I’ll give it a try.
Edit: Nope did not work.


#14