Fail fast: stop running tests on first failure

circle.yml

#1

CircleCI doc states that:

all test commands are run to completion, but a non-zero exit code during setup will cause the build to fail early.

Adding a new option to “fail early” during test just like during setup would be great, so that the build execution would be faster and test commands following a failed one would not be executed.

My use case for the second point is this:

  • the latest test command of my circleci triggers a new parametrized build using the API. This new build runs additional and longer test cases.
  • I don’t want to trigger this additional build if any of the previous faster tests have failed: I’d prefer the build to be stopped and be marked as failed.

Fail fast: make it a (default) option
#2

This feature can provide significant CircleCI builder ec2 cost savings in the enterprise edition.
The use case is as follows:

  1. In feature branches, enable fail fast - stop running tests on first failure
  2. In master/integration branches - disable fail fast (the default behavior). This would allow to gather test pass/fail stats and other cool stuff you would expect from a stable branch.

#3

I’m wondering why CircleCI haven’t implemented this yet.

‘post’ is a great place to do cleanup, and right now I’m waiting 10+ minutes for my test reports even though I know the build failed in my first test suite.

Unfortunately this means that CircleCI isn’t an appropriate tool for a fast feedback loop, because I can’t just make the build fail-fast if my unit tests are broken.


#4

Agreed, this is a major annoyance! We often have multiple test steps and we organize them from “fastest to slowest” for the tightest feedback loop. This limitation in CircleCI means our test steps always look like

    - linter && unit_tests && integration_tests && etc.. etc..

so we can approximate “fail fast” functionality, but it’s pretty annoying if you have some integration setup stuff you only want to do if, say, the linter and the unit tests pass.


#5

+1 fail-fast would be much better for us.


#6

Absolutely agree. My first step is an static code analysis that takes 1 minute, if it fails I don’t need to spend another 15 minutes running tests.


#7

We’d appreciate if we can bail out under some conditions i.e. unit tests are failing, no point to test further also we’d love to fail not just the container but whole build as in all containers should stop immediately.


#8

As an alternative to the feature request, put any “fast-fail” tests in dependencies: post. Anything fails there the entire build stops.

Test that are okay to run no matter what so that you may get a full scope of what works and what doesn’t can continue to be placed in the test phase.


Parallel tests (cuc+rspec) w/failures exit early leaving less workers to finish
NPM exit code 1
#9

I think this is already implementable in user space anyway.


#10