Parallel tests (cuc+rspec) w/failures exit early leaving less workers to finish

Build Environment Running Tests
, , ,
1

We have 7x parallelism, with cucumber running first then rspec (using knapsack pro). The problem is that when a cucumber test fails on a worker, that worker fully exits and leaves fewer than 7 to run rspec (not all workers have cucumber failures).

Then, instead of taking a few minutes, rspec runs on 2 of 7 and takes 20+ minutes to finish all of rspec.

Is there a way to allow these workers to continue but ultimately fail if any of the test runs return non-zero exit codes?

Sample config

# ----
# cucumber (first, because rspec operates in queue mode for better remainder balancing)
- restore_cache:
    key: v2-cucumber-vcr-{{ .Branch }}
- run: |
      mkdir -p test-reports/cucumber
      bundle exec rake "knapsack_pro:cucumber[--format junit --out test-reports/cucumber/junit.xml]"
- store_test_results:
    path: "test-reports/cucumber"
- store_artifacts:
    path: target/scenarios
    destination: scenarios

# ----
# rpec
- restore_cache:
    key: v2-rspec-vcr-{{ .Branch }}
- run: |
      mkdir -p test-reports/rspec
      bundle exec rake "knapsack_pro:queue:rspec[--format progress --format RspecJunitFormatter -o test-reports/rspec/junit.xml]"
- store_test_results:
    path: "test-reports/rspec"
2

I found a few related topics: Fail fast: stop running tests on first failure

and tip in this one: Run all test steps even if one fails to use when=always and you could add this to the step for RSpec with knapsack_pro Queue Mode.

Here is more details about when attribute: https://circleci.com/docs/2.0/configuration-reference/#the-when-attribute

1 Like