I changed my test step to use circleci tests run to take advantage of the new --fail-fast feature.
Previously I was using the ruby/rspecs-test orb.
Now I change to use this step’s command:
RSPEC_FILES=$(circleci tests glob "spec/**/*_spec.rb" | circleci tests split --split-by=timings) echo $RSPEC_FILES | circleci tests run --command="xargs bundle exec rspec --profile 10 --format RspecJunitFormatter --out /tmp/test-results/rspec/results.xml --format progress" --batch-count=3 --fail-fast --test-results-path="../tmp/test_results"
I use parallelism: 3 so the tests are splitted in 3 lists.
After outputting RSPEC_FILES to the console, I the RSPEC_FILES result for all 3 tasks and confirmed that all test files are properly used. The combined result is the same as if I change parallelism to 1
The issue is then the circle tests run is executed, everything works but for each 3 task, only a subset of the expected test is executed. I see about ~150 examples executed per task instead of ~500.
When I change parallelism: 1, the very same command is executing all my examples (~1500)
I don’t understand why this happens, does anybody have an idea ?
I can also confirm the issue by looking at the time to execute the tests, is usually takes ~10min per task and now it takes ~3min