Running circleci tests run does not run all my examples

Build Environment Running Tests
, ,
1

Hi,
I changed my test step to use circleci tests run to take advantage of the new --fail-fast feature.
Previously I was using the ruby/rspecs-test orb.

Now I change to use this step’s command:

RSPEC_FILES=$(circleci tests glob "spec/**/*_spec.rb" | circleci tests split --split-by=timings)

echo $RSPEC_FILES | circleci tests run --command="xargs bundle exec rspec --profile 10 --format RspecJunitFormatter --out /tmp/test-results/rspec/results.xml --format progress" --batch-count=3 --fail-fast --test-results-path="../tmp/test_results"

I use parallelism: 3 so the tests are splitted in 3 lists.

After outputting RSPEC_FILES to the console, I the RSPEC_FILES result for all 3 tasks and confirmed that all test files are properly used. The combined result is the same as if I change parallelism to 1

The issue is then the circle tests run is executed, everything works but for each 3 task, only a subset of the expected test is executed. I see about ~150 examples executed per task instead of ~500.

When I change parallelism: 1, the very same command is executing all my examples (~1500)
I don’t understand why this happens, does anybody have an idea ?

I can also confirm the issue by looking at the time to execute the tests, is usually takes ~10min per task and now it takes ~3min

Thank you

1 Like
2

I have the same issue. I tried splitting tests by names instead of timings but the result is the same.

3

I have reached the support and this is currently a limitation :
“You will only get test results within the CircleCI UI for the last batch that executed. This is in the process of being resolved. This also means that test splitting by timing may not be perfect until this is resolved.”

4

Thank you for the answer. So it’s useless right now since it looks like the whole workflow status is based on this batch status, i.e. I know that one of the tests is failing but as it’s not a part of this last batch, it fails silently (the workflow is green). I hope this will be fixed soon.