Using Ruby orb to run RSpec Tests Isn't Running All Tests

I’m working with a Rails project that has a fairly large set of tests. These tests have been executed on Circle using the ruby orb and the rspec-test task.

When tests get executed using

          order: random

around 300 tests are executed.

When I run using rspec/bundle directly (not using the ruby orb) using

- run:
          command: "bundle exec rspec --order random"
          name: Run rspec tests

around 2700 tests executed. Any idea what’s going on? I’m using the ruby@2.0.0 orb but I’ve been able to replicate this with the ruby@2.1.0 orb.

to confirm, you’re expecting 2700 tests to be executed correct? Are you setting any parallelism level?

If you set the rerun-fail param to false, does it make a difference in the number of tests that are run??

This is correct. I’m expecting to see 2700 tests executed.

I’m not setting any parallelism in these runs, using rspec or otherwise.

Interesting, could you try setting the rerun-fail param explicitly to false like I mentioned above and letting me know how many tests get executed?

I used the 2.1.0 version of the Ruby orb and re-ran the tests with the rerun-fail parameter set to false and there were still 900 tests run instead of 2700.

Got it. Any chance you could send me a link to one of your jobs to so I can take a closer look?