Using Ruby orb to run RSpec Tests Isn't Running All Tests

Build Environment Running Tests
, ,
1

I’m working with a Rails project that has a fairly large set of tests. These tests have been executed on Circle using the ruby orb and the rspec-test task.

When tests get executed using

ruby/rspec-test:
          order: random

around 300 tests are executed.

When I run using rspec/bundle directly (not using the ruby orb) using

- run:
          command: "bundle exec rspec --order random"
          name: Run rspec tests

around 2700 tests executed. Any idea what’s going on? I’m using the ruby@2.0.0 orb but I’ve been able to replicate this with the ruby@2.1.0 orb.

2

to confirm, you’re expecting 2700 tests to be executed correct? Are you setting any parallelism level?

If you set the rerun-fail param to false, does it make a difference in the number of tests that are run??

3

This is correct. I’m expecting to see 2700 tests executed.

I’m not setting any parallelism in these runs, using rspec or otherwise.

4

Interesting, could you try setting the rerun-fail param explicitly to false like I mentioned above and letting me know how many tests get executed?

5

I used the 2.1.0 version of the Ruby orb and re-ran the tests with the rerun-fail parameter set to false and there were still 900 tests run instead of 2700.

6

Got it. Any chance you could send me a link to one of your jobs to sebastian@circleci.com so I can take a closer look?