Cucumber Tests No Longer Run In Order Since Circle 2.0

Build Environment Running Tests
, , ,
1

Since upgrading to 2.0 our cucumber tests no longer run in any discernible order as they once did on 1.0.

I understand that 2.0 tries to balance out the tests evenly by the looking at previous runs. But as the occasional test will fail and re-run, this doesn’t work very well for the purposes of testing. Worst of all, the same test will often run multiple times across multiple containers and other tests will not run at all.

In Circle 1.0 they ran in the order they were in our folder. In our case, in alphabetical order. So we always knew that test “A” would always be the first test in container 0 for example.

Has anyone any idea how we can get back to something like this. Where the same tests will always run in same container every time we do a run. Other than creating a folder for each set of tests and setting up all new test suites. Also possibly how to disable the test balancer.

2

Hi Will. Our tests don’t auto balance by default, can you share your config.yml, please?

3

config.yml is as follows:

version: 2.0

jobs:
  "ruby-2.2":
    docker:
      - image: circleci/ruby:2.2-node
      - image: circleci/postgres:9.4.12-alpine
    working_directory: ~/circleci-demo-workflows
    steps:
      - checkout
      - run: bundle install --path vendor/bundle
      - run: bundle exec rake db:create db:schema:load
      - run:
          name: Run tests
          command: rake

  "ruby-2.3":
    docker:
      - image: circleci/ruby:2.3-node
      - image: circleci/postgres:9.4.12-alpine
    working_directory: ~/circleci-demo-workflows
    steps:
      - checkout
      - run: bundle install --path vendor/bundle
      - run: bundle exec rake db:create db:schema:load
      - run:
          name: Run tests
          command: rake

  "ruby-2.4":
    docker:
      - image: circleci/ruby:2.4-node
      - image: circleci/postgres:9.4.12-alpine
    working_directory: ~/circleci-demo-workflows
    steps:
      - checkout
      - run: bundle install --path vendor/bundle
      - run: bundle exec rake db:create db:schema:load
      - run:
          name: Run tests
          command: rake

workflows:
  version: 2
  build:
    jobs:
      - "ruby-2.2"
      - "ruby-2.3"
      - "ruby-2.4"
4

This is likely going to be something in your Rakefile then. Can you share that?

5

Can I message you privately. I don’t think it a good idea to make show my rake file publicly. Just incase.

6

That’s fair. Can I ask you to open a ticket instead though? That way I can get more eyes on it if needed, and it’s easier to track.

7

Send it to you there. Will open a ticket though as well.

1 Like