Tests randomly fail on CircleCI 2.0

Build Environment
1

Some of my tests fail on CircleCi 2.0. They do not fail on 1.0 so I’m not sure what the issue is.

My test setup is.

build:
    <<: *defaults
    <<: *setup_env
    docker:
      - image: circleci/build-image:ubuntu-14.04-XXL-upstart-1189-5614f37
        command: /sbin/init
      - image: levlaz/elasticsearch-docker-ci:5.1.1
    resource_class: xlarge
    working_directory: ~/example
    parallelism: 2
    steps:
    - checkout
    - *restore_dir
    - *restore_js
    - run: mkdir -p $CIRCLE_ARTIFACTS $CIRCLE_TEST_REPORTS
    - run:
        working_directory: $HOME/example
        command: 'echo ''Etc/UTC'' | sudo tee -a /etc/timezone; sudo dpkg-reconfigure
          -f noninteractive tzdata; sudo service mysql restart; sudo service postgresql
          restart; '
    - run:
        working_directory: $HOME/example
        command: 'sudo redis-cli ping >/dev/null 2>&1 || sudo service redis-server start; '

    - run: echo -e "export RAILS_ENV=test\nexport RACK_ENV=test" >> $HOME/.circlerc
    - run: sed -i.bak "/gem ['\"]growl_notify\|autotest-fsevent\|rb-appscript\|rb-fsevent['\"].*, *$/ N; s/\n *//g; /gem ['\"]growl_notify\|autotest-fsevent\|rb-appscript\|rb-fsevent['\"]/ d" Gemfile
    - run: |-
        mkdir -p config && echo 'test:
          adapter: mysql2
          database: circle_ruby_test
          username: ubuntu
          host: localhost
        ' > config/database.yml
    - run: |-
        export RAILS_ENV="test"
        export RACK_ENV="test"
        bundle exec rake db:create db:schema:load --trace
    - run: case $CIRCLE_NODE_INDEX in 0)  bundle exec rspec spec ;; esac
2

Readers may need more info to be able to help. What build output do you get? Is this browser tests, other integration tests, unit tests?

The usual reason for (integration/browser) test failure is that a required service was not ready at the time it was requested (e.g. Selenium Server, PhantomJS, a RDBMS, etc). In your case I would suspect Elasticsearch. The solution often is to loop-wait until all required services are ready.

3

They are controller tests. For example, I have this method in my controller

  def show
    user = User.where.not(uuid: [nil, ""])
      .find_by(uuid: params[:uuid])
    user.send_reset_password_instruction
  end

And the tests fail with this

Failure/Error:
        expect do
          get :show, uuid: user.uuid
        end.to change { ActionMailer::Base.deliveries.count }.by(1)
      
        expected result to have changed by 1, but was changed by 0

Note that they pass on Circle CI 1.0 so it’s not like I changed anything

4

OK. I expect the best approach here is to trace the test in the usual way, as if it were failing locally, to see why it is failing. That should indicate the problem, and since they worked before, I imagine fixing one will fix all of them.

They are controller tests

My earlier question was trying to determine how atomic they are. Controller tests sound like unit tests to me (least likely to break) but I am conscious you have a db:create db:schema:load call before your tests - if the database needs to be up then they’re integration tests, and I’d suspect a database problem.

Don’t forget you can SSH into the box post-fail to debug your tests - it makes the edit/test loop much shorter than guessing, committing, pushing, and waiting.

5

I’ve been able to trace the issue down to tests with skip_before_action filters. The filters are not being being skipped in the tests unless manually specified. I believe this to be a bug because it’s not the same way in circlecl 1.0

6

I don’t know, but I think that’s unlikely because these are just VMs. There might be some environmental differences, but other than that, there’s not much that can affect Ruby code. Your tests look like they are database related, so SSH into the box and get a console client to see how the database is being mutated before/after the failing test.

7

They are not database related. I ssh’ed and the method in the controller was not being called at all. I manually set the controller to skip the before action that was in the controller in the tests and the test passed

it do
  controller.class.skip_before_filter :authenticate_user!
end
8

OK. I’m not at all versed in Ruby, so unfortunately I don’t know what the significance of your discovery is :smiley_cat:

9

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.