Circle ci tests are failing randomly

Community
, , ,
1

My tests are always green in local machine. I do have codebase built on Ruby on Rails framework. But in CI my tests fails randomly and after doing retry with cache with 3-4 times it gets passed. Most of the time I see error as:

Failures:

  1) the survey types table when click on Data link when click on survey types link creates a new survey type
     Failure/Error: value.chars.each { |c| find(selector).native.send_keys c }
     
     Selenium::WebDriver::Error::StaleElementReferenceError:
       stale element reference: element is not attached to the page document
         (Session info: chrome=64.0.3282.140)
         (Driver info: chromedriver=2.35.528139 (47ead77cb35ad2a9a83248b292151462a66cd881),platform=Linux 3.13.0-141-generic x86_64)
     # /home/ubuntu/.bundle_end_to_end/ruby/2.3.0/gems/selenium-webdriver-3.8.0/lib/selenium/webdriver/remote/response.rb:69:in `assert_ok'
     # /home/ubuntu/.bundle_end_to_end/ruby/2.3.0/gems/selenium-webdriver-3.8.0/lib/selenium/webdriver/remote/response.rb:32:in `initialize'
     # /home/ubuntu/.bundle_end_to_end/ruby/2.3.0/gems/selenium-webdriver-3.8.0/lib/selenium/webdriver/remote/http/common.rb:81:in `new'
     # /home/ubuntu/.bundle_end_to_end/ruby/2.3.0/gems/selenium-webdriver-3.8.0/lib/selenium/webdriver/remote/http/common.rb:81:in `create_response'
     # /home/ubuntu/.bundle_end_to_end/ruby/2.3.0/gems/selenium-webdriver-3.8.0/lib/selenium/webdriver/remote/http/default.rb:104:in `request'
     # /home/ubuntu/.bundle_end_to_end/ruby/2.3.0/gems/selenium-webdriver-3.8.0/lib/selenium/webdriver/remote/http/common.rb:59:in `call'
     # /home/ubuntu/.bundle_end_to_end/ruby/2.3.0/gems/selenium-webdriver-3.8.0/lib/selenium/webdriver/remote/bridge.rb:164:in `execute'
     # /home/ubuntu/.bundle_end_to_end/ruby/2.3.0/gems/selenium-webdriver-3.8.0/lib/selenium/webdriver/remote/oss/bridge.rb:579:in `execute'
     # /home/ubuntu/.bundle_end_to_end/ruby/2.3.0/gems/selenium-webdriver-3.8.0/lib/selenium/webdriver/remote/oss/bridge.rb:370:in `send_keys_to_element'
     # /home/ubuntu/.bundle_end_to_end/ruby/2.3.0/gems/selenium-webdriver-3.8.0/lib/selenium/webdriver/common/element.rb:154:in `send_keys'
     # ./spec/support/form_helpers.rb:11:in `block in fill_form_input'
     # ./spec/support/form_helpers.rb:11:in `each'
     # ./spec/support/form_helpers.rb:11:in `fill_form_input'
     # ./spec/support/sessions_management.rb:6:in `block in sign_in'
     # /home/ubuntu/.bundle_end_to_end/ruby/2.3.0/gems/capybara-2.17.0/lib/capybara/session.rb:341:in `within'
     # /home/ubuntu/.bundle_end_to_end/ruby/2.3.0/gems/capybara-2.17.0/lib/capybara/dsl.rb:50:in `block (2 levels) in <module:DSL>'
     # /home/ubuntu/.bundle_end_to_end/ruby/2.3.0/gems/capybara-2.17.0/lib/capybara/rspec/matcher_proxies.rb:14:in `within'
     # ./spec/support/sessions_management.rb:5:in `sign_in'
     # ./spec/side_menus/data/survey_types_spec.rb:45:in `block (4 levels) in <top (required)>'

My circleci.yml looks like:

machine:
  environment:
    DBUS_SESSION_BUS_ADDRESS: /dev/null
    RAILS_SECRET_KEY: '80c529174258d662b1d15e80e74ed934768f806c9307c2a15e38ab45f1101e0c30a6bcf9d3e809f7635ad4e56160234be3eec667712f8733ec0bf50f4e180585'
  timezone:
    Europe/Berlin
  ruby:
    version: 2.3.3
  node:
    version: v7.4.0

dependencies:
  # we automatically cache and restore many dependencies between
  # builds. If you need to, you can add custom paths to cache:
  cache_directories:
    - "custom_1"   # relative to the build directory
    - "~/custom_2"

dependencies:
  pre:
    - curl -L -o google-chrome.deb https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
    - sudo dpkg -i google-chrome.deb
    - sudo sed -i 's|HERE/chrome\"|HERE/chrome\" --disable-setuid-sandbox|g' /opt/google/chrome/google-chrome
    - rm google-chrome.deb
    - | # epic build time improvement - see https://github.com/elm-lang/elm-compiler/issues/1473#issuecomment-245704142
      if [ ! -d sysconfcpus/bin ];
      then
        git clone https://github.com/obmarg/libsysconfcpus.git;
        cd libsysconfcpus;
        ./configure;
        make && sudo make install;
        cd ..;
      fi
    - bundle check --path=/home/ubuntu/.bundle || bundle install --path=/home/ubuntu/.bundle --jobs=4 --retry=3:
        pwd: backend/ruby
    - npm install -g bower:
        pwd: frontend
    - ruby ./with_retry.rb npm install -g elm@0.18:
        pwd: frontend
    - bower install:
        pwd: frontend
    - npm install:
        pwd: frontend
    - ruby ./with_retry.rb elm package install --yes:
        pwd: frontend
    - sysconfcpus -n 1 elm-make --yes: # Compile packages
        pwd: frontend
    - sysconfcpus -n 1 elm-make --yes --output /dev/null src/elm/Main.elm: # Compile our sources
        pwd: frontend
    - sysconfcpus -n 1 npm run build:
        pwd: frontend
    - mkdir deployment:
        pwd: frontend
    - cp -R dist/* deployment:
        pwd: frontend
    - mv config/database.yml.example config/database.yml:
        pwd: backend/ruby
    - mv config/secrets.yml.example config/secrets.yml:
        pwd: backend/ruby
    - bundle exec rake db:reset:
        pwd: backend/ruby
    - bundle check --path=/home/ubuntu/.bundle_end_to_end || bundle install --path=/home/ubuntu/.bundle_end_to_end --jobs=4 --retry=3:
        pwd: frontend_end_to_end

test:
  override:
    - RAILS_ENV=test bin/rake spec:
        pwd: backend/ruby
    - sysconfcpus -n 1 npm start:
        pwd: frontend
        background: true
    - RAILS_ENV=development bin/rails s -p 3000 -b 0.0.0.0:
        pwd: backend/ruby
        background: true
    - wget --waitretry=5 --retry-connrefused -v http://127.0.0.1:8080/
    - bundle exec rspec ./spec:
        pwd: frontend_end_to_end

deployment:
  production:
    branch: master
    commands:
      - bundle exec mina deploy TO=production:
          pwd: backend/ruby
      - bash ./deploy-frontend.sh:
          pwd: frontend
      - bundle exec mina restart_puma_inspection TO=production:
          pwd: backend/ruby
  staging:
    branch: staging
    commands:
      - rm -rf dist/*:
          pwd: frontend
      - sysconfcpus -n 1 npm run stage:
          pwd: frontend
      - bundle exec mina deploy TO=staging:
          pwd: backend/ruby
      - bash ./deploy-frontend-staging.sh:
          pwd: frontend
      - bundle exec mina passenger_restart TO=staging:
          pwd: backend/ruby

Any idea how can I fix this permanently?

2

Any ideas how this could be further investigated?

3

Where there is a difference between local and remote behaviour in web drivers like Selenium, often the issue is that you’re not waiting long enough for an element to be ready.

I have a bunch of browser tests, and I’ve found that rather than a fixed sleep (e.g. two seconds) it is better to repeatedly test the browser for something (e.g. element visibility) until a timeout of five seconds (sleeping for, say, 0.2 sec between check). Sometimes it’ll wait for 1.8 sec (where it would have passed the old sleep) and sometimes it’ll take 2.2 sec (where it would have failed).

This is down to your laptop having better processing power available than the server - remember the server is shared by other users.

4

This topic was automatically closed 166 days after the last reply. New replies are no longer allowed.