Tests fail, are slow with headless Chrome on CircleCI

[00da1009]

Hi!

I am trying to migrate our test suite to Headless Chrome from PhantomJS. We are on CircleCI 2.0.

My issue is that the tests run extremely slowly (>30 mins) and about 70% of tests fail. When running the same test suite on my local computer, the browser tests pass in 10 mins. When running in CircleCI using Poltergeist, the tests pass in about 5 mins.

Can anyone help me? Here are my settings:

Capybara.register_driver :headless_chrome do |app|
  capabilities = Selenium::WebDriver::Remote::Capabilities.chrome(
    chromeOptions: { args: %w(headless disable-gpu window-size=1024,768) }
  )

  Capybara::Selenium::Driver.new(app, browser: :chrome, desired_capabilities: capabilities)
end

Capybara.server = :puma
Capybara.default_max_wait_time = 30
Capybara.javascript_driver = :headless_chrome

[halfer]

Some thoughts:

1. Is there any pattern in the test failure?
2. Do you get different failures for each run?
3. Are you able to get a screenshot for a failure?

Perhaps you would also paste your CircleCI config. I am wondering if there is a server component that could do with a few seconds of sleep, so it is always ready before you try to use it.

[00da1009]

I changed the config to only run one test and put in some code to save a screenshot at the end of that test, but the test runner doesn’t seem to even get that far most of the time. I repeated the build several times in a row and here are some of the failure reports I get:

     1.1) Failure/Error:
            page.set_rack_session(
               ...
            )
          
          Net::ReadTimeout:
            Net::ReadTimeout
          # ./spec/support/sign_in_helpers.rb:90:in `create_auth_logic_session'
          # ./spec/support/sign_in_helpers.rb:25:in `sign_in_as_an_admin'
          ...
          # -e:1:in `<main>'

     1.2) Failure/Error: page.save_screenshot(screenshot_path)
          
          Net::ReadTimeout:
            Net::ReadTimeout
          # ./spec/spec_helper.rb:158:in `save_timestamped_screenshot'
          # ./spec/spec_helper.rb:165:in `block (2 levels) in <top (required)>'
          # -e:1:in `<main>'

     1.1) Failure/Error:
            page.set_rack_session(
              'user_credentials' => user.persistence_token,
              'user_credentials_id' => user.id,
              'current_account_id' => Account.current_id
            )
          
          Selenium::WebDriver::Error::UnknownError:
            unknown error: Devtools port number file contents <34733> were in an unexpected format
              (Driver info: chromedriver=71.0.3578.33 (269aa0e3f0db08097f0fe231c7e6be200b6939f7),platform=Linux 4.4.0-137-generic x86_64)
          # ./spec/support/sign_in_helpers.rb:90:in `create_auth_logic_session'
          # ./spec/support/sign_in_helpers.rb:25:in `sign_in_as_an_admin'
          ...
          # -e:1:in `<main>'

I am getting a combination of these errors (since I have it set to 3 retries) on every build, but the same test passes consistently on my local computer.

Here is a Gist of my config. https://gist.github.com/tb0yd/6e3d2fe498f9eb7273a497249b97abc5

[halfer]

OK, cool - so you have some errors. Search for them and see if other people are getting any errors of these types; they are bound to have happened to others before. Stack Overflow should have loads of references. Here’s one, from a quick web search.

I don’t do Ruby, but if you are generally familiar with it, get an SSH session on your post-fail machine, and treat the problem as if you were debugging locally. You can edit your tests and re-run them from the console.

[00da1009]

Wheeeeeeeeeeee my tests run now!!

Thanks for the tips! I didn’t know you could SSH into failing builds, and that really helped.

Apparently I was on an old version of Chrome and/or Chromedriver. Updating to the latest stable chrome and Chromedriver 2.38 fixed my issue.

[halfer]

You’re most welcome. Yes, SSH is nearly essential for headless browser test errors.