Elasticsearch integration with CI 2.0 [Rails project]

[AhsenArif]

Hi.
I’m working on migrating to Circle CI 2.0. I have a Ruby on Rails project setup with elastic search that has been working fine with 1.0 configuration.

On trying to configure CI to 2.0, I’m having a problem with setting elasticsearch port for Test Environment in Circle CI.
My config.yml file:

version: 2

defaults: &defaults
  working_directory: ~/repo
  docker:
    - image: circleci/ruby:2.4.1-node
      environment:
        - BUNDLE_JOBS: 4
        - BUNDLE_RETRY: 3
        - BUNDLE_PATH: vendor/bundle
        - PGHOST: 127.0.0.1
        - PGUSER: postgres
        - RAILS_ENV: test
    - image: mdillon/postgis:9.5
      environment:
        - POSTGRES_USER: postgres
        - POSTGRES_DB: a_test
        - POSTGRES_PASSWORD: 
        - RACK_ENV: test
    - image: circleci/redis:4.0.9
      environment:
        - REDIS_URL: "redis://localhost:6379/"
    - image: docker.elastic.co/elasticsearch/elasticsearch:6.2.2
      environment:
        - cluster.name: elasticsearch
        - xpack.security.enabled: false
        - transport.host: localhost
        - network.host: 127.0.0.1
        - http.port: 9250
        - discovery.type: single-node

jobs:
  build:
    <<: *defaults
    steps:
      - checkout
      # Download and cache dependencies
      - restore_cache:
          keys:
            - bundle-{{ arch }}-{{ checksum "Gemfile.lock" }}
      - run: bundle install
      - save_cache:
          paths:
            - ./vendor/bundle
          key: bundle-{{ arch }}-{{ checksum "Gemfile.lock" }}
      - persist_to_workspace:
          root: .
          paths:
            - vendor/bundle
      - run:
          name: Wait for DB
          command: dockerize -wait tcp://localhost:5432 -timeout 1m
      - run:
          name: Wait for Redis
          command: dockerize -wait tcp://localhost:6379 -timeout 1m
      - run:
          name: Database setup
          command: |
            bundle exec rake db:schema:load --trace
            bundle exec rake db:migrate
      - run:
          name: Run tests
          command: |
            mkdir -p ~/repo/tmp/test-results/rspec
            bundle exec rspec --color --require spec_helper spec --format progress --out ~/repo/tmp/test-results/rspec/results.xml --format progress
      - store_artifacts:
          path: ~/repo/tmp/screenshots
          destination: test-screenshots
      - store_test_results:
          path: ~/repo/tmp/test-results

When I reach the Run tests command Circle CI throws following Error:

Since I’m running the CI in test Env, my specs need to run on port 9250. I’ve configured them in spec_helper.rb like this:

config.before :each, elasticsearch: true do
    Elasticsearch::Extensions::Test::Cluster.start(port: 9250) unless Elasticsearch::Extensions::Test::Cluster.running?
  end

  config.after :each, elasticsearch: true do
    Elasticsearch::Model.client.indices.delete index: '_all'
  end

  config.after :suite do
    Elasticsearch::Extensions::Test::Cluster.stop(port: 9250) if Elasticsearch::Extensions::Test::Cluster.running?
  end
end

And elastic search is initialized by my project for Test Env as:

if Rails.env.test?
  Elasticsearch::Model.client = Elasticsearch::Client.new host: 'localhost:9250',
                                                          logger: Logger.new(STDOUT),
                                                          log: true
end

Anyone else who has had this problem. ?

[leboshi]

Not sure if you were ever able to resolve this, but since I just had to deal with this last night…

tl;dr: create a set of arguments that includes network_host, and pass it to all calls to Elasticsearch::Extensions::Test::Cluster:

  test_cluster_config = { port: 9250, number_of_nodes: 1, timeout: 120, network_host: '_local_' }

  config.before :each, elasticsearch: true do
    Elasticsearch::Extensions::Test::Cluster.start(test_cluster_config) unless Elasticsearch::Extensions::Test::Cluster.running?(test_cluster_config)
  end

etc.

Here’s what’s going on: the Docker image starts Elasticsearch, but since your Rails app isn’t actually running within the context of that image, it doesn’t have access to the Elasticsearch binaries, including the CLI. That means that Elasticsearch::Extensions::Test::Cluster.start and stop can’t actually manipulate any clusters. That code is still necessary to make your tests pass locally. However, the loading of the Elasticsearch Docker image, running on port 9250, is the workaround in CI so that a new test cluster doesn’t need to be spun up.

But if you look at the code for start, stop, running?, etc., each call delegates to a new instance of Elasticsearch::Extensions::Test::Cluster::Cluster, which grabs default parameters for anything not supplied. If you don’t supply the network_host parameter, it has to look up a default. The default, however, varies by version, so it tries to call into the Elasticsearch CLI to find it. That’s what’s missing. So by providing a full set of parameters to all calls, your spec helper doesn’t ever need to call out to the Elasticsearch CLI when it’s on the CI environment, and that error magically disappears.

[stephenhmarsh]

I’m running version 2.1. I have this problem as well, same code throwing same error, but @leboshi 's answer has not solved it.

Still get:

Errno::ENOENT:
  No such file or directory - Cannot find Elasticsearch launch script from [elasticsearch] -- did you pass a correct path?
./spec/rails_helper.rb:122:in `block (2 levels) in <top (required)>'

[stephenhmarsh]

Actually, I was finally able to get it to work by NAMING THE CLUSTER.

- image: docker.elastic.co/elasticsearch/elasticsearch:6.5.4
        environment:
          - cluster.name: group-project-test-cluster
          - xpack.security.enabled: false
          - transport.host: localhost
          - network.host: 127.0.0.1
          - http.port: 9250
          - discovery.type: single-node

Then, using @leboshi 's suggestion to save the config in a variable and use it EVERYWHERE:

es_test_cluster_config = { port: 9250, number_of_nodes: 1, network_host: '_local_', cluster_name: 'group-project-test-cluster' }

[yagudaev]

@stephenhmarsh super! Thank you! Setting it to port 9200 which is the default meant I didn’t need to do any environment variable setup or anything