Elasticsearch integration with CI 2.0

Build Environment
1

Hi
When I tried to solve the same problem like Elasticsearch integration with CI 2.0 [Rails project]
Even I set a value for cluster_name, still got Cannot find Elasticsearch launch script from [elasticsearch] error.

So I tried build a test environment with script below

version: 2
defaults: &defaults
  working_directory: ~/xxx
  docker:
    - image: circleci/ruby:2.6.1-node-browsers
    - image: postgres:9.5
    - image: redis
      environment:
        xxxx
    - image: docker.elastic.co/elasticsearch/elasticsearch:6.8.0
      environment:
        - cluster.name: es-test-cluster
        - xpack.security.enabled: false
        - transport.host: localhost
        - network.host: 127.0.0.1
        - http.port: 9250
        - discovery.type: single-node

jobs:
  build:
    <<: *defaults
    steps:
      - run:
          xxxx

and used ssh connection to debugging.
It looks like there no elasticsearch installation!

> circleci@6d07ad02c109:~$ which elasticsearch
> circleci@6d07ad02c109:~$ 
> circleci@xxxxxx:~$ find / -type d -name 'elasticsearch'
> circleci@xxxxxx:~$

Anyone else who has had this problem?

2

Hello @gccj,

Welcome to the forum.

Looking at your config I see you have set up a secondary service container for elasticsearch. Each - image: is a separate docker image that is accessible via the network. This would mean you can connect to this elasticsearch container from a client installed in your main image.

The command you are executing: which elasticsearch is executing on the circleci/ruby:2.6.1-node-browsers docker image which does not have the elasticsearch binary installed.

3

Thanks @KyleTryon
I’d like to install elasticsearch manually on circleci/ruby:2.6.1-node-browsers docker image.

4

Hello,

You have two options to install any software.

  1. You may install manually via the run command.
    https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html

  2. You can create a custom docker image using circleci/ruby:2.6.1-node-browsers as your base image. From there you can publish your own Docker image on the Dockerhub and use this in your projects.

FROM circleci/ruby:2.6.1-node-browsers
RUN ...

Creating a custom Docker image is usually the best option as there are less potential points of failures. A built image using circleci/ruby:2.6.1-node-browsers will benefit from any cached layers from this image on our host and will only need to download any additional layers created from the DockerHub.

One note, I would recommend dropping the patch version in the image. Often patches are required to fix issues that may arrise with the image (such as moving/changing packages). Building off 2.6 will allow you to pull the most recent patch which may help avoid issues.

5

@KyleTryon What happens if someone does not want to install Elasticsearch manually. ?
The problem with provided Elasticsearch images are that the path to location of elasticsearch is not available.

E.g. using the same config as given in the question

version: 2
defaults: &defaults
  working_directory: ~/xxx
  docker:
    - image: circleci/ruby:2.6.1-node-browsers
    - image: postgres:9.5
    - image: redis
      environment:
        xxxx
    - image: docker.elastic.co/elasticsearch/elasticsearch:6.8.0
      environment:
        - cluster.name: es-test-cluster
        - xpack.security.enabled: false
        - transport.host: localhost
        - network.host: 127.0.0.1
        - http.port: 9250
        - discovery.type: single-node

This Elasticsearch cluster requires giving the args :command by setting the ENV variable TEST_CLUSTER_COMMAND with the path to the Elasticsearch file. (on local system the path is something like '/Users/.../.../elasticsearch-6.8.0/bin/elasticsearch'
Otherwise it throws the error:

Errno::ENOENT:
  No such file or directory - Cannot find Elasticsearch launch script from [elasticsearch] -- did you pass a correct path?

If we use docker image of Elasticsearch, then can you please tell what can be the expected path of where the ES is installed on the directory structure. ?

6

Hi @AhsenArif
You can access Elasticsearch service via network.
The endpoint of Elasticsearch is localhost:{port you set}
That means you haven’t necessary to boot any Elasticsearch service on primary container.

1 Like
7

Hi @KyleTryon
Thank you for your kindness!
I was able to do my stuff both the way to use docker image and the way to install es manually.

8

Glad to help @gccj!

@AhsenArif, @gccj is correct above. Please let us know if we can assist you further.

Happy building!