Managing Elasticsearch Cluster


#1

I’ve got it to the point where the test are running, now. I’m stuck on a few failing tests where we’re using 1 to spin up a 2nd elasticsearch cluster for some migrating-data tests, but there’s not local elasticsearch command, since we’re using the elasticsearch docker image.

One more question, related to the Elasticsearch Test Cluster linked previously. It allows me to specify the command it uses (by default elasticsearch. I modified it to try and use docker run elasticsearch ..., but the ruby container doesn’t have a docker executable, even if I apt-get install docker. I grabbed the local circle-builder script, and can run the picard container, but how do I get into the ruby container inside that (Inception noise here) to investigate?


#2

I ran into this problem with a customer running custom commands for a Postgres instance. I tried two approaches: one using apt-get to install postgres-cli. That worked fine. I also tried using our monolithic image. That was painful since it can also start up other services and databases inside its own container. Once it was up though, it was easier to rely on that for everything the customer needed.

To start with, I would search for a way to install the elasticsearch CLI.

I’m pretty sure you can’t start a Docker image inside the Docker executor. The Docker command may already be available, but you’ll hit errors with certain docker CLI commands.


#3

My eventual plan was to build a custom image with the elasticsearch tool baked-in, but I was going to wait for private image support before I started down that path.

That actually seems to be a blocker for building several of our projects at this point, as well.


#4

It would be handy to be able to choose a specific container with which to run each step command on, instead of it always running the commands on the first container. This will allow you to use the elasticsearch-cli docker image for the query test commands, instead of having to build a monolithic container image just for CI.


#5

@paul @petemill We plan on rolling out Docker Engine support sometime in the near future. This will allow you to use Docker Compose among other Docker features that you have become accustomed to. I believe this may help address some of the concerns expressed here. Let me know what you all think!


#6

I just wanted to add that we already have this feature available for a limited number of customers. Give us the word and we can add you to the list and enable it for you. :slight_smile:


#7

Hi, @MikeyYeahYeah.

I’m using the official Elasticsearch image from Docker Hub. I believe that I’m running into the same problem that you noted in your original post in this thread - I’m not sure how to start Elasticsearch or how to connect to Elasticsearch from my first (Python) image.

Do I need Docker Engine support? If so, can you grant it to me? Otherwise, do you have any tips?

This is one of my builds:
https://circleci.com/gh/brainix/spool/512

I appreciate any help, and thank you!
Raj


#8

@brainix

I don’t have experience with Elasticsearch, but does the cluster health API endpoint work for all Elasticsearch use cases?

If so, then maybe you can block your main container on a while loop that queries that endpoint until Elasticsearch is ready. Your check condition will depend on this API–I’m not sure if it will return a 200 when a cluster is still replicating.

More info in this Stackoverflow answer.


#9

Hi, Eric.

Thanks for your help! I was waiting for my Elasticsearch node to come up in a loop. In my circle.yml file, I have the following build step:

wget --tries 180 --timeout=15 --read-timeout=20 --waitretry=1 --retry-connrefused http://127.0.0.1:9200/ &> /dev/null

I think that my issue is that the Docker container itself doesn’t start Elasticsearch? And that the docker command doesn’t seem to be available, so I can’t start Elasticsearch in a build step? I’m a little bit lost here myself, so I’m making note of my issue here in case anyone else runs into it or knows how to solve it.

For reference, here’s a link to one of my builds:
https://circleci.com/gh/brainix/spool/514

Thanks again!
Raj


#10

It seems like port 9200 is not being exposed properly. I am looking into this issue.


#11

@brainix I was able to get it working with the following image: levlaz/elasticsearch-docker-ci:5.1.1

https://circleci.com/gh/ryanwohara/test-elasticsearch/27#config/containers/0


#12

CircleCI is incredible. Thank you for your help!


#13

For those who run into this issue and want to use elasticsearch, the full description is on the project homepage on DockerHub. It’s important to echo the warning from there: this is not intended for production use.


#14