ElasticSearch Container got Killed

Build Environment
1

I’m using official elastic search container, and I guess it’s getting killed.

I’ve got this message:

 Native controller process has stopped - no new native processes can be started

Exited with code 137
2

This behavior - Elasticsearch crashing with “native controller process has stopped” - is commonly seen when running ES inside Docker, but I’m not sure how to fix on CircleCI yet.

The most extensive thread documenting the issues and some workarounds is here: https://github.com/elastic/elasticsearch/issues/25067

Elasticsearch’s documentation for how to best host ES5 on Docker is here: https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html

Some of the needed workarounds (setting ulimits etc on the container) aren’t available (so far as I can see) from the CircleCI YAML configuration (some are available, because they are just about setting environment variables).

In our Elasticsearch production running under Docker, we’ve also had to set sysctl variable vm.max_map_count to a higher value on the Docker host (not container). In fact, Elasticsearch 5 won’t start without this sysctl variable set high, unless you’re set environment variable discovery.type=single-node (or otherwise arrange to bypass the boot-time check). But that config issue doesn’t come up in the 25067 thread above.

Note that these issues are new with Elasticsearch 5; earlier versions don’t have similar issues.

3

Hi @jaredrhine!

I appreciate very much your help!

I found one solution, for now. Limiting HEAP size of Elastic Search.

I found out that container got kille (exit code 137), because it was using much memory.

Follow our config.yml:

 - image: docker.elastic.co/elasticsearch/elasticsearch:5.6.2
        environment:
          - discovery.type=single-node
          - http.host=0.0.0.0
          - transport.host=127.0.0.1
          - xpack.security.enabled=false
          - ES_JAVA_OPTS=-Xms750m -Xmx750m

Thank you very much!

Source: https://github.com/10up/wp-local-docker/issues/6

1 Like
4

Thanks for following up with your fix.

Your change actually fixed my issue too, in a roundabout way. I had set memory limits before, but had used ES_JAVA_OPTIONS instead of ES_JAVA_OPTS!

This change highlights that the problem appears to be not that Elasticsearch doesn’t have enough memory, but instead has too much (2G) allocated by default compared to the container size.

1 Like
5

Glad to be helpful!

Yes, I’ve read that exit code 137 means that the container got killed, usually for high resources consumption.

I guess if we limit memory, we obly Elastic Search to do pagination or something like that…to not be so greedy.