ElasticSearch exists with code 137

Build Environment
,
1

I have the following CI configuration:

version: 2
jobs:
  py-3.8: &build-template
    docker:
      - image: qlands/odktools_circleci:20210804
      - image: docker.elastic.co/elasticsearch/elasticsearch:6.8.14
        environment:
          - bootstrap.memory_lock=true
          - discovery.type=single-node
          - http.host=0.0.0.0
          - transport.host=127.0.0.1
          - xpack.security.enabled=false
          - "ES_JAVA_OPTS=-Xms750m -Xmx750m"
        ports:
          - 9200:9200    
    working_directory: ~/work
    steps:
      - checkout
      - restore_cache:
          key: requirements-{{ .Branch }}-{{ .Environment.CIRCLE_JOB }}-{{ checksum "requirements.txt" }}
      - run:          
          name: Run servers
          command: |
            sudo /etc/init.d/redis-server start
            sudo service mongodb start
            sudo service mysql start
            sudo service mosquitto start
            tail -f /dev/null
          background: true
      - run:
          # Install dependencies with --user to enable easier caching
          name: Install dependencies
          command: |
            pip install -r requirements.txt --user
            pip install codecov --user
            /home/circleci/.local/bin/codecov
            mysql -h localhost -u root --ssl-mode=DISABLED --password=circleci --execute='CREATE SCHEMA IF NOT EXISTS formshare'
            python3 create_config.py --daemon --capture_output --mysql_host localhost --mysql_user_name root --mysql_user_password circleci --repository_path /opt/formshare_repository --odktools_path /opt/odktools --elastic_search_host localhost --elastic_search_port 9200 --formshare_host localhost --formshare_port 5900 --forwarded_allow_ip localhost --pid_file /opt/formshare_gunicorn/formshare.pid --error_log_file /opt/formshare_log/error_log /opt/formshare_config/development.ini
            ln -s /opt/formshare_config/development.ini ./development.ini
            python3 configure_celery.py ./development.ini
            python3 configure_flatten.py
            python3 setup.py install --user
            python3 setup.py compile_catalog
            /home/circleci/.local/bin/disable_ssl ./development.ini
            /home/circleci/.local/bin/configure_alembic ./development.ini .
            /home/circleci/.local/bin/configure_mysql ./development.ini .
            /home/circleci/.local/bin/configure_tests ./development.ini .
            /home/circleci/.local/bin/alembic upgrade head
            pip install -e git+https://github.com/qlands/formshare_test_plugin.git@main#egg=formshare_test_plugin --user
      - run:
          name: Test code is well formatted
          command: |
            /home/circleci/.local/bin/black --version
            /home/circleci/.local/bin/black formshare --check || exit;
      - run:
          name: Run Celery in the background
          command: |
            export COLUMNS=80
            /home/circleci/.local/bin/celery -A formshare.config.celery_app worker --loglevel=info -Q FormShare
          background: true
      - run:
          name: Run tests
          command: |
            export FORMSHARE_PYTEST_RUNNING=true
            /home/circleci/.local/bin/pytest -s --cov=formshare
      - run:
          name: upload coverage report
          command: |
            /home/circleci/.local/bin/coverage xml
            (/home/circleci/.local/bin/codecov --required -X search gcov pycov -f coverage.xml) || echo 'Codecov failed to upload'
      - run:
          name: Building artifacts
          command: |
            python3 setup.py sdist
            python3 setup.py bdist_wheel
      - store_artifacts:
          path: dist/
          destination: ./
      - save_cache:
          key: requirements-{{ .Branch }}-{{ .Environment.CIRCLE_JOB }}-{{ checksum "requirements.txt" }}
          paths:
            - /home/circleci/.local/lib
            - /home/circleci/.local/bin
workflows:
  version: 2
  build:
    jobs:
      - py-3.8

The complete tests used to take about 40 minutes. But since we increased the number of tests we are experiencing that ElasticSearch gets canceled with code 137.

Could this be a memory issue? Do I need a higher resource_class?

2

Hi @qlands! Usually 137 means the container was killed, probably for high resource consumption. That doesn’t necessarily mean you need a higher resource class if you can control how much memory ElasticSearch is using or you can increase the memory available for Docker.

I know this GitHub issue is a little old, but it seems like the conversation here might help you identify which process is using up memory.