Max memory of 0

#1

I’m using machine: true to execute tests in a docker-compose setting. The job in .circleci/config.yml looks like this:

run_tests:
  machine: true
  working_directory: ~/app
  steps:
    - checkout
    - run: docker-compose up -d this_server
    - run: docker-compose exec this_server python -m pytest

Building and running the containers happens without issue. When it comes time to run pytest, I get this:

Hint: Exit code 137 typically means the process is killed because it was running out of memory
Hint: Check if you can optimize the memory usage in your app
Hint: Max memory usage of this container is 0
 according to /sys/fs/cgroup/memory/memory.max_usage_in_bytes

Why does the container have 0 memory? It was my understanding that machine: true provided a VM with 8GB of RAM. Certainly not 0GB. What am I doing wrong?

#2

It should be 8G, yes :upside_down_face:

What machine image are you using? If you are using the default (i.e. you don’t specify one in your config) then take a look at the docs and see how to switch it. From memory, the default image is Debian Jessie, and this is old enough to be out of support now. Switch to a later image - the image names should be in the docs and/or in this forum.

Then retest your problem to see if it goes away.

#3

There is not enough information here.

Can you share your docker compose file?

Can you share which container had exit code 137?

Per the kernel docs this field does not set the limit, it tells you how much memory has been used.

memory.max_usage_in_bytes # show max memory usage recorded

#4

My apologies. The docker-compose.yml:

version: "3"
services:
  this_server:
    build: docker/server
    volumes:
      - .:/app
    entrypoint: python -m main
    ports:
      - "443:5000"
    depends_on:
        - this_mongo
  this_mongo:
    image: mongo
    volumes:
      - mongo-data:/data/db
      - ./dumps:/home
volumes:
  mongo-data:

Essentially it’s a Flask app with a MongoDB backend. One thing I’ve uncovered is that it seems the culprit (from running small test scripts that succeeded and some that failed) is the line:
client = pymongo.MongoClient("this_mongo")
Without this line, a script run with the step
- run: docker-compose exec this_server python test.py
executes fine. But by simply adding this line, it results in the error posted above. Is there something about the service name this_mongo being passed to MongoClient that CircleCI doesn’t agree with?

I hope this is enough information. If not I’m happy to provide anything else.

#5

Quite a few Mongo issues have been reported in this forum. I assume that you have not manually pulled mongo prior to the docker-compose up, so that will do a pull for you - see if the hash has changed in the console output? Since you are implicitly using the latest tag, I wonder if there has been a new release that works differently - perhaps it is a major release.

Try pinning this to a specific version (mongo:x.y.z) based on what was working for you previously.

Also, delete your local mongo image and try running the containers via Compose locally - you should get the same failure there.

#6

I’ve tried two things unsuccessfully so far:

  • switched Mongo image to circleci/mongo for the Mongo container
  • deleted my local Mongo image so the build would pull mongo and mirror the latest image being used in the circleci build

The circleci/mongo image resulted in the same exit code. Pulling the latest mongo locally had no effect on the results, and these tests are still able to run fine locally, while failing on circleci.

#7

I believe that the Docker Compose config format allows the specification of per-container RAM limits. Could you try limiting Mongo to, say, 3G of RAM? I wonder if it is grabbing more than it should, leaving the Python container with insufficient RAM to do its work.

#8

That does seem logical. Unfortunately I tried that and got the same error, no differences.

I’m really at a loss as to what’s causing this, especially since the error is so vague.

#9

So I do believe we’ve at least clearly identified the culprit:

.circleci/config.yml:

version: 2.1

jobs:
    run_tests:
        machine: true
        working_directory: ~/app
        steps:
            - checkout
            - run: docker-compose up -d this_server
            - run: docker-compose exec this_server python -c "import pymongo; print(pymongo.MongoClient('this_mongo'))"

workflows:
    run_all:
        jobs:
            - run_tests

docker-compose.yml:

version: "2"
services:
  this_server:
    build: docker/server
    volumes:
      - .:/app
    entrypoint: python -m main
    ports:
      - "443:5000"
    depends_on:
        - this_mongo
  this_mongo:
    image: mongo
    volumes:
      - mongo-data:/data/db
      - ./dumps:/home
    mem_limit: 2G
volumes:
  mongo-data:

This configuration produces the error:

#!/bin/bash -eo pipefail
docker-compose exec this_server python -c "import pymongo; print(pymongo.MongoClient('this_mongo'))"
MongoClient(host=['this_mongo:27017'], document_class=dict, tz_aware=False, connect=True)
Exited with code 137

Hint: Exit code 137 typically means the process is killed because it was running out of memory
Hint: Check if you can optimize the memory usage in your app
Hint: Max memory usage of this container is 0
 according to /sys/fs/cgroup/memory/memory.max_usage_in_bytes

So it’s quite clearly the initialization of MongoClient that is causing the issue. I also tested this with a dummy connection string pymongo.MongoClient("this_fails") and it produces the same Exit Code 137.

#10

Now this is interesting.

This script that I’m trying to run: essentially populates a test database. I started spraying the script with print statements, and it actually does make it all the way through to the end; it makes a connection to this_mongo, it loads some data from a file, and inserts that data into collections in this_mongo successfully. All the way to the end. And then it exits with code 137, and is considered to have failed.

I tried putting sys.exit(0) at the end of the script, to no avail.

So surprisingly, the interaction with the mongo container is actually happening just fine. It just throws an error code after all of that, for reasons I once again do not understand.

For clarity:

import sys
import os
import yaml
import pymongo

client = pymongo.MongoClient("this_mongo")
print("CLIENT INITIALIZED")

with open(os.path.abspath("test/test_database.yml")) as f:
    database = yaml.safe_load(f)
print("DATA LOADED")

for collection, docs in database.items(): # keys are a, b, c, d
    client.interecho_beta.drop_collection(f"{collection}_test")
    client.interecho_beta.__getattr__(f"{collection}_test").insert_many(docs)
    print("COLLECTION {} POPULATED".format(collection))

print("EVERYTHING IS DONE")
sys.exit(0)
#!/bin/bash -eo pipefail
docker-compose exec this_server python test.py
CLIENT INITIALIZED
DATA LOADED
COLLECTION a POPULATED
COLLECTION b POPULATED
COLLECTION c POPULATED
COLLECTION d POPULATED
EVERYTHING IS DONE
Exited with code 137

Hint: Exit code 137 typically means the process is killed because it was running out of memory
Hint: Check if you can optimize the memory usage in your app
Hint: Max memory usage of this container is 0
 according to /sys/fs/cgroup/memory/memory.max_usage_in_bytes
#11

Well this is embarrassing. How often do forum posts like this end with an error that was totally on the user’s end?

The this_server container was exiting after a few seconds because while initializing the Flask server, it came across the need for some files that were in .gitignore, and thus were not available to CircleCI. The container was up for long enough to immediately run a docker-compose exec ..., but would then crash (I suppose) due to the entrypoint failing. It wasn’t until I added steps of

- run: bash -c "sleep 5"
- run: docker ps -a
- run: docker-compose logs this_server

That I was able to identify this.

Anyway, thanks for your help.

3 Likes
#12

I’m really glad you were able to get this sorted. Thanks for posting your solution to.

I could write a book with the number of times I’ve banged my head against a desk for hours to discover the problem was something simple. :slight_smile:

This also helps me understand why the memory usage was reported at zero because the container effectively never started it seems.

#13

Nice work! You could also do docker-compose up this_server (without the -d) to see what the logs are at start-up. The detachment is the way to go under normal conditions, but to see if everything is OK, not doing that may be useful in the future.

closed #14

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.