Building and running the containers happens without issue. When it comes time to run pytest, I get this:
Hint: Exit code 137 typically means the process is killed because it was running out of memory
Hint: Check if you can optimize the memory usage in your app
Hint: Max memory usage of this container is 0
according to /sys/fs/cgroup/memory/memory.max_usage_in_bytes
Why does the container have 0 memory? It was my understanding that machine: true provided a VM with 8GB of RAM. Certainly not 0GB. What am I doing wrong?
What machine image are you using? If you are using the default (i.e. you don’t specify one in your config) then take a look at the docs and see how to switch it. From memory, the default image is Debian Jessie, and this is old enough to be out of support now. Switch to a later image - the image names should be in the docs and/or in this forum.
Essentially it’s a Flask app with a MongoDB backend. One thing I’ve uncovered is that it seems the culprit (from running small test scripts that succeeded and some that failed) is the line: client = pymongo.MongoClient("this_mongo")
Without this line, a script run with the step - run: docker-compose exec this_server python test.py
executes fine. But by simply adding this line, it results in the error posted above. Is there something about the service name this_mongo being passed to MongoClient that CircleCI doesn’t agree with?
I hope this is enough information. If not I’m happy to provide anything else.
Quite a few Mongo issues have been reported in this forum. I assume that you have not manually pulled mongo prior to the docker-compose up, so that will do a pull for you - see if the hash has changed in the console output? Since you are implicitly using the latest tag, I wonder if there has been a new release that works differently - perhaps it is a major release.
Try pinning this to a specific version (mongo:x.y.z) based on what was working for you previously.
Also, delete your local mongo image and try running the containers via Compose locally - you should get the same failure there.
switched Mongo image to circleci/mongo for the Mongo container
deleted my local Mongo image so the build would pull mongo and mirror the latest image being used in the circleci build
The circleci/mongo image resulted in the same exit code. Pulling the latest mongo locally had no effect on the results, and these tests are still able to run fine locally, while failing on circleci.
I believe that the Docker Compose config format allows the specification of per-container RAM limits. Could you try limiting Mongo to, say, 3G of RAM? I wonder if it is grabbing more than it should, leaving the Python container with insufficient RAM to do its work.
#!/bin/bash -eo pipefail
docker-compose exec this_server python -c "import pymongo; print(pymongo.MongoClient('this_mongo'))"
MongoClient(host=['this_mongo:27017'], document_class=dict, tz_aware=False, connect=True)
Exited with code 137
Hint: Exit code 137 typically means the process is killed because it was running out of memory
Hint: Check if you can optimize the memory usage in your app
Hint: Max memory usage of this container is 0
according to /sys/fs/cgroup/memory/memory.max_usage_in_bytes
So it’s quite clearly the initialization of MongoClient that is causing the issue. I also tested this with a dummy connection string pymongo.MongoClient("this_fails") and it produces the same Exit Code 137.
This script that I’m trying to run: essentially populates a test database. I started spraying the script with print statements, and it actually does make it all the way through to the end; it makes a connection to this_mongo, it loads some data from a file, and inserts that data into collections in this_mongo successfully. All the way to the end. And then it exits with code 137, and is considered to have failed.
I tried putting sys.exit(0) at the end of the script, to no avail.
So surprisingly, the interaction with the mongo container is actually happening just fine. It just throws an error code after all of that, for reasons I once again do not understand.
For clarity:
import sys
import os
import yaml
import pymongo
client = pymongo.MongoClient("this_mongo")
print("CLIENT INITIALIZED")
with open(os.path.abspath("test/test_database.yml")) as f:
database = yaml.safe_load(f)
print("DATA LOADED")
for collection, docs in database.items(): # keys are a, b, c, d
client.interecho_beta.drop_collection(f"{collection}_test")
client.interecho_beta.__getattr__(f"{collection}_test").insert_many(docs)
print("COLLECTION {} POPULATED".format(collection))
print("EVERYTHING IS DONE")
sys.exit(0)
#!/bin/bash -eo pipefail
docker-compose exec this_server python test.py
CLIENT INITIALIZED
DATA LOADED
COLLECTION a POPULATED
COLLECTION b POPULATED
COLLECTION c POPULATED
COLLECTION d POPULATED
EVERYTHING IS DONE
Exited with code 137
Hint: Exit code 137 typically means the process is killed because it was running out of memory
Hint: Check if you can optimize the memory usage in your app
Hint: Max memory usage of this container is 0
according to /sys/fs/cgroup/memory/memory.max_usage_in_bytes
Well this is embarrassing. How often do forum posts like this end with an error that was totally on the user’s end?
The this_server container was exiting after a few seconds because while initializing the Flask server, it came across the need for some files that were in .gitignore, and thus were not available to CircleCI. The container was up for long enough to immediately run a docker-compose exec ..., but would then crash (I suppose) due to the entrypoint failing. It wasn’t until I added steps of
Nice work! You could also do docker-compose up this_server (without the -d) to see what the logs are at start-up. The detachment is the way to go under normal conditions, but to see if everything is OK, not doing that may be useful in the future.
