Not able to connect to mongo container

Build Environment Databases
, , , ,
1

Hi I am not able to connect to mongoDB container.
The tests were running fine till may 1st.
Haven’t changed the config.yml.
config.yml

docker:
      # use django-base, which has basic stuff needed to run tests
      - image: doselect/django-base:latest
      - image: postgres:9.6
        ports:
          - "5432:5432"
        environment:
          POSTGRES_USER: root
          POSTGRES_DB: circle-test_test
      - image: rabbitmq
        ports:
          - "5672:5672"
          - "15672:15672"
          - "55672:55672"
      - image: doselect/mongo-fixture:latest
        environment:
          - MONGO_INITDB_DATABASE: analytics_db
        ports:
          - "27017:27017"
      - image: neo4j:latest
        environment:
          - NEO4J_AUTH: neo4j/central
        ports:
          - "7474:7474"
          - "7687:7687"
      - image: redis
        ports:
          - "6379:6379"

doselect/mongo-fixture:latest is a public image. we use this to initialise user and its role.

I tried to rebuild previous tests which were already running, those builds were also failing because of the error.

The logs for mongo-fixture:latest container

about to fork child process, waiting until server is ready for connections.
forked process: 275
2019-05-04T12:14:21.060+0000 I CONTROL  [main] ***** SERVER RESTARTED *****
2019-05-04T12:14:21.066+0000 I CONTROL  [initandlisten] MongoDB starting : pid=275 port=27017 dbpath=/data/db 64-bit host=f16d2fe56531
2019-05-04T12:14:21.066+0000 I CONTROL  [initandlisten] db version v3.2.20
2019-05-04T12:14:21.066+0000 I CONTROL  [initandlisten] git version: a7a144f40b70bfe290906eb33ff2714933544af8
2019-05-04T12:14:21.066+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1t  3 May 2016
2019-05-04T12:14:21.066+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2019-05-04T12:14:21.066+0000 I CONTROL  [initandlisten] modules: none
2019-05-04T12:14:21.066+0000 I CONTROL  [initandlisten] build environment:
2019-05-04T12:14:21.066+0000 I CONTROL  [initandlisten]     distmod: debian81
2019-05-04T12:14:21.066+0000 I CONTROL  [initandlisten]     distarch: x86_64
2019-05-04T12:14:21.066+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2019-05-04T12:14:21.066+0000 I CONTROL  [initandlisten] options: { net: { bindIp: "127.0.0.1", port: 27017, ssl: { mode: "disabled" } }, processManagement: { fork: true, pidFilePath: "/tmp/docker-entrypoint-temp-mongod.pid" }, systemLog: { destination: "file", logAppend: true, path: "/proc/159/fd/1" } }
2019-05-04T12:14:21.083+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=40G,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),verbose=(recovery_progress),
child process started successfully, parent exiting
2019-05-04T12:14:21.101+0000 I NETWORK  [HostnameCanonicalizationWorker] Starting hostname canonicalization worker
2019-05-04T12:14:21.102+0000 I FTDC     [initandlisten] Initializing full-time diagnostic data capture with directory '/data/db/diagnostic.data'
2019-05-04T12:14:21.102+0000 I NETWORK  [initandlisten] waiting for connections on port 27017
2019-05-04T12:14:21.140+0000 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:55874 #1 (1 connection now open)
2019-05-04T12:14:21.141+0000 I NETWORK  [conn1] end connection 127.0.0.1:55874 (0 connections now open)

/usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/init.js
2019-05-04T12:14:21.181+0000 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:55876 #2 (1 connection now open)
Successfully added user: {
	"user" : "central",
	"roles" : [
		{
			"role" : "dbAdmin",
			"db" : "xxxxx"
		}
	]
}
2019-05-04T12:14:21.197+0000 I NETWORK  [conn2] end connection 127.0.0.1:55876 (0 connections now open)


killing process with pid: 275
2019-05-04T12:14:21.214+0000 I CONTROL  [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
2019-05-04T12:14:21.214+0000 I FTDC     [signalProcessingThread] Shutting down full-time diagnostic data capture
2019-05-04T12:14:21.214+0000 I CONTROL  [signalProcessingThread] now exiting
2019-05-04T12:14:21.214+0000 I NETWORK  [signalProcessingThread] shutdown: going to close listening sockets...
2019-05-04T12:14:21.214+0000 I NETWORK  [signalProcessingThread] closing listening socket: 6
2019-05-04T12:14:21.214+0000 I NETWORK  [signalProcessingThread] closing listening socket: 7
2019-05-04T12:14:21.214+0000 I NETWORK  [signalProcessingThread] removing socket file: /tmp/mongodb-27017.sock
2019-05-04T12:14:21.214+0000 I NETWORK  [signalProcessingThread] shutdown: going to flush diaglog...
2019-05-04T12:14:21.214+0000 I NETWORK  [signalProcessingThread] shutdown: going to close sockets...
2019-05-04T12:14:21.214+0000 I STORAGE  [signalProcessingThread] WiredTigerKVEngine shutting down
2019-05-04T12:14:21.300+0000 I STORAGE  [signalProcessingThread] shutdown: removing fs lock...
2019-05-04T12:14:21.300+0000 I CONTROL  [signalProcessingThread] dbexit:  rc: 0

Job was canceled

Before when the tests were successfully running the logs for mongo container was like.

about to fork child process, waiting until server is ready for connections.
forked process: 252
2019-04-30T08:54:56.425+0000 I CONTROL  [main] ***** SERVER RESTARTED *****
2019-04-30T08:54:56.433+0000 I CONTROL  [initandlisten] MongoDB starting : pid=252 port=27017 dbpath=/data/db 64-bit host=58ddb0c907e9
2019-04-30T08:54:56.433+0000 I CONTROL  [initandlisten] db version v3.2.20
2019-04-30T08:54:56.433+0000 I CONTROL  [initandlisten] git version: a7a144f40b70bfe290906eb33ff2714933544af8
2019-04-30T08:54:56.433+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1t  3 May 2016
2019-04-30T08:54:56.433+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2019-04-30T08:54:56.433+0000 I CONTROL  [initandlisten] modules: none
2019-04-30T08:54:56.433+0000 I CONTROL  [initandlisten] build environment:
2019-04-30T08:54:56.433+0000 I CONTROL  [initandlisten]     distmod: debian81
2019-04-30T08:54:56.433+0000 I CONTROL  [initandlisten]     distarch: x86_64
2019-04-30T08:54:56.433+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2019-04-30T08:54:56.433+0000 I CONTROL  [initandlisten] options: { net: { bindIp: "127.0.0.1", port: 27017, ssl: { mode: "disabled" } }, processManagement: { fork: true, pidFilePath: "/tmp/docker-entrypoint-temp-mongod.pid" }, systemLog: { destination: "file", logAppend: true, path: "/proc/61/fd/1" } }
2019-04-30T08:54:56.450+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=40G,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),verbose=(recovery_progress),
2019-04-30T08:54:56.483+0000 I CONTROL  [initandlisten] 
2019-04-30T08:54:56.483+0000 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2019-04-30T08:54:56.483+0000 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2019-04-30T08:54:56.483+0000 I CONTROL  [initandlisten] 
child process started successfully, parent exiting
2019-04-30T08:54:56.490+0000 I FTDC     [initandlisten] Initializing full-time diagnostic data capture with directory '/data/db/diagnostic.data'
2019-04-30T08:54:56.490+0000 I NETWORK  [initandlisten] waiting for connections on port 27017
2019-04-30T08:54:56.492+0000 I NETWORK  [HostnameCanonicalizationWorker] Starting hostname canonicalization worker
2019-04-30T08:54:56.533+0000 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:44860 #1 (1 connection now open)
2019-04-30T08:54:56.538+0000 I NETWORK  [conn1] end connection 127.0.0.1:44860 (0 connections now open)

/usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/init.js
2019-04-30T08:54:56.652+0000 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:44866 #2 (1 connection now open)
Successfully added user: {
	"user" : "central",
	"roles" : [
		{
			"role" : "dbAdmin",
			"db" : "xxxxx"
		}
	]
}
2019-04-30T08:54:56.701+0000 I NETWORK  [conn2] end connection 127.0.0.1:44866 (0 connections now open)


killing process with pid: 252
2019-04-30T08:54:56.727+0000 I CONTROL  [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
2019-04-30T08:54:56.727+0000 I FTDC     [signalProcessingThread] Shutting down full-time diagnostic data capture
2019-04-30T08:54:56.727+0000 I CONTROL  [signalProcessingThread] now exiting
2019-04-30T08:54:56.727+0000 I NETWORK  [signalProcessingThread] shutdown: going to close listening sockets...
2019-04-30T08:54:56.727+0000 I NETWORK  [signalProcessingThread] closing listening socket: 6
2019-04-30T08:54:56.727+0000 I NETWORK  [signalProcessingThread] closing listening socket: 7
2019-04-30T08:54:56.727+0000 I NETWORK  [signalProcessingThread] removing socket file: /tmp/mongodb-27017.sock
2019-04-30T08:54:56.727+0000 I NETWORK  [signalProcessingThread] shutdown: going to flush diaglog...
2019-04-30T08:54:56.727+0000 I NETWORK  [signalProcessingThread] shutdown: going to close sockets...
2019-04-30T08:54:56.727+0000 I STORAGE  [signalProcessingThread] WiredTigerKVEngine shutting down
2019-04-30T08:54:56.932+0000 I STORAGE  [signalProcessingThread] shutdown: removing fs lock...
2019-04-30T08:54:56.932+0000 I CONTROL  [signalProcessingThread] dbexit:  rc: 0

MongoDB init process complete; ready for start up.

2019-04-30T08:54:57.754+0000 I CONTROL  [initandlisten] MongoDB starting : pid=61 port=27017 dbpath=/data/db 64-bit host=58ddb0c907e9

I tried ssh into the container but all the /proc/ logs were access denied.
I tried running the mongo-fixture image locally and it worked fine.
There are no changes in the mongo-fixture image.

Because of this issue we are not able to test anything.
Thanks.

2

Several posts have been made about Mongo recently here; do a search in this forum, and sort by recency.

3

None of the posts gives information why this is happening ? or what can be done to resolve it.

Was there some update pushed from circleci side? I see the kernel version is changed.

Plus after ssh into the machine all the process are in defunct state.

1 Like
4

The image is doselect/django-base:latest, which presumably is not maintained by CircleCI (who will stick to circleci/ prefixes). Have a look at the (40 character) image digest in your last few successful runs, and then see if the digest is different when the failures started. If so, that is very likely to be your culprit.

I would suggest pinning the image to that hash (doselect/django-base@sha256:<hash-goes-here>) to see if that fixes it. If it does, then the trouble is upstream. I would advise removing latest and swapping to a known working version anyway - you will get major version changes without some sort of pinning, and that will make your CI brittle.

Don’t leave it pinned to a hash though - select a working version and use that, and then bump it up periodically based on tested images.

5

I have a similar issue with @kloudd
When I build an image using circleci/mongo including a db init script as

FROM circleci/mongo:4
COPY mongo-init.js /docker-entrypoint-initdb.d/

using an init script such as

db.auth('root', 'password');
org = db.getSiblingDB('organization');
org.createUser({
   user: 'organization',
   pwd: 'password',
   roles: [ "readWrite" ]
});
org.company.createIndex({ aggregateId: 1 });
org.company.createIndex({ timestamp: 1 });

and then use this in my circleci config as

version: 2
jobs:
  build:

  docker:
    - image: circleci/golang

    - image: paulvitic/circle-mongo
      environment:
        MONGO_INITDB_ROOT_USERNAME: root
        MONGO_INITDB_ROOT_PASSWORD: password
        MONGO_INITDB_DATABASE: admin

I can not connect to the container as I run the go mongo integration tests

On the other hand I have no problems running integration tests with, e.g. circleci rabbitMQ container in a similar fashion

6

Try getting a shell after a failed build, and see if you can connect to Mongo using the console binary. The usual problem here is not waiting for Mongo to spin up before starting your test suite.

7

I wait for containers to respond with the following

    steps:
      - checkout

      - run:
          name: Waiting for MongoDB to be ready
          command: |
            for i in `seq 1 30`;
            do
              nc -z localhost 27017 && echo Success && exit 0
              echo -n .
              sleep 1
            done
            echo Failed waiting for MongoDB && exit 1

      - run:
          name: Waiting for RabbitMQ to be ready
          command: |
            for i in `seq 1 30`;
            do
              nc -z localhost 5672 && echo Success && exit 0
              echo -n .
              sleep 1
            done
            echo Failed waiting for RabbitMQ && exit 1

      - run:
          name: Copy organization source into GOPATH src directory and pull all dependencies
          command: |
            cp -r organization/src/organization.prodeng /go/src/organization.prodeng
            cd /go/src/organization.prodeng
            go get -v -t -d ./...

      - run:
          name: Execute tests
          environment:
            PROFILE: "ci"
          command: |
            go test -v ./...

Again, this works with RabbitMQ image, but not MongoDB image. In fact if I try the official MongoDB docker image without the database init script, and it also works. The problem seems to happen when there is an init script in /docker-entrypoint-initdb.d/ directory. I can see in the logs that the init script runs, but then there is no connection.

8

@paulvitic
What i feel there were some changes in dockerd.

I could clearly see some upgraded docker version.

I guess they should either should have updated the same to their users.

Anyways my issue was only with initializing the mongoDB or creating users for the same.

I used a CirlceCI Image then also i was not able to Initialize DB with -

          MONGO_INITDB_DATABASE: xxxxxx```

What i did for resolution : 

1-  I just spawned a container with mongo with a circleCI image. 
2- installed mongo client on my testing container and Then i did initialized my mongo container with DB and superusers. Through my testing container using mongo clients. 

To install the clients i used following steps. 

 - run:
      name: update /etc/apt/sources.list for apt-get update
      command: |
        sudo rm /etc/apt/sources.list
        echo "deb http://archive.debian.org/debian/ jessie-backports main" | sudo tee -a /etc/apt/sources.list
        echo "deb-src http://archive.debian.org/debian/ jessie-backports main" | sudo tee -a /etc/apt/sources.list
        echo "Acquire::Check-Valid-Until false;" | sudo tee -a /etc/apt/apt.conf.d/10-nocheckvalid
        echo 'Package: *\nPin: origin "archive.debian.org"\nPin-Priority: 500' | sudo tee -a /etc/apt/preferences.d/10-archive-pin
        sudo apt-get update
   - run:
      name: install mongo client org-tools
      command: |
        sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv XXXXXXX
        echo "deb http://repo.mongodb.org/apt/ubuntu trusty/mongodb-org/3.2 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.2.list
        sudo apt-get update


After this i initialized my DB and super user. 

I have made changes in source list, also Because i could see changes in the kernel version of circleCI container. 

Hence on update it was breaking up. So i had to make changes to source clients, to install mongo-clients. 

I hope this works for you. 

I also raised concern with the CCI team. 
They said they updated something internally, But they were not sure if the issue was because of that update. 
From that point of time there was no reply from the CCI team.
9

@halfer
Sir the containers get purged, after the tests, doing SSH.

the proc logs are also not available.

i wonder how are you suggesting to connect to the mongo container after the failed buid.

10

Thank you @kloudd. I will try this instead.
I am not too happy with this workaround as it makes the local dev machine integration testing and integration testing during ci quiet different from each other and therefore presents unnecessary complexity. Do you have a ticket for this so I can vote up?

1 Like