Not able to connect to mongo container

Hi I am not able to connect to mongoDB container.
The tests were running fine till may 1st.
Haven’t changed the config.yml.
config.yml

docker:
      # use django-base, which has basic stuff needed to run tests
      - image: doselect/django-base:latest
      - image: postgres:9.6
        ports:
          - "5432:5432"
        environment:
          POSTGRES_USER: root
          POSTGRES_DB: circle-test_test
      - image: rabbitmq
        ports:
          - "5672:5672"
          - "15672:15672"
          - "55672:55672"
      - image: doselect/mongo-fixture:latest
        environment:
          - MONGO_INITDB_DATABASE: analytics_db
        ports:
          - "27017:27017"
      - image: neo4j:latest
        environment:
          - NEO4J_AUTH: neo4j/central
        ports:
          - "7474:7474"
          - "7687:7687"
      - image: redis
        ports:
          - "6379:6379"

doselect/mongo-fixture:latest is a public image. we use this to initialise user and its role.

I tried to rebuild previous tests which were already running, those builds were also failing because of the error.

The logs for mongo-fixture:latest container

about to fork child process, waiting until server is ready for connections.
forked process: 275
2019-05-04T12:14:21.060+0000 I CONTROL  [main] ***** SERVER RESTARTED *****
2019-05-04T12:14:21.066+0000 I CONTROL  [initandlisten] MongoDB starting : pid=275 port=27017 dbpath=/data/db 64-bit host=f16d2fe56531
2019-05-04T12:14:21.066+0000 I CONTROL  [initandlisten] db version v3.2.20
2019-05-04T12:14:21.066+0000 I CONTROL  [initandlisten] git version: a7a144f40b70bfe290906eb33ff2714933544af8
2019-05-04T12:14:21.066+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1t  3 May 2016
2019-05-04T12:14:21.066+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2019-05-04T12:14:21.066+0000 I CONTROL  [initandlisten] modules: none
2019-05-04T12:14:21.066+0000 I CONTROL  [initandlisten] build environment:
2019-05-04T12:14:21.066+0000 I CONTROL  [initandlisten]     distmod: debian81
2019-05-04T12:14:21.066+0000 I CONTROL  [initandlisten]     distarch: x86_64
2019-05-04T12:14:21.066+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2019-05-04T12:14:21.066+0000 I CONTROL  [initandlisten] options: { net: { bindIp: "127.0.0.1", port: 27017, ssl: { mode: "disabled" } }, processManagement: { fork: true, pidFilePath: "/tmp/docker-entrypoint-temp-mongod.pid" }, systemLog: { destination: "file", logAppend: true, path: "/proc/159/fd/1" } }
2019-05-04T12:14:21.083+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=40G,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),verbose=(recovery_progress),
child process started successfully, parent exiting
2019-05-04T12:14:21.101+0000 I NETWORK  [HostnameCanonicalizationWorker] Starting hostname canonicalization worker
2019-05-04T12:14:21.102+0000 I FTDC     [initandlisten] Initializing full-time diagnostic data capture with directory '/data/db/diagnostic.data'
2019-05-04T12:14:21.102+0000 I NETWORK  [initandlisten] waiting for connections on port 27017
2019-05-04T12:14:21.140+0000 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:55874 #1 (1 connection now open)
2019-05-04T12:14:21.141+0000 I NETWORK  [conn1] end connection 127.0.0.1:55874 (0 connections now open)

/usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/init.js
2019-05-04T12:14:21.181+0000 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:55876 #2 (1 connection now open)
Successfully added user: {
	"user" : "central",
	"roles" : [
		{
			"role" : "dbAdmin",
			"db" : "xxxxx"
		}
	]
}
2019-05-04T12:14:21.197+0000 I NETWORK  [conn2] end connection 127.0.0.1:55876 (0 connections now open)


killing process with pid: 275
2019-05-04T12:14:21.214+0000 I CONTROL  [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
2019-05-04T12:14:21.214+0000 I FTDC     [signalProcessingThread] Shutting down full-time diagnostic data capture
2019-05-04T12:14:21.214+0000 I CONTROL  [signalProcessingThread] now exiting
2019-05-04T12:14:21.214+0000 I NETWORK  [signalProcessingThread] shutdown: going to close listening sockets...
2019-05-04T12:14:21.214+0000 I NETWORK  [signalProcessingThread] closing listening socket: 6
2019-05-04T12:14:21.214+0000 I NETWORK  [signalProcessingThread] closing listening socket: 7
2019-05-04T12:14:21.214+0000 I NETWORK  [signalProcessingThread] removing socket file: /tmp/mongodb-27017.sock
2019-05-04T12:14:21.214+0000 I NETWORK  [signalProcessingThread] shutdown: going to flush diaglog...
2019-05-04T12:14:21.214+0000 I NETWORK  [signalProcessingThread] shutdown: going to close sockets...
2019-05-04T12:14:21.214+0000 I STORAGE  [signalProcessingThread] WiredTigerKVEngine shutting down
2019-05-04T12:14:21.300+0000 I STORAGE  [signalProcessingThread] shutdown: removing fs lock...
2019-05-04T12:14:21.300+0000 I CONTROL  [signalProcessingThread] dbexit:  rc: 0

Job was canceled

Before when the tests were successfully running the logs for mongo container was like.

about to fork child process, waiting until server is ready for connections.
forked process: 252
2019-04-30T08:54:56.425+0000 I CONTROL  [main] ***** SERVER RESTARTED *****
2019-04-30T08:54:56.433+0000 I CONTROL  [initandlisten] MongoDB starting : pid=252 port=27017 dbpath=/data/db 64-bit host=58ddb0c907e9
2019-04-30T08:54:56.433+0000 I CONTROL  [initandlisten] db version v3.2.20
2019-04-30T08:54:56.433+0000 I CONTROL  [initandlisten] git version: a7a144f40b70bfe290906eb33ff2714933544af8
2019-04-30T08:54:56.433+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1t  3 May 2016
2019-04-30T08:54:56.433+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2019-04-30T08:54:56.433+0000 I CONTROL  [initandlisten] modules: none
2019-04-30T08:54:56.433+0000 I CONTROL  [initandlisten] build environment:
2019-04-30T08:54:56.433+0000 I CONTROL  [initandlisten]     distmod: debian81
2019-04-30T08:54:56.433+0000 I CONTROL  [initandlisten]     distarch: x86_64
2019-04-30T08:54:56.433+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2019-04-30T08:54:56.433+0000 I CONTROL  [initandlisten] options: { net: { bindIp: "127.0.0.1", port: 27017, ssl: { mode: "disabled" } }, processManagement: { fork: true, pidFilePath: "/tmp/docker-entrypoint-temp-mongod.pid" }, systemLog: { destination: "file", logAppend: true, path: "/proc/61/fd/1" } }
2019-04-30T08:54:56.450+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=40G,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),verbose=(recovery_progress),
2019-04-30T08:54:56.483+0000 I CONTROL  [initandlisten] 
2019-04-30T08:54:56.483+0000 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2019-04-30T08:54:56.483+0000 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2019-04-30T08:54:56.483+0000 I CONTROL  [initandlisten] 
child process started successfully, parent exiting
2019-04-30T08:54:56.490+0000 I FTDC     [initandlisten] Initializing full-time diagnostic data capture with directory '/data/db/diagnostic.data'
2019-04-30T08:54:56.490+0000 I NETWORK  [initandlisten] waiting for connections on port 27017
2019-04-30T08:54:56.492+0000 I NETWORK  [HostnameCanonicalizationWorker] Starting hostname canonicalization worker
2019-04-30T08:54:56.533+0000 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:44860 #1 (1 connection now open)
2019-04-30T08:54:56.538+0000 I NETWORK  [conn1] end connection 127.0.0.1:44860 (0 connections now open)

/usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/init.js
2019-04-30T08:54:56.652+0000 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:44866 #2 (1 connection now open)
Successfully added user: {
	"user" : "central",
	"roles" : [
		{
			"role" : "dbAdmin",
			"db" : "xxxxx"
		}
	]
}
2019-04-30T08:54:56.701+0000 I NETWORK  [conn2] end connection 127.0.0.1:44866 (0 connections now open)


killing process with pid: 252
2019-04-30T08:54:56.727+0000 I CONTROL  [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
2019-04-30T08:54:56.727+0000 I FTDC     [signalProcessingThread] Shutting down full-time diagnostic data capture
2019-04-30T08:54:56.727+0000 I CONTROL  [signalProcessingThread] now exiting
2019-04-30T08:54:56.727+0000 I NETWORK  [signalProcessingThread] shutdown: going to close listening sockets...
2019-04-30T08:54:56.727+0000 I NETWORK  [signalProcessingThread] closing listening socket: 6
2019-04-30T08:54:56.727+0000 I NETWORK  [signalProcessingThread] closing listening socket: 7
2019-04-30T08:54:56.727+0000 I NETWORK  [signalProcessingThread] removing socket file: /tmp/mongodb-27017.sock
2019-04-30T08:54:56.727+0000 I NETWORK  [signalProcessingThread] shutdown: going to flush diaglog...
2019-04-30T08:54:56.727+0000 I NETWORK  [signalProcessingThread] shutdown: going to close sockets...
2019-04-30T08:54:56.727+0000 I STORAGE  [signalProcessingThread] WiredTigerKVEngine shutting down
2019-04-30T08:54:56.932+0000 I STORAGE  [signalProcessingThread] shutdown: removing fs lock...
2019-04-30T08:54:56.932+0000 I CONTROL  [signalProcessingThread] dbexit:  rc: 0

MongoDB init process complete; ready for start up.

2019-04-30T08:54:57.754+0000 I CONTROL  [initandlisten] MongoDB starting : pid=61 port=27017 dbpath=/data/db 64-bit host=58ddb0c907e9

I tried ssh into the container but all the /proc/ logs were access denied.
I tried running the mongo-fixture image locally and it worked fine.
There are no changes in the mongo-fixture image.

Because of this issue we are not able to test anything.
Thanks.

Several posts have been made about Mongo recently here; do a search in this forum, and sort by recency.

None of the posts gives information why this is happening ? or what can be done to resolve it.

Was there some update pushed from circleci side? I see the kernel version is changed.

Plus after ssh into the machine all the process are in defunct state.

The image is doselect/django-base:latest, which presumably is not maintained by CircleCI (who will stick to circleci/ prefixes). Have a look at the (40 character) image digest in your last few successful runs, and then see if the digest is different when the failures started. If so, that is very likely to be your culprit.

I would suggest pinning the image to that hash (doselect/django-base@sha256:<hash-goes-here>) to see if that fixes it. If it does, then the trouble is upstream. I would advise removing latest and swapping to a known working version anyway - you will get major version changes without some sort of pinning, and that will make your CI brittle.

Don’t leave it pinned to a hash though - select a working version and use that, and then bump it up periodically based on tested images.