Circleci/mongo randomly fails

We’re using circleci/mongo:3 in our .circleci/config.yml and for about 2 weeks it randomly fails some times to boot.
The error:

2019-05-02T12:25:38.857+0000 E -        [main] Error saving history file: FileOpenFailed: Unable to open() file /home/mongodb/.dbshell: Unknown error

Our job definition:

version: 2
jobs:
  # CLI publishing
  cli_publish:
    docker:
      - image: circleci/node:8
      - image: circleci/mongo:3
        environment:
          MONGO_INITDB_ROOT_USERNAME: prisma
          MONGO_INITDB_ROOT_PASSWORD: prisma
      - image: circleci/postgres:10.3-alpine
        environment:
          POSTGRES_USER: prisma
          POSTGRES_PASSWORD: prisma
          POSTGRES_PW: prisma
          POSTGRES_DB: prisma
      - image: circleci/mysql:5
        environment:
          MYSQL_ROOT_PASSWORD: prisma
    steps:
      - checkout
      - run: sudo apt install postgresql-client
      - run: echo "//registry.npmjs.org/:_authToken=${NPM_TOKEN}" > ~/.npmrc
      - run: cd cli && yarn
      - run:
          name: Waiting for PostgreSQL to start
          command: |
            for i in `seq 1 10`;
            do
              nc -z localhost 5432 && echo Success && exit 0
              echo -n .
              sleep 2
            done
            echo Failed waiting for Postgres && exit 1
      - run:
          name: Wait for MySQL to start
          # preinstalled in circleci/* docker image
          command: dockerize -wait tcp://127.0.0.1:3306 -timeout 120s
      - run:
          name: Wait for Mongo to start
          # preinstalled in circleci/* docker image
          command: dockerize -wait tcp://127.0.0.1:27017 -timeout 120s
      - run: ./cli/scripts/test_ci.sh
      - run: ./cli/scripts/publish.sh

An example build that just worked: https://circleci.com/gh/prisma/prisma/5154
A build that failed: https://circleci.com/gh/prisma/prisma/5171

You can see the full job definition here: https://github.com/prisma/prisma/blob/master/.circleci/config.yml

I also tried out

  • circleci/mongo:4-ram
  • circleci/mongo:3.4
  • mongo:4
  • circleci/mongo:4.1-bionic-ram
  • circleci/mongo:3.6
  • adding command: [--smallfiles] as found on some circleci docs page
  • and putting mongo to the end in the docker images list:
      - image: circleci/node:8
      - image: circleci/postgres:10.3-alpine
        environment:
          POSTGRES_USER: prisma
          POSTGRES_PASSWORD: prisma
          POSTGRES_PW: prisma
          POSTGRES_DB: prisma
      - image: circleci/mysql:5
        environment:
          MYSQL_ROOT_PASSWORD: prisma
      - image: circleci/mongo:3
        command: [--smallfiles]
        environment:
          MONGO_INITDB_ROOT_USERNAME: prisma
          MONGO_INITDB_ROOT_PASSWORD: prisma

As all of this didn’t work, this seems to be a problem with CircleCI, as it used to work.

If you need more information, please let me know!

We’re having the same issue - mongo sometimes fails to finish booting in circleci.

One note - the error you posted doesn’t seem to be related to the issue, it happens during successful runs too (check the successful job you posted https://circleci.com/gh/prisma/prisma/5154)

Error saving history file: FileOpenFailed: Unable to open() file /home/mongodb/.dbshell: Unknown error

During failed runs, the mongo container logs stop here:

2019-05-02T12:25:38.915+0000 I CONTROL  [signalProcessingThread] shutting down with code:0

Whereas successful runs include the same output as failed runs but continue from there:

2019-04-30T17:54:14.965+0000 I CONTROL  [signalProcessingThread] shutting down with code:0

MongoDB init process complete; ready for start up.

2019-04-30T17:54:15.992+0000 I CONTROL  [initandlisten] MongoDB starting : pid=7 port=27017 dbpath=/data/db 64-bit host=a3bbefded09f
2019-04-30T17:54:15.992+0000 I CONTROL  [initandlisten] db version v3.6.12

I can’t link to our jobs since they’re private but we’re having the exact same problem. Using mongo:4.1, also tried mongo:4.0. It started happening intermittently some time in the past couple weeks and we could get around it by re-running workflows until they passed. As of yesterday it’s been happening consistently.

I am experiencing the same exact issue here with ALL circleci/mongo:3.X and mongo:3.X images.

I unfortunately can’t link jobs either as they are private.

Do you mean it is not intermittent for you? Log a support ticket at support@circleci.com and describe the problem in detail, including links to failed builds. A non-intermittent example will be helpful for engineers to see what the issue is.

This is a bit confusing - your image that worked is circleci/mongo:3 and your image that failed uses mongo:4. They are obviously not the same images - do you have example links where the image was the same, but the results were different?

Just a quick follow-up.

I reached out to support who pointed me in the direction of the current open issues with the official mongo images, namely: https://github.com/docker-library/mongo/issues/323

I’m not entirely convinced this is related to that issue though since prior successful runs log save errors as well.

In order to mitigate this, I removed the MONGO_INITDB_ROOT_USERNAME and MONGO_INITDB_ROOT_PASSWORD env vars so that the mongo container wouldn’t attempt to create a user and would init an auth-less db - my usecase doesn’t truly require an auth-ed db (tests which leverage a temp mongo db).




I did attempt to manually create a root db user in CI but couldn’t re-authenticate connections with it so I’m not sure whats going on there.

For anyone interested in trying it:

 - run:
     name: Create Mongo Super user
     command: |
       mongo --eval 'db.createUser({ user: "user", pwd: "password", roles: [ { role: "root", db: "admin" } ] });'
       mongo -u user -p password --authenticationDatabase admin --eval 'db.getUsers()'  # THIS FAILS
1 Like

I’m experiencing the same issue here. It ran previously just fine and suddenly stopped working without touching the config.

oakhan3 thank you! I was able to disable the auth as mongo image was just for testing purposes.

1 Like

We’re experiencing same issue on out circleci account.

We try to run locally docker images

  • circleci/mongo
  • mongo:3.2
  • mongo:4.0
  • custom images
    All images locally work fine with MONGO_INIT_* env customization.

It’s maybe regarding circleci container orchestration issue?

Just wanted to bump this thread and share that this is still very much an issue

It may be worth logging this as a specific support issue. I can’t see that anyone has logged it from this thread, and I wonder if it might not get looked at without a ticket.