Circleci/mongo randomly fails

Build Environment
, , , ,
1

We’re using circleci/mongo:3 in our .circleci/config.yml and for about 2 weeks it randomly fails some times to boot.
The error:

2019-05-02T12:25:38.857+0000 E -        [main] Error saving history file: FileOpenFailed: Unable to open() file /home/mongodb/.dbshell: Unknown error

Our job definition:

version: 2
jobs:
  # CLI publishing
  cli_publish:
    docker:
      - image: circleci/node:8
      - image: circleci/mongo:3
        environment:
          MONGO_INITDB_ROOT_USERNAME: prisma
          MONGO_INITDB_ROOT_PASSWORD: prisma
      - image: circleci/postgres:10.3-alpine
        environment:
          POSTGRES_USER: prisma
          POSTGRES_PASSWORD: prisma
          POSTGRES_PW: prisma
          POSTGRES_DB: prisma
      - image: circleci/mysql:5
        environment:
          MYSQL_ROOT_PASSWORD: prisma
    steps:
      - checkout
      - run: sudo apt install postgresql-client
      - run: echo "//registry.npmjs.org/:_authToken=${NPM_TOKEN}" > ~/.npmrc
      - run: cd cli && yarn
      - run:
          name: Waiting for PostgreSQL to start
          command: |
            for i in `seq 1 10`;
            do
              nc -z localhost 5432 && echo Success && exit 0
              echo -n .
              sleep 2
            done
            echo Failed waiting for Postgres && exit 1
      - run:
          name: Wait for MySQL to start
          # preinstalled in circleci/* docker image
          command: dockerize -wait tcp://127.0.0.1:3306 -timeout 120s
      - run:
          name: Wait for Mongo to start
          # preinstalled in circleci/* docker image
          command: dockerize -wait tcp://127.0.0.1:27017 -timeout 120s
      - run: ./cli/scripts/test_ci.sh
      - run: ./cli/scripts/publish.sh

An example build that just worked: https://circleci.com/gh/prisma/prisma/5154
A build that failed: https://circleci.com/gh/prisma/prisma/5171

You can see the full job definition here: https://github.com/prisma/prisma/blob/master/.circleci/config.yml

I also tried out

  • circleci/mongo:4-ram
  • circleci/mongo:3.4
  • mongo:4
  • circleci/mongo:4.1-bionic-ram
  • circleci/mongo:3.6
  • adding command: [--smallfiles] as found on some circleci docs page
  • and putting mongo to the end in the docker images list:
      - image: circleci/node:8
      - image: circleci/postgres:10.3-alpine
        environment:
          POSTGRES_USER: prisma
          POSTGRES_PASSWORD: prisma
          POSTGRES_PW: prisma
          POSTGRES_DB: prisma
      - image: circleci/mysql:5
        environment:
          MYSQL_ROOT_PASSWORD: prisma
      - image: circleci/mongo:3
        command: [--smallfiles]
        environment:
          MONGO_INITDB_ROOT_USERNAME: prisma
          MONGO_INITDB_ROOT_PASSWORD: prisma

As all of this didn’t work, this seems to be a problem with CircleCI, as it used to work.

If you need more information, please let me know!

2

We’re having the same issue - mongo sometimes fails to finish booting in circleci.

One note - the error you posted doesn’t seem to be related to the issue, it happens during successful runs too (check the successful job you posted https://circleci.com/gh/prisma/prisma/5154)

Error saving history file: FileOpenFailed: Unable to open() file /home/mongodb/.dbshell: Unknown error

During failed runs, the mongo container logs stop here:

2019-05-02T12:25:38.915+0000 I CONTROL  [signalProcessingThread] shutting down with code:0

Whereas successful runs include the same output as failed runs but continue from there:

2019-04-30T17:54:14.965+0000 I CONTROL  [signalProcessingThread] shutting down with code:0

MongoDB init process complete; ready for start up.

2019-04-30T17:54:15.992+0000 I CONTROL  [initandlisten] MongoDB starting : pid=7 port=27017 dbpath=/data/db 64-bit host=a3bbefded09f
2019-04-30T17:54:15.992+0000 I CONTROL  [initandlisten] db version v3.6.12

I can’t link to our jobs since they’re private but we’re having the exact same problem. Using mongo:4.1, also tried mongo:4.0. It started happening intermittently some time in the past couple weeks and we could get around it by re-running workflows until they passed. As of yesterday it’s been happening consistently.

3

I am experiencing the same exact issue here with ALL circleci/mongo:3.X and mongo:3.X images.

I unfortunately can’t link jobs either as they are private.

4

Do you mean it is not intermittent for you? Log a support ticket at support@circleci.com and describe the problem in detail, including links to failed builds. A non-intermittent example will be helpful for engineers to see what the issue is.

5

This is a bit confusing - your image that worked is circleci/mongo:3 and your image that failed uses mongo:4. They are obviously not the same images - do you have example links where the image was the same, but the results were different?

6

Just a quick follow-up.

I reached out to support who pointed me in the direction of the current open issues with the official mongo images, namely: https://github.com/docker-library/mongo/issues/323

I’m not entirely convinced this is related to that issue though since prior successful runs log save errors as well.

In order to mitigate this, I removed the MONGO_INITDB_ROOT_USERNAME and MONGO_INITDB_ROOT_PASSWORD env vars so that the mongo container wouldn’t attempt to create a user and would init an auth-less db - my usecase doesn’t truly require an auth-ed db (tests which leverage a temp mongo db).




I did attempt to manually create a root db user in CI but couldn’t re-authenticate connections with it so I’m not sure whats going on there.

For anyone interested in trying it:

 - run:
     name: Create Mongo Super user
     command: |
       mongo --eval 'db.createUser({ user: "user", pwd: "password", roles: [ { role: "root", db: "admin" } ] });'
       mongo -u user -p password --authenticationDatabase admin --eval 'db.getUsers()'  # THIS FAILS
1 Like
7

I’m experiencing the same issue here. It ran previously just fine and suddenly stopped working without touching the config.

oakhan3 thank you! I was able to disable the auth as mongo image was just for testing purposes.

1 Like
8

We’re experiencing same issue on out circleci account.

We try to run locally docker images

  • circleci/mongo
  • mongo:3.2
  • mongo:4.0
  • custom images
    All images locally work fine with MONGO_INIT_* env customization.

It’s maybe regarding circleci container orchestration issue?

9

Just wanted to bump this thread and share that this is still very much an issue

10

It may be worth logging this as a specific support issue. I can’t see that anyone has logged it from this thread, and I wonder if it might not get looked at without a ticket.