How to permanently shutdown CircleCI default Postgres, Mongodb etc?

Build Environment
1

For various reasons (want specific version matching production environment, custom config etc) we are running our own Postgres and other servers via Docker in our CircleCI builds.

Despite the fact I did not request them in my circle.yml it seems that CircleCI starts up some services including Postgres and Mongodb by default.

I previously sought advice about this here and was advised to sudo service postgresql stop before trying to start my docker containers

This works most of the time but some percentage of the time the container fails to start with:

$ docker run -d --name postgres -p 5432:5432 postgres:latest
b3718197824603663612db13f9f1216d298bb4355f9ea332305aa3a7556a052a
Error response from daemon: Cannot start container b3718197824603663612db13f9f1216d298bb4355f9ea332305aa3a7556a052a: failed to create endpoint postgres on network bridge: listen tcp 0.0.0.0:5432: bind: address already in use

docker run -d --name postgres -p 5432:5432 postgres:latest
 returned exit code 1

I thought… well maybe the port has not been completely freed yet, so I added a step, after the shutdown but before starting my container, that runs a bash script with a wait loop using netcat polling to check that the port is free.

This still fails sometimes with same error, after netcat has reported that the port is unoccupied. So now I suspect that the default postgres service is being restarted automatically and I effectively have a race condition to start my container before the system restarts the clashing service.

Please advise a proper solution for this issue

2

So I re-ran the build with SSH enabled

It worked fine this time, as it does the majority of the time, which makes it harder to debug the problem.

I don’t really know what to look for but for example:

$ sudo tail /var/log/upstart/mongod.log
2016-10-11T12:03:53.797+0000 I NETWORK  [signalProcessingThread] closing listening socket: 5
2016-10-11T12:03:53.797+0000 I NETWORK  [signalProcessingThread] closing listening socket: 6
2016-10-11T12:03:53.797+0000 I NETWORK  [signalProcessingThread] removing socket file: /tmp/mongodb-27017.sock
2016-10-11T12:03:53.797+0000 I NETWORK  [signalProcessingThread] shutdown: going to flush diaglog...
2016-10-11T12:03:53.797+0000 I NETWORK  [signalProcessingThread] shutdown: going to close sockets...
2016-10-11T12:03:53.797+0000 I STORAGE  [signalProcessingThread] shutdown: waiting for fs preallocator...
2016-10-11T12:03:53.797+0000 I STORAGE  [signalProcessingThread] shutdown: closing all files...
2016-10-11T12:03:53.799+0000 I STORAGE  [signalProcessingThread] closeAllFiles() finished
2016-10-11T12:03:53.799+0000 I STORAGE  [signalProcessingThread] shutdown: removing fs lock...
2016-10-11T12:03:53.799+0000 I CONTROL  [signalProcessingThread] dbexit:  rc: 0

this does not appear to show repeated attempts to start the mongo service after I stopped it, so I am unsure what is the source of the port clash

I am wondering if Docker itself has a problem
https://github.com/docker/docker/issues/8714

3

currently trying an alternative approach where I retry the docker run until it works, instead of testing for port availability

in my circle.yml

    - "for i in {1..5}; do
        docker run -d --name postgres
          -p 5432:5432
          postgres:latest
        && break || sleep 3;
      done"
4

Same issue here. Although I stop postgresql service in pre dependencies and check that port is free with netstat, I get randomly the “bind: address already in use” issue with “docker-compose up”…

1 Like
5

Same issue here… have you found a solution to the problem?

edit: this seems to work

    - sudo service postgresql stop
    # wait for postgresql to shutdown
    - while sudo lsof -Pi :5432 -sTCP:LISTEN -t; do sleep 1; done
6

@olalonde, do you confirm that your solution works everytime? I tried it but it continues to fails sometimes.

7

I think it has failed at least once since I added that script :confused:

8

Also stuck with trying to get rid of the mongod process. Have tried killing it in just about every section of circle.yml, killing it multiple times, etc.

EDIT: See How to stop default/initial started services

9

The first line seems to be the key on Ubuntu 14.04:

- echo manual | sudo tee /etc/init/postgresql.override
- sudo service postgresql stop

Thanks to @reedflinch and http://askubuntu.com/questions/19320/how-to-enable-or-disable-services

Will update if issue occures again…

2 Likes
10

So… I’m still getting that error and have tried all the solutions :frowning: So annoying…

Screen Shot 2017-05-12 at 12.12.19.png2570Ă—1196 146 KB

Edit:

Now using this… appears to work for now, hopefully it will last longer than last workaround

    - |
      echo manual | sudo tee /etc/init/postgresql.override
      while sudo lsof -Pi :5432 -sTCP:LISTEN -t; do
        sudo service postgresql stop;
        sleep 1;
      done
      make run-test
11

Have you tried 2.0? It gives you full control over the build environment so only services that you define end up running. This will be the best long term strategy.

1 Like
12

Yes, I’m in the process of moving to 2.0 now.