Docker run -> error: bind address already in use

networking
docker

#1

Hello,

I am using the same circle.yml (for about 6 months) with no issue (main settings):

machine:
  services:
    - docker
  ruby:
    #version: rbx-2.0.0-p451
  node:
    version: 0.12.0

dependencies:
  override:
    - docker login -e $DOCKER_EMAIL -u $DOCKER_USER -p $DOCKER_PASSWORD
    - docker info
    - docker pull $ML7_LATEST_DOCKER_IMAGE
    - docker pull $ML8_LATEST_DOCKER_IMAGE

But the build stopped working today. The error is:

docker run --name ml7 -d -p 8000:8000 -p 8001:8001 -p 8002:8002 -p $ML_APP_PORT:$ML_APP_PORT -p $ML_XCC_PORT:$ML_XCC_PORT $ML7_LATEST_DOCKER_IMAGE

WARNING: Your kernel does not support memory swappiness capabilities, memory swappiness discarded.
c3957d12e6e7b2fcb7d234d06a8768535e956238f1c4c945952bffde9c470a0c

Error response from daemon: Cannot start container c3957d12e6e7b2fcb7d234d06a8768535e956238f1c4c945952bffde9c470a0c: listen tcp 0.0.0.0:9306: bind: address already in use docker run --name ml7 -d -p 8000:8000 -p 8001:8001 -p 8002:8002 -p $ML_APP_PORT:$ML_APP_PORT -p $ML_XCC_PORT:$ML_XCC_PORT $ML7_LATEST_DOCKER_IMAGE returned exit code 1

How to troubleshoot / fix this issue?


#2

Can you try to SSH into the container and run netstat -plntu to see what is taking up that port? Can you paste the output of that command here.


#3

ubuntu@box1350:~$ netstat -plntu
(No info could be read for “-p”: geteuid()=1000 but you should be root.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:9312 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:5984 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:2181 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:25672 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:46088 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:27017 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:6379 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:11211 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:4369 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:28017 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:6099 0.0.0.0:* LISTEN -
tcp 0 0 10.0.4.1:53 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN -
tcp 0 0 10.0.3.101:53 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:5432 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:9306 0.0.0.0:* LISTEN -
tcp6 0 0 :::5672 :::* LISTEN -
tcp6 0 0 :::6379 :::* LISTEN -
tcp6 0 0 :::6099 :::* LISTEN -
tcp6 0 0 :::5432 :::* LISTEN -
udp 0 0 10.0.4.1:53 0.0.0.0:* -
udp 0 0 127.0.0.1:53 0.0.0.0:* -
udp 0 0 10.0.3.101:53 0.0.0.0:* -
udp 0 0 0.0.0.0:67 0.0.0.0:* -
udp 0 0 0.0.0.0:68 0.0.0.0:* -
udp 0 0 172.17.42.1:123 0.0.0.0:* -
udp 0 0 10.0.4.1:123 0.0.0.0:* -
udp 0 0 10.0.3.101:123 0.0.0.0:* -
udp 0 0 127.0.0.1:123 0.0.0.0:* -
udp 0 0 0.0.0.0:123 0.0.0.0:* -
udp 0 0 0.0.0.0:5353 0.0.0.0:* -
udp 0 0 127.0.0.1:11211 0.0.0.0:* -
udp 0 0 0.0.0.0:45731 0.0.0.0:* -
udp6 0 0 :::58433 :::* -
udp6 0 0 :::123 :::* -
udp6 0 0 :::5353 :::* -


#4

Changing port 9306 in my environment variable addresses the issue.

Is 9306 a reserved port?


#5

Hello,

It is not a reserved port, but it may be in use. Could you please rerun the commend as sudo so we can see the process names?


#6

Warning: Permanently added ‘[54.242.202.162]:64600’ (ECDSA) to the list of known hosts.
Last login: Sat Oct 31 17:00:27 2015 from 10.65.152.103
ubuntu@box1057:~$ sudo netstat -plntu
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:53748 0.0.0.0:* LISTEN 667/java
tcp 0 0 10.0.4.1:53 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN -
tcp 0 0 10.0.3.65:53 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 574/sshd
tcp 0 0 0.0.0.0:5432 0.0.0.0:* LISTEN 856/postgres
tcp 0 0 0.0.0.0:9306 0.0.0.0:* LISTEN 1949/searchd
tcp 0 0 0.0.0.0:9312 0.0.0.0:* LISTEN 1949/searchd
tcp 0 0 127.0.0.1:5984 0.0.0.0:* LISTEN 1210/beam.smp
tcp 0 0 0.0.0.0:2181 0.0.0.0:* LISTEN 667/java
tcp 0 0 0.0.0.0:25672 0.0.0.0:* LISTEN 1572/beam.smp
tcp 0 0 0.0.0.0:27017 0.0.0.0:* LISTEN 674/mongod
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:6379 0.0.0.0:* LISTEN 1918/redis-server *
tcp 0 0 127.0.0.1:11211 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:4369 0.0.0.0:* LISTEN 1355/epmd
tcp 0 0 0.0.0.0:28017 0.0.0.0:* LISTEN 674/mongod
tcp 0 0 0.0.0.0:6099 0.0.0.0:* LISTEN 2229/Xvfb
tcp6 0 0 :::5432 :::* LISTEN 856/postgres
tcp6 0 0 :::5672 :::* LISTEN 1572/beam.smp
tcp6 0 0 :::6379 :::* LISTEN 1918/redis-server *
tcp6 0 0 :::6099 :::* LISTEN 2229/Xvfb
udp 0 0 0.0.0.0:39348 0.0.0.0:* -
udp 0 0 10.0.4.1:53 0.0.0.0:* -
udp 0 0 127.0.0.1:53 0.0.0.0:* -
udp 0 0 10.0.3.65:53 0.0.0.0:* -
udp 0 0 0.0.0.0:67 0.0.0.0:* -
udp 0 0 0.0.0.0:68 0.0.0.0:* 528/dhclient3
udp 0 0 172.17.42.1:123 0.0.0.0:* -
udp 0 0 10.0.4.1:123 0.0.0.0:* -
udp 0 0 10.0.3.65:123 0.0.0.0:* -
udp 0 0 127.0.0.1:123 0.0.0.0:* -
udp 0 0 0.0.0.0:123 0.0.0.0:* -
udp 0 0 0.0.0.0:5353 0.0.0.0:* -
udp 0 0 127.0.0.1:11211 0.0.0.0:* -
udp6 0 0 :::35334 :::* -
udp6 0 0 :::123 :::* -
udp6 0 0 :::5353 :::* -


#7

I have the same problem, it seems some ports that I want to use such as 5432 for postgres and 27017 for mongodb, are already assigned despite the fact that these servers are not listed as services in my circle.yml

Then when I try to start my own Docker containers exposing those ports it fails with “failed to create endpoint postgres on network bridge: listen tcp 0.0.0.0:5432: bind: address already in use”

$ netstat -plntu
(No info could be read for "-p": geteuid()=1000 but you should be root.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:5984          0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:27017           0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:6099            0.0.0.0:*               LISTEN      -
tcp        0      0 10.0.4.1:53             0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:5432            0.0.0.0:*               LISTEN      -
tcp6       0      0 :::27018                :::*                    LISTEN      -
tcp6       0      0 :::3306                 :::*                    LISTEN      -
tcp6       0      0 :::6099                 :::*                    LISTEN      -
tcp6       0      0 fe80::acdf:dcff:fe11:53 :::*                    LISTEN      -
tcp6       0      0 :::5432                 :::*                    LISTEN      -
tcp6       0      0 :::5433                 :::*                    LISTEN      -
udp        0      0 10.0.4.1:53             0.0.0.0:*                           -
udp        0      0 0.0.0.0:67              0.0.0.0:*                           -
udp        0      0 0.0.0.0:68              0.0.0.0:*                           -
udp        0      0 172.18.0.1:123          0.0.0.0:*                           -
udp        0      0 10.0.4.1:123            0.0.0.0:*                           -
udp        0      0 172.17.71.67:123        0.0.0.0:*                           -
udp        0      0 127.0.0.1:123           0.0.0.0:*                           -
udp        0      0 0.0.0.0:123             0.0.0.0:*                           -
udp        0      0 0.0.0.0:33182           0.0.0.0:*                           -
udp6       0      0 fe80::acdf:dcff:fe11:53 :::*                                -
udp6       0      0 fe80::50d5:c6ff:fe9:123 :::*                                -
udp6       0      0 fe80::68a5:a0ff:fef:123 :::*                                -
udp6       0      0 fe80::42:d1ff:feaf::123 :::*                                -
udp6       0      0 fe80::20ab:e0ff:fe3:123 :::*                                -
udp6       0      0 ::1:123                 :::*                                -
udp6       0      0 fe80::acdf:dcff:fe1:123 :::*                                -
udp6       0      0 :::123                  :::*                                -
udp6       0      0 :::18351                :::*                                -

I note that the PID column is empty

Is there a way I can free up these ports?

It complicates my deployment somewhat if I have to assign randomised port numbers within CircleCI to avoid these phantom clashes


#8

FWIW I am new to CircleCI

I just read that CircleCI tries to auto-detect your databases, like it does everything else

I tried:

database:
  override:
    - echo "Override CircleCI" auto-detected databases"

in case CircleCI was auto-detecting my project used postgres and mongo… but still same problem, port clash


#9

ah, you have to sudo netstat -plntu to see the PIDs!

So this shows that indeed postgres, mongodb and mysql are already running and getting in my way :frowning:

$ sudo netstat -plntu
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:5984          0.0.0.0:*               LISTEN      20732/beam.smp
tcp        0      0 0.0.0.0:27017           0.0.0.0:*               LISTEN      1474/mongod
tcp        0      0 0.0.0.0:6099            0.0.0.0:*               LISTEN      45/Xvfb
tcp        0      0 10.0.4.1:53             0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1746/sshd
tcp        0      0 0.0.0.0:5432            0.0.0.0:*               LISTEN      2136/postgres
tcp6       0      0 :::3306                 :::*                    LISTEN      2040/mysqld
tcp6       0      0 :::6099                 :::*                    LISTEN      45/Xvfb
tcp6       0      0 fe80::9c18:44ff:fe3d:53 :::*                    LISTEN      -
tcp6       0      0 :::5432                 :::*                    LISTEN      2136/postgres
udp        0      0 0.0.0.0:31459           0.0.0.0:*                           1593/dhclient
udp        0      0 10.0.4.1:53             0.0.0.0:*                           -
udp        0      0 0.0.0.0:67              0.0.0.0:*                           -
udp        0      0 0.0.0.0:68              0.0.0.0:*                           1593/dhclient
udp        0      0 172.18.0.1:123          0.0.0.0:*                           2767/ntpd
udp        0      0 10.0.4.1:123            0.0.0.0:*                           2767/ntpd
udp        0      0 172.17.99.159:123       0.0.0.0:*                           2767/ntpd
udp        0      0 127.0.0.1:123           0.0.0.0:*                           2767/ntpd
udp        0      0 0.0.0.0:123             0.0.0.0:*                           2767/ntpd
udp6       0      0 fe80::9c18:44ff:fe3d:53 :::*                                -
udp6       0      0 fe80::42:f3ff:feb8::123 :::*                                2767/ntpd
udp6       0      0 fe80::9c18:44ff:fe3:123 :::*                                2767/ntpd
udp6       0      0 fe80::b402:67ff:fed:123 :::*                                2767/ntpd
udp6       0      0 ::1:123                 :::*                                2767/ntpd
udp6       0      0 :::123                  :::*                                2767/ntpd
udp6       0      0 :::20850                :::*                                1593/dhclient

How do I prevent this situation?


#10

You can see all of the services that are started by default here: https://circleci.com/docs/build-image-trusty/

If you don’t want these to be started, then you should shut them down via your circle.yml

For instance:

machine:
  post:
    - sudo service postgresl stop

#11
machine:
  post:
    - sudo service postgresl stop
  services:
    - docker

gave me the following error under “Configure the build”:


invalid command

Action failed: Configure the build

#12

Turns out that sudo service postgresl stop needs to be executed in test instead. Not sure why.


#13