Networking using docker-compose in docker executor problem

docker
circle.yml

#1

Hi,

I am new to circleci and so far I was really happy with it.
Now I ran into a problem and I don’t know what I’m doing wrong.
Maybe this is something very easy, but I don’t see the it.

In short

I can’t make containers talk to each other on circleci.

Problem

Basically what I wanted to do is start a server container and a client container, and then let them talk to each other.
I created a minimal example here: https://github.com/mRcSchwering/circleci-integration-test

The README.md basically explains the desired outcome.
I have a .circleci/config.yml like this:

version: 2
jobs:
  build:
    docker:
      - image: docker:18.03.0-ce-git
    steps:
      - checkout
      - setup_remote_docker
      - run:
          name: Install docker-compose
          command: |
            apk --update add py2-pip
            /usr/bin/pip2 install docker-compose
            docker-compose --version
      - run:
          name: Start Container
          command: |
            docker-compose up -d
            docker-compose ps
      - run:
          name: Let client talk to server
          command: |
            docker-compose run client psql -h server -p 5432 -U postgres -c "\l"

In a docker container, docker-compose is installed, which is then used to start a server and a client (postgres here). In the last step I am telling the client to query the server. However, it cannot find the server:

#!/bin/sh -eo pipefail
docker-compose run client psql -h server -p 5432 -U postgres -c "\l"
Starting project_server_1 ... 

^@^@psql: could not connect to server: Connection refused
	Is the server running on host "server" (172.18.0.2) and accepting
	TCP/IP connections on port 5432?
Exited with code 2

Files

The docker-compose.yml looks like this

version: '2'

services:
  server:
    image: postgres:9.5.12-alpine
    networks:
      - internal
    expose:
      - '5432'
  client:
    build:
      context: .
    networks:
      - internal
    depends_on:
        - server

networks:
  internal:
    driver: bridge

where the client is built from a dockerfile like this

FROM alpine:3.7
RUN apk --no-cache add postgresql-client && rm -rf /var/cache/apk/*

Note

If I repeat everything on my Linux (also with docker-in-docker) it works.
But I guess some things work completely different on circleci.
I found some people mentioning that on circleci networking and bind mounts can be tricky but I didn’t find anything that can help me.
There is this doc but I thought I am doing this already.
Then there is the protobuf project from johanbrandhorst on github where he seems to do the same thing on circleci successfully.
But I cannot figure out what’s different there…

Anyway I would really appreciate your help. So far I have given up on this.

Best
Marc


#2

Not really. If you are running Docker Compose on CircleCI, unless you’re doing something unusual, it is the same as if you are running it locally.

My first thought is that all of your DC services share a custom network, in which case there’s no point setting a custom network. They all normally share the default network, so try that instead. (In fact, even if you do need internal for other reasons, I’d be inclined to remove it temporarily, just to start with the simplest possible config).

Also, you haven’t set a project name in the DC start call (e.g. docker-compose --project-name mytests) and a result of that is that your internal network names would be aliased with a prefix of the directory name your project is sitting in.

That would mean if your directory name is app then your hostname would be specified with -h app_server. For me that feels a bit brittle, so I prefer to specify a project name.

To debug this, try this:

  • Run your build again with the SSH option, and wait for it to fail
  • SSH in and ensure that DC is still up with docker ps
  • Shell into the client using docker exec -it <client-container-name> sh
  • Double-check the networking and the DNS name by pinging the other container (ping app_server as per the project naming rule I mentioned earlier)

Once you have confirmed the server is running and you have the name correct, if it still does not work, try this:

  • Shell into the client and try making a PostgreSQL client connection. If this works in an SSH session, that would point to the connection being made before the db server has fully started, and a sleep or wait might fix this

#3

Hahaa :slight_smile:
It was indeed the server still starting.
A sleep 5 solved it.

Thanks a lot.
I was somewhere in the spheres of docker-in-docker networking issues.


#4

This topic was automatically closed 41 days after the last reply. New replies are no longer allowed.