Docker compose containers ignoring /etc/hosts

#1

So I am trying to migrate to circle 2.0 (late, I know), and I running into an issue with my docker compose setup where it was fine in 1.0.

I have 4 containers, and the server needs to be able to connect to the other containers.

Here is the docker-compose.yml:

server:
  build: .
  command: nodemon /app/app.js
  volumes:
    - .:/app
    - ./container/cert.pem:/etc/ssl/certs/cert.pem
    - ./container/key.pem:/etc/ssl/private/key.pem
    - ./container/ca.pem:/etc/ssl/certs/ca.pem
  links:
    - mongo:mongo
    - redis:redis
    - elasticsearch:elasticsearch
  restart: "on-failure:10"

mongo:
  build: ./container/

redis:
  image: redis:4.0.8

elasticsearch:
  image: elasticsearch:2.4
  ports:
    - "9200:9200"
    - "9300:9300"

So everything seems to be working fine, all containers come up, but server can’t connect the other containers which is weird, because here is the /etc/hosts inside the server container:

127.0.0.1 localhost
172.17.0.3 csmapi_elasticsearch_1 9f1f4689d9a5
172.17.0.4 csmapi_mongo_1 e2b565610ef9
172.17.0.2 csmapi_redis_1 dbfc8724512a
172.17.0.3 csmapi_elasticsearch_1 9f1f4689d9a5

So if csmapi_server_1 tries to connect to mongodb://csmapi_mongo_1, it fails, but through testing, if I connect instead to the IP (172.17.0.4), it works.

So why, even though the hosts file is correct, is my container not able to resolve the other containers hostnames? Why does it not seem to be using /etc/hosts?

#2

Can you share the config.yml file? Or open a ticket if it’s private.

#3

Try getting an SSH session on the build container, shell into the server container, and try ping csmapi_mongo_1. Does that work? You might have to install ping to get this to work.

#4

@halfer I have tried ping and I get unknown host csmapi_mongo_1

@drazisil here is the relevant config.yml:

env_config: &env_config
  working_directory: ~/blah/csm-api
  machine: true

restore_repo: &restore_repo
  restore_cache:
    keys:
      - repo-v1-{{ .Branch }}-{{ .Revision }}
      - repo-v1-{{ .Branch }}
      - repo-v1

docker_login: &docker_login
  docker login -u $DOCKER_USER -p $DOCKER_PASS

env_set: &env_set
  if [ "$CIRCLE_BRANCH" = "circleci-20-test" ]; then echo "export ENV=dev" >> "$BASH_ENV"; else echo "export ENV=$CIRCLE_BRANCH" >> "$BASH_ENV"; fi

version: 2
jobs:
  build:
    <<: *env_config
    steps:
      - *restore_repo
      - checkout
      - run: *docker_login
      - run: *env_set
      - run: sed -i "s/NODE_ENV=.*/NODE_ENV=$ENV/" Dockerfile
      - run: sed -i "s/GIT_SHA1=.*/GIT_SHA1=$CIRCLE_SHA1/" Dockerfile
      - run: mkdir -p ~/docker
      - save_cache:
          key: repo-v1-{{ .Branch }}-{{ .Revision }}
          paths:
            - .

  test:
    <<: *env_config
    steps:
      - *restore_repo
      - run: *docker_login
      - run: *env_set
      - run: make test_unit
      - run: echo "NODE_ENV=$ENV" >> env.list
      - run: echo "MONGO_HOST=mongodb://csmapi_mongo_1" >> env.list
      - run: echo "MONGO_DB=api-local" >> env.list
      - run: echo "MONGO_USER=api" >> env.list
      - run: echo "MONGO_PASS=password" >> env.list
      - run: echo "MONGO_AUTH_DB=admin" >> env.list
      - run: echo "REDIS_HOST=csmapi_redis_1" >> env.list
      - run: echo "ES_HOST=csmapi_elasticsearch_1" >> env.list
      - run: make test_integration

workflows:
  version: 2
  build_test_and_deploy:
    jobs:
      - build:
          filters:
            branches:
              only:
                - circleci-20-test

      - test:
          requires:
            - build
          filters:
            branches:
              only:
                - circleci-20-test

The test job fails at make test_integration because the integration tests need to have csmapi_server_1 connect to csmapi_mongo_1 and even though the env.list file is being read properly and mongodb://csmapi_mongo_1 is being used, the integration tests fail with ‘Unable to connect to database at mongodb://csmapi_mongo_1/api-local’. I can connect manually with mongo shell using the same connection information.

I don’t know why dns resolution isn’t using /etc/hosts, but it seems to be something with circle 2.0. If I change

- run: echo "MONGO_HOST=mongodb://csmapi_mongo_1" >> env.list

to

- run: echo "MONGO_HOST=mongodb://172.17.0.4" >> env.list

then the integration test connects to mongo but then fails at redis (because the same issue). All containers seems to be on the same network (‘bridge’).

#5

Try pinging the service name instead, mongo. I’m used to service names becoming DNS names, but I think that’s because I tend to use DC 2.0 rather than 3.x. What version are you using? It appears to have been omitted from your YAML file.

#6

so it looks like I made a mistake when running ping and I wasn’t actually in the container. It seems like I can ping both mongo and csmapi_mongo_1 from the server container which tells me there is something wrong with how node is trying to reach the mongo container. let me test some other things and get back

2 Likes