Running dockerized services as part of integration tests

docker

#1

Hello, everyone,

I am migrating to CircleCi 2.0 from a working 1.0 build. Because it’s a mixed language setup of Scala, Python and Javascript, I am using a custom image derived from circleci/openjdk:8-node-browsers. I was able to run pretty much everything smoothly, caching of maven and npm dependencies works just fine.

But I have one integration test that is blowing up. It uses docker-it-scala on top of Spotify’s docker-client to pull and run an Elasticsearch service for the duration of the integration test (and shuts it down afterwards). Right now it’s failing during the startup phase. I can tell it pulls the image, sends a command to docker to start the container, gets an id back, then hangs. It times out after 600 seconds, which is the deadline I set on my test. I cannot rule out a problem with the library itself, but it seems that the CircleCI remote docker interface is not responding properly to the command.

Here’s the log from mvn -X:

    INFO   [17:39:33.445] [pool-17-thread-2] com.spotify.docker.client.DefaultDockerClient -  Creating container with ContainerConfig: ContainerConfig{hostname=null, domainname=null, user=null, attachStdin=false, attachStdout=null, attachStderr=null, portSpecs=null, exposedPorts=[9200, 9300], tty=false, openStdin=null, stdinOnce=null, env=[discovery.type=single-node], cmd=null, image=elasticsearch:5.1.2, volumes=[], workingDir=null, entrypoint=null, networkDisabled=null, onBuild=null, labels=null, macAddress=null, hostConfig=HostConfig{binds=[], blkioWeight=null, blkioWeightDevice=null, blkioDeviceReadBps=null, blkioDeviceWriteBps=null, blkioDeviceReadIOps=null, blkioDeviceWriteIOps=null, containerIdFile=null, lxcConf=null, privileged=null, portBindings={9200=[PortBinding{hostIp=0.0.0.0, hostPort=}], 9300=[PortBinding{hostIp=0.0.0.0, hostPort=}]}, links=null, publishAllPorts=null, dns=null, dnsOptions=null, dnsSearch=null, extraHosts=null, volumesFrom=null, capAdd=null, capDrop=null, networkMode=null, securityOpt=null, devices=null, memory=null, memorySwap=null, memorySwappiness=null, memoryReservation=null, nanoCpus=null, cpuPeriod=null, cpuShares=null, cpusetCpus=null, cpusetMems=null, cpuQuota=null, cgroupParent=null, restartPolicy=null, logConfig=null, ipcMode=null, ulimits=null, pidMode=null, shmSize=null, oomKillDisable=null, oomScoreAdj=null, autoRemove=null, pidsLimit=null, tmpfs=null, readonlyRootfs=null, storageOpt=null}, stopSignal=null, healthcheck=null, networkingConfig=null} 
INFO   [17:39:36.343] [pool-17-thread-2] com.spotify.docker.client.DefaultDockerClient -  Starting container with Id: dddfb60c4460c7970a8e1b3f83b81d3e5ed54bc3436cfc1c233a2d849b026419 
ERROR  [17:49:33.370] [main] com.worldsense.pipeline.EsIndexerTransformerTest -  Exception during container initialization 
java.util.concurrent.TimeoutException: Futures timed out after [600 seconds]
        at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:223)
        at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:227)
        at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:190)
        at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
        at scala.concurrent.Await$.result(package.scala:190)
        at com.whisk.docker.DockerKit$class.startAllOrFail(DockerKit.scala:51)

#2

What version of that Scala image are you using, and how are you starting it up? Is it a secondary image in your YAML, or are you starting it manually with docker run?


#3

The Scala code is the code under test, checked out from github (private repo). The config reads more or less like this:

# CircleCI configuration file.
version: 2
jobs:
  build:
    docker:
      - image: worldsense/circleci:1.0.1
    steps:
      - checkout
      - setup_remote_docker
      - run:
          command: mvn -X -Pcircleci2 -Dtest=com.worldsense.pipeline.EsIndexerTransformerTest test

I have omitted a sequence of steps restore_cache / mvn dependency:go-offline / save_cache, which I believe is not relevant.

The test itself is running docker commands via Docker API. It runs just fine on CircleCI 1.0, as well as on my local machine. From the docs on separation of environments I suspect there is a --network flag I can pass to the test so it connects to the service successfully. I have also investigated passing other environment variables such as in this thread


#4

Ah, right. OK, so I think this is an architectural issue.

When CircleCI sees the image list, it takes the first one and starts a container for your builds. Any subsequent ones are spun up and used as secondary services merged into the same networking stack (this does not apply in your case since you only have one).

So, when you get to the run command, you cannot contact Docker because you are inside that container. Docker is on the outside and is not reachable, by design.

I guess a solution is to use docker:17.05.0-ce-git as your base image, which will give you an empty Docker environment (Docker in Docker) and then you can spin up stuff before your tests run.


#5

Thanks, Jon.

I don’t think that my client is not being able to contact the Docker daemon. I can see that the image is correctly pulled, so my docker client is talking to someone. Event starting the service seems to work, because it gives me a container ID. But the service is unreachable (or at least it’s not reachable on 0.0.0.0:9200), so my health check never succeeds. I imagine the service is reachable on a different address, though, but I don’t really know what addresses I could test.

I have circleci/openjdk:8-node-browsers as my base image, and it would be painful to change the base image. It would be nice to have docker:17.05.0-ce as a service image, and I suspect many other people migrating to 2.0 from 1.0 will hit this problem. I expected the remote docker from setup_remote_docker to work in the same fashion as a “pure” docker daemon, otherwise docker-compose would not work at all.


#6

Well, the alternative to that is to install Docker manually in the image you have - that’s pretty easy to do as an alternative. However your most recent comments suggest that Docker is installed already in worldsense/circleci:1.0.1, and is started as well.

Are you able to do docker ps in a run step to verify this?


#7

When running the image locally, I see /usr/bin/docker is there. Using docker history I confirmed it was installed by circleci/openjdk:

<missing>                                                                 3 weeks ago         /bin/sh -c DOCKERIZE_URL="https://circle-downloads.s3.amazonaws.com/circleci-images/cache/linux-amd64/dockerize-latest.tar.gz"   && curl --silent --show-error --location --fail --retry 3 --output /tmp/dockerize-linux-amd64.tar.gz $DOCKERIZE_URL   && tar -C /usr/local/bin -xzvf /tmp/dockerize-linux-amd64.tar.gz   && rm -rf /tmp/dockerize-linux-amd64.tar.gz   && dockerize --version                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               8.53MB              
<missing>                                                                 3 weeks ago         /bin/sh -c COMPOSE_URL="https://circle-downloads.s3.amazonaws.com/circleci-images/cache/linux-amd64/docker-compose-latest"   && curl --silent --show-error --location --fail --retry 3 --output /usr/bin/docker-compose $COMPOSE_URL   && chmod +x /usr/bin/docker-compose   && docker-compose version                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       10.9MB              
<missing>                                                                 3 weeks ago         /bin/sh -c set -ex   && export DOCKER_VERSION=$(curl --silent --fail --retry 3 https://download.docker.com/linux/static/stable/x86_64/ | grep -o -e 'docker-[.0-9]*-ce\.tgz' | sort -r | head -n 1)   && DOCKER_URL="https://download.docker.com/linux/static/stable/x86_64/${DOCKER_VERSION}"   && echo Docker URL: $DOCKER_URL   && curl --silent --show-error --location --fail --retry 3 --output /tmp/docker.tgz "${DOCKER_URL}"   && ls -lha /tmp/docker.tgz   && tar -xz -C /tmp -f /tmp/docker.tgz   && mv /tmp/docker/* /usr/bin   && rm -rf /tmp/docker /tmp/docker.tgz   && which docker   && (docker version || true) 

However the daemon is not running and the service is not installed. I believe this is intentional. I’m trying to create a bridge network and pass it to the test will let you know how it goes.


#8

hi again,

I think you’re right and I will need a full docker engine running inside my custom image, otherwise I’d need an overlay network between the two, which doesn’t seem to be the case. But that might be an interesting feature for setup_remote_docker.

My theory as to why docker compose works, it’s because all containers in the composition run on the remote docker, so they can communicate between themselves.


#9

Absolutely, yes. I use DC inside a CircleCI container to run integration tests across ~10 other images, and they connect with each other fine.


#10