Hey all,
We’re running a CI pipeline starting with the docker image “circleci/node:12.13.0” and recently started seeing some weird behavior. We bring up a docker cluster with docker-compose to run a bunch of tests. We’re used to seeing docker-compose print the stdoout from each of the containers and seeing that in our circleci dashboard. But recently (possibly after moving from an alpine linux image to circleci/node:12.13.0) it started exhibiting this behavior where at some point, the output of one of our containers (the one actually running the tests) would just stop showing up in the circleci dashboard. It wasn’t that output entirely stopped, since once that container exited, we’d see the output of docker-compose in that step saying things like “container exited with exit code XXXX.” I ssh’d into the machine and could see all the output I expected in the docker logs for the container, so the output/execution was mostly working correctly.
I tried a bunch of things to fix this and nothing worked until on a hunch I switched from using the docker-compose in the image to the release one from github. (This was based on a vague hunch I had that the one in the container was running python2, and that the issue I was seeing was related to buffered output fixed in the python3 implementation). This worked! We now see all the output every time.
I’m posting this partly to ask “why did this work” and also partly to help others in case they run into the same issue and partly to alert the circleci team that something might be weird with their official docker compose binary.
fwiw the docker-compose from the circleci image is version
docker-compose version 1.25.4, build unknown
and from github https://github.com/docker/compose/releases/download/1.25.4/docker-compose-(uname -s)-(uname -m)
docker-compose version 1.25.4, build 8d51620a