I have a docker image from a vendor, for a piece of proprietary software that our team needs to integrate with. I have the image in a private dockerhub repository, and I can pull it and run it locally with no problem:
I would expect to be able to do the same thing in my CI build, but it doesn’t work:
circleci@8e622a3afdea:~$ docker run -d --name mediaflux2 --publish 8888:8888 --mac-address 02:42:ac:11:00:02 eosadler/tigerdata
15d9483d6ca426343e4ace1db71395686d2172d1a9cb13c301e62c19f0e6a72b
circleci@8e622a3afdea:~$ curl 0.0.0.0:8888
curl: (7) Failed to connect to 0.0.0.0 port 8888 after 0 ms: Connection refused
circleci@8e622a3afdea:~$ curl localhost:8888
curl: (7) Failed to connect to localhost port 8888 after 0 ms: Connection refused
I can get it to run if I put the whole thing in a docker-compose, but that makes my CI build go from ~3 minutes to > 20 minutes, which won’t work for us.
Can you provide your config.yml file or a cut-down version that results in the same issue?
The fact that if you move your config into a compose file it takes so much longer is odd, unless it is getting stuck on some form of timeout issue as it loads the environment.
One thing, why are you specifying a mac address, rather than just letting docker do so? The one you have selected is in the range that docker will use for networks and containers. Within a simple deployment with just one docker network defined that value will be assigned to the first container started, but the CircleCI environment may have already assigned it.
We are specifying the mac address because the license for the closed-source software requires it to run on that exact mac address. (Yes this is a pain, believe me I know and if I could change this I would.) I believe, based on the investigation into this problem, that this mac address is what will be assigned if it is the first docker container on a system. Unfortunately that means in practice that unless the address is passed specifically the software on the container won’t run.
This circleci config using docker-compose, (with this compose.yml and Dockerfile) passes but takes > 20 minutes. I suspect that this is because of a known bug in geckodriver. I have tried the workarounds for this bug, and I have also tried switching to chome for browser tests, but they did not make the docker-compose ci build run any faster. Advice welcome on this too, because if we could make the docker-compose solution fast it would solve the issue.
This is the latest config.yml I’ve tried. It says it starts mediaflux, but nothing is available on port 8888. I re-ran it with ssh enabled, made an ssh connection to the running ci build, and tested it from there too. Consistently, I can run the docker run command from my original post and it will work on my laptop but not on the ci build.
I wonder if the port 8888 issue is that I have to use a machine executor? I’m in the process of trying that now, but it seems suspicious to me that my build can happily run postgres on port 5432 and my tests can connect to that, so why is this container with port 8888 any different?
Your question about the mac address is a good one. I made another ssh connection to look at the mac addresses of the running containers, and sure enough that mac address has already been assigned, to the public.ecr.aws/eks-distro/kubernetes/pause:3.6 container. That’s not something I run myself, I guess it’s part of the circleci infrastructure? Is there a way to give it a different mac address, or any other workarounds you can think of?
circleci@5b5a565ea382:~$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
99215e911749 postgres:13.5-alpine "docker-entrypoint.s…" 28 seconds ago Up 27 seconds sleepy_swirles
792eee67962e cimg/ruby:3.2-browsers "/docker-entrypoint.…" 32 seconds ago Up 27 seconds laughing_greider
5b5a565ea382 public.ecr.aws/eks-distro/kubernetes/pause:3.6 "/pause" 32 seconds ago Up 27 seconds 0.0.0.0:54782->54782/tcp, :::54782->54782/tcp vigorous_gates
circleci@5b5a565ea382:~$ docker inspect sleepy_swirles |grep MacAddress
"MacAddress": "",
circleci@5b5a565ea382:~$ docker inspect laughing_greider |grep MacAddress
"MacAddress": "",
circleci@5b5a565ea382:~$ docker inspect vigorous_gates |grep MacAddress
"MacAddress": "02:42:ac:11:00:02",
"MacAddress": "02:42:ac:12:00:02",
"MacAddress": "02:42:ac:11:00:02",
Switching the environment and refactoring your config.yml may allow you to run an environment that does not load containers before your script executes. As you are currently using a docker based environment, trying a machine based environment will allow you more control.
The only real fix is going to be to work with the closed-source software provider as the choice of that mac address could not be much worse. It’s choice by them or whoever licenced the software at your end means that the people involved were not fully aware of how docker assigns mac addresses. By selecting what would be the first available assigned mac address you are at risk of the Docker environment changing between releases and things breaking. Things are likely to get even more complicated if you are starting to move to a Kubernetes based configuration.
Trying to find any Docker published statements on mac address assignment is not easy, but the values assigned are based on [02:24:AA:BB:CC:DD] with the first 2 values being the vendor ID and AA.BB.CC.DD is the assigned IPv4 address of the container.
SOLVED! We switched to a machine executor and started the vendor’s container before any other container and that works. We had to adjust our build to make it work on a machine executor but that wasn’t too bad. Build time now ~6 minutes, which we can totally live with.