Building a Docker image in 2.0

build-image
docker

#1

So I signed up for Circle CI 2.0 because I was led to believe that there would be better Docker functionality. But I have no idea how to set up my configuration and the Docs don’t make it clear. I actually liked the config set up in Circle 1.0 but the Docker functionality was lacking.
What I basically want is this:
I have a project on github that whenever I push new changes I want Circle CI to grab that project and run the Docker file which would in turn build a Docker Image. Once that image is created I want to run that container and run commands against that container such as
"docker exec".

I have looked at the docs and haven’t found anything that fits my description.
It seems like I have to build a container for each service such as my host (Ubuntu) and my database (Mongo) which is confusing because I never had to do that in Circle 1.0. I would just run the Docker file and it would take care of that for me.


#2

I need to do something similar and it is pretty painful to get working in CircleCI 2.0. The worst part is that once I did figure out how to do it, there is no support for layer caching. In CircleCI 1.0, we pull the previously built images and then do the build so only the changed layers need to be rebuilt, but that doesn’t work with CircleCI 2.0 and you have to do a full rebuild of the Dockerfile every time.


#3

That is factually inaccurate. There is an entire page in our docs for it:

https://circleci.com/docs/2.0/docker-layer-caching/

You probably just need an image with Docker installed to do that. If you have any specific questions about 2.0, feel free to ask.

Just add the commands you need to build and exec that Docker image and see what happens.


#4

From my testing my statement is true. Build a Docker image, push it to Docker Hub, then in another build, pull that same image from Docker Hub and then build it. It should complete very quickly and use the cache for each layer but this does not happen.


#5

One thing to keep in mind about CircleCI is that you’re building on a cloud (fleet, distributed system, etc). Each platform has a number of host machines. Once you run a build on host machine A, your image is cached on that machine. For each next build, there’s a chance you could get host machine B, C, etc.

2.0 offers caching, but host machines don’t share a cache. As you run more builds, the fleet cache will get warmed up for your image.


#6

A post was split to a new topic: Docker “–cache-from” flag not working


#7

I think that we’re talking about different things. I’m talking about the cache not being populated when pulling from Docker Hub using the machine executorType. I think you’re talking about using the Remote Docker Environment.

Also, I understand that it’s a cloud environment, but to us, the chance that the cache might not be available is a non-starter. Builds randomly having to be done from scratch and taking 45-60 minutes instead of 5-12 just isn’t workable. The method of pulling the image from Docker Hub works with CircleCI 1.0 and it’s really surprising that it doesn’t with CircleCI 2.0.


#8

How big is your image…? Can you link me to where your builds are taking 40+ additional minutes just to pull the image? That doesn’t sound right to me.


#9

Pulling the cached images usually takes about 2 minutes and then rebuilding our application takes 5-15 minutes depending on what’s changed.

What makes it take 45-60 minutes is to build the entire image. We install and/or build libraries that we use (which can take 25-35 minutes), install used Python/Node.js modules (can take 5-10 minutes) and then build our application.


#10

This almost seems related to this bug https://github.com/docker/docker/issues/23684


#11

Could you elaborate?
I guess my specific question is what is a basic config that will allow me to build an image from a Docker file without having to build an image by hand first?


#12

https://circleci.com/docs/2.0/building-docker-images/#example


#13

use a primary image that already has Docker (recommended)
or install it during a build like we do here

Any chance you can suggest a primary image for this purpose? It’s a bit ridiculous to give only one example, which according to you isn’t reocmmended


#14

https://hub.docker.com/_/docker/

Except it has no languages installed, so I point back to my last comment: build your own image.


#15

Ah thanks, I didn’t realise that one can be used as docker client, as the documentation talks about it having a docker server.


#16