Caching Docker image with Docker Hub

Helpful Resources Tips, Tricks, and Hacks
1

I wrote original content here: https://github.com/kimh/circleci-build-recipies/tree/better-docker-cache

The following circle.yml demonstrates how we can use Docker Hub for caching.

machine:
  services:
    - docker

dependencies:
  pre:
    - docker login -e $DOCKER_EMAIL -u $DOCKER_USER -p $DOCKER_PASS

  override:
    - docker pull kimh/build-image:latest || true

test:
  override:
    - docker build -t kimh/build-image:latest .
    - docker push kimh/build-image:latest

Here is Dockerfile.

FROM phusion/passenger-ruby21:latest

RUN apt-get update
RUN apt-get install sl

WORKDIR /tmp
ADD Gemfile /tmp/
ADD Gemfile.lock /tmp/
RUN bundle install

What we are doing in circle.yml is very simple:

  • Pull image from Docker Hub that was pushed from previous builds
  • Build image from Dockerfile
  • Push the image back to Docker Hub for the subsequent builds

In short, we are using Docker Hub as our image cache store.

To see how caching works, you can compare this and this builds.

In the first build, docker build took about 3 mins and docker push took 2 mins, but both commands finish immediately in the second build thanks to cache.

2

The two recipies you link to are the same. Do you mind sharing the correct ones?

3

@steevel Sorry! I just fixed the link.

4

How are we handling the fact that now you are pushing a latest image in your registry? You might use that registry also for CD, how do you make sure you are not going to deploy a potentially broken image? Also, how do you handle the fact that now multiple branches share the same cache? This could lead to invalidating the cache more often at best, or to undeterministic results in your builds at worst

5

I am guessing that for private builds we would also want to have private Docker repos which cost additional money.

6

This is how CodeShip handles the problem (using a Docker registry and then tagging per branch): https://codeship.com/documentation/docker/caching/

7

@itajaja Sorry, my approach is a bit rough and doesn’t guarantee what you’ve mentioned. And yes, as @agrothberg mentioned, I think we should be able to do similar approach that CodeShip does.

However, I believe the approach doesn’t work in Docker 1.10. Please see: Docker 1.10.0 is available (Beta)

1 Like
8

I made a shell script that helps the caching issue. Workaround Docker cache issue with docker-cache-shim

It’s working for my workflow and hopefully it does for others.