Cannot cache docker builds on 1.9.1+

Build Environment
1

Hi there, we are evaluating using CircleCI for our flagship web product builds, but are having some issues caching docker builds. We use docker build’s build-args argument to pass an SSH key down to the container which means we can only use docker 1.9.1+.

However the solution proposed on the CircleCI doc fails on 1.9.1 because docker save gives the following error:
mkdir -p ~/docker; docker save $IMAGE_NAME > ~/docker/image.tar
Error response from daemon: chtimes /var/lib/docker/tmp/docker-export-185042988/47ed1778199d0fade4926082ad19a1891e3ed967c65cca90642842e158054342: invalid argument
It seems related to this issue: docker/issues/20324

On 1.10 no failure, but cache is not working because of this other docker issue: docker/issues/20380

I then went onto a second strategy which is to use docker pull and actually sounds better as recommended on: https://karthikv.net/articles/circleci-docker-flow/, however this will not work on 1.10 because of the content-addressable storage changes, see: docker/issues/20316

Last option is to use docker 1.9.1 with pull caching, but it does not seem to use the cache when building, we pull our image from a private registry, so could it be related to this issue: https://groups.google.com/forum/#!topic/docker-user/Wki9c9Qcu-E

I would appreciate any pointer as I spent quite a while trying to get this to work. Sorry that I had to cut github links, this support forum would otherwise give me this error: “Sorry, new users can only put 2 links in a post.”

1 Like
2

Sadly there is no great support for Docker caching at the moment. Please check out this doc for more information.

1 Like
3

I have followed the example of saving/loading from the doc, but I can only get this to work on Ubuntu 12.04 with Docker 1.8.

On Ubuntu 14.04 with Docker 1.9, it always seems to ignore the cache when building the image.

Are there any way of running Docker 1.9+ with local caching (i.e. no remote repository) at the moment?

Yaml for context:

machine:
  services:
    - docker

dependencies:
  cache_directories:
    - "~/docker"

  override:
    - docker info
    - if [[ -e ~/docker/image.tar ]]; then docker load -i ~/docker/image.tar; fi
    - docker build -t $CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME .
    - mkdir -p ~/docker; docker save $CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME > ~/docker/image.tar

...
1 Like
4

I’m following the same snippet on 14.04 environment and caching also doesn’t work. The image is loaded (I added a docker images command to my circle.yml) but the docker build command still builds from scratch.

5

Likewise. Ironically, when I ssh into the box and rebuild the image, it uses the cache (presumably from the build having just run, not the load).

Of course, we can’t remove images, so there’s no way for me to test it interactively in ssh.

This is most of my build time. I can’t imagine they’d prefer to download a gig every single build versus caching that same gig.

6

Also having this problem. I followed the trick outlined at https://circleci.com/docs/docker/#caching-docker-layers but it doesn’t work. The image is correctly cached in ~/docker and subsequent builds can ls the image but for some reason, docker doesn’t use the cached layers.

Here’s my circle.yml:

machine:
  services:
    - docker

dependencies:
  cache_directories:
    - "~/docker"
  override:
    - make docker-load
    - make build
    - make docker-save

database:
  override:
    - echo "Override CircleCI auto-detected databases"

test:
  # see https://discuss.circleci.com/t/docker-run-error-bind-address-already-in-use/294/10
  pre:
    - sudo service postgresql stop
    - $(aws ecr get-login --region us-west-1)
  override:
    - npm run docker:test

And bits of my Makefile:

# hack to get caching in circleci
docker-load:
	ls ~/docker || true
	[[ -e ~/docker/image.tar ]] && docker load -i ~/docker/image.tar || true
.PHONY: docker-save
docker-save:
	mkdir -p ~/docker
	docker save ${IMAGE} > ~/docker/image.tar
	ls ~/docker || true
.PHONY: docker-build
docker-build:
	docker build --pull \
		--rm=false \
		--build-arg NPM_TOKEN=${NPM_TOKEN} \
		-t ${IMAGE} .
	docker tag ${IMAGE} ${SHORT_NAME}:test
7

Having the same problem here. Image seems to be loading correctly but docker ignores the cache.

8

This needs to get its own post. Having this issue as well.