Cache docker images that are pulled from a repository

docker
cache

#1

If a docker image is pulled from a repository, it should be a cached layer but this is not the case in Circle 2.0. For reference, this does work in Circle 1.0.


Why do tickets keep being locked?
#2

Hi Dave, can you share a link as an example of this happening on 2.0, and the expected behavior on 1.0?


#3

I don’t have time to setup an example repo and build, but here’s the steps:

  1. Pull previous image from Docker repo
  2. Build new image
  3. Push new image to Docker repo

On CircleCI 1.0, the new image will use the cached layers from previous image that was pulled from the repo but that is not the case in CircleCI 2.0 and I’ve tried several different things to make that work.

It’s probably also worth pointing out that I’m using the machine executor because building a Docker image in the docker executor is painful and requires a crazy amount of steps that I’d prefer to not have to mess with.


Still need to add mechanism to update existing cache key
Still need to add mechanism to update existing cache key
#4

I don’t have time to setup an example repo and build,

You can share a link to historical builds. Engineers at CircleCI can view customer build pages given a link.

It’s probably also worth pointing out that I’m using the machine executor because building a Docker image in the docker executor is painful and requires a crazy amount of steps that I’d prefer to not have to mess with.

For your use case, I suggest using the Docker executor, not the Machine executor. We have a whitelist-only feature, docker layer caching.
That’s specifically designed to provide layer caching for building docker images. To contact your support representative to request this, you can open a support ticket via our UI, or send an email to sayhi@circleci.com


#5

Here’s some recent test builds that I used to try and get the caching to work:
https://circleci.com/gh/numetricbi/numetric/6002
https://circleci.com/gh/numetricbi/numetric/6001

That docker layer caching is not guaranteed to always work so it’s a non-starter for us. A full build can take 45-60 minutes instead of 5-15 for when the cache is available, so unless the cache is guaranteed to be available then we just can’t use that feature.

Also, the boilerplate required to use the Docker executor to build Docker images is REALLY annoying.


#6

Thank you for sharing the links. I took a look at your build history and see what you mean about a 45 min build time. If you need guaranteed cache-hits for docker layer caching, I believe there isn’t an option for you at this time with either the Docker or Machine executor.

CircleCI 2.0 may have a feature after the closed beta that fits your project’s caching needs, but not now.


#7

So should I start looking into other CI platforms? We’ve been waiting over 6 months for a promised solution and it appears that just isn’t going to come ( Support Docker versions newer than v1.9.1 ).


#8

@dave-nm Thanks for your feedback on this.

We’re working on some documentation and metrics that will better explain the level of cache performance that can be expected from the Docker Layer Caching feature. We expect it to be reliable enough to be helpful in the vast majority of cases.

I’ll update here when we have more info to share.

I understand the frustration with the boilerplate needed to get it set up. We’ll be looking at ways to improve that too. That said, once set up, we’ve found it works well for people.

Thanks again for sharing your experiences with 2.0. Feedback during Beta is very helpful for us to make the product people need.


#9

From your comment, it sounds like you’re documenting the current sub-optimal caching and not doing anything to actual resolve it. Is that correct? To be honest, “reliable enough to be helpful in the vast majority of cases” sounds like marketing speak for “that feature doesn’t work all that well but we’re not going to do anything about it with the hope that people don’t leave”.

Basically, CircleCI 2.0 appears to have increased flexibility by using Docker but for the basic case where I just want to build Docker images and use Docker to run tests, it got worse from 1.0. People keep complaining about that and I’ve yet to see a real response recognizing that and stating if it will actually be fixed or if we just need to start looking elsewhere.


#10

Thanks again for the feedback. Sorry I’m not doing a good job of communicating our position here. I’m on the support team. I spoke to one of the lead 2.0 engineers before replying. We’re definitely not trying to cover anything up with marketing speak :slight_smile:

The current wording in the docs doesn’t set expectations very well. We should be phrasing guarantees in the form of “we expect n% cache misses a month” rather than “it may not work”. This is the data we’re gathering during the Beta.

So why might there be a cache miss? These are some of the scenarios we’re monitoring:

  • Cloud provider unexpectedly terminates a machine or has an outage.
  • A machine / Docker segfaults and needs to be recreated
  • Machines are recycled due to a Docker release or security issues (we haven’t had to do this yet, but it could be required every month or so)

This is why we can’t give 100% guarantees. In a complex hosted environment I think most people would agree that this is inevitable. The feature itself works well and we’re using customer feedback to make setting it up easier.

Something that might be helpful to know is If you are using docker for building images only and each image is tagged uniquely, you can set exclusive: false https://circleci.com/docs/2.0/docker-layer-caching/#exclusive so that concurrent builds can make use of the same cache.

I’m working on improving the documentation right now and our 2.0 engineers are gathering metrics which we’ll share when we have a good base line.

If you decide to give it a try, please contact cs@circleci.com with the name of your GitHub or Bitbucket org and we’ll get it enabled for you. Full access to Docker Layer Caching is free during the Beta. Your feedback will help us make the feature you need.


#11

That all makes sense but this sort of information has already been stated several times and the question that keeps going unanswered. So I repeat, “is anything going to be done to fix the issues?”

Basically, people had come up with 2 ways to make caching of Docker images work with builds in CircleCI 1.0 and one doesn’t work 2.0 and the other is slow and clunky. With the official position on Docker appearing to be go to 2.0, it feels like there should be an experience that isn’t worse than 1.0 before a recommendation like that is made. Basically, improving the documentation would be helpful but without improvements to the product that can only go so far.

On a side note, using unique tags for deployed builds can work but for dev and testing builds we can’t/don’t want to use unique tags so exclusive: false is a non-starter for us.


#13