Docker image cache - Not pulling changes

docker
cache
circle.yml

#1

Hi, i’m using our custom cotdsa/docker-circlebuild:latest image, and had an issue with our private repo build not pulling the latest image i.e:

Build-agent version 0.0.1061-e121eb2 (2016-12-11T02:22:03+0000)
Starting container cotdsa/docker-circlebuild:latest
  using image cotdsa/docker-circlebuild@sha256:d0247904842a5e8bc08292692b5549f3c5fef413dca65cb1a4a76e3b2ceb550a

Instead of:

Build-agent version 0.0.1061-e121eb2 (2016-12-11T02:22:03+0000)
Starting container cotdsa/docker-circlebuild:epoch-1481774076
  image not cached, downloading cotdsa/docker-circlebuild:epoch-1481774076

I’m building using Docker Hub Automated Builds so I hacked around it by pushing out to a new branch/label, referencing FROM with the old image, which forced it to pull the image update. Can you please investigate?

Also, it’d be great if you supported sha256 pulls:

  • image: cotdsa/docker-circlebuild@sha256:9134ea8c5133bbc8335650f66d384f31023bce995449bad294b7ad3357e6c007

Thanks!


Docker images not cached between workflow jobs
#2

Can you try using an exact version number.

For more info please see the FAQ: Frequently Asked Questions

Please do not use circleci/build-image:latest!! Generally, it’s a bad idea to use the latest
tag of a Docker image in any case. Using a specific tag guarantees that
you are using the same version of the container image. For more
information on why we don’t recommend latest, check out these blog posts below:

http://container-solutions.com/docker-latest-confusion/


#3

Hi levlaz, thanks for the response.

I appreciate the sentiment of the FAQ, but you’re perpetuating the confusion and giving bad advice in relation to discouraging the use of latest.

latest is a tag just like any other, you can’t suggest not to use it and then turn around and recommend the ubuntu-14.04-XL-783-8f4b56f tag for instance. latest just happens to be the default when you don’t specify a tag.

It seems that when you mention exact version number, you actually intended on referring to the image digest, which is immutable (and supports the FAQs use of the word guarantee ) and which I suggested should be supported as per my sha256 comment.


#4

Hi acaire,

Our engineers on CircleCI 2.0 pretty unanimously agree to not use the latest tag. That tag might be reasonable under some circumstances and for some teams, but we recommend against using it and may eventually scan for it to bar its use. As a user on the 2.0 platform, you don’t have full control over the Docker engine that’s running your builds. That means you don’t have enough control to force Docker to update what latest refers to. This is by design because this is a shared environment.

What Lev meant by “exact version number” was an immutable tag associated with your Docker image. Your suggestion for a sha256 pulls seems reasonable, so I’ll pass that on.


#5

Hi Eric, thanks for the clarification - Now the holy war is out of the way, let’s move on with the outstanding issue still affecting the master tag as per your policy.

Build # 18779 (the build immediately before the build image was modified):

Starting container cotdsa/docker-circlebuild:master
  using image cotdsa/docker-circlebuild@sha256:129526d4b972abcce0104a04238292ae743a4fd198bbd3a25c4ddd91f9b75fb3

Build # 18780 (the build immediately after the build image was modified):

Starting container cotdsa/docker-circlebuild:master
  image not cached, downloading cotdsa/docker-circlebuild:master
master: Pulling from cotdsa/docker-circlebuild
b3e1c725a85f: Pulling fs layer
4daad8bdde31: Pulling fs layer
63fe8c0068a8: Pulling fs layer
4a70713c436f: Pulling fs layer
bd842a2105a8: Pulling fs layer
53ff8af3cf73: Pulling fs layer
74a913562754: Pulling fs layer
28fa543c565a: Pulling fs layer
53ff8af3cf73: Waiting
28fa543c565a: Waiting
74a913562754: Waiting
4a70713c436f: Waiting
bd842a2105a8: Waiting
63fe8c0068a8: Verifying Checksum
63fe8c0068a8: Download complete
4daad8bdde31: Verifying Checksum
4daad8bdde31: Download complete
bd842a2105a8: Verifying Checksum
bd842a2105a8: Download complete
4a70713c436f: Verifying Checksum
4a70713c436f: Download complete
53ff8af3cf73: Verifying Checksum
53ff8af3cf73: Download complete
b3e1c725a85f: Verifying Checksum
b3e1c725a85f: Download complete
28fa543c565a: Verifying Checksum
28fa543c565a: Download complete
b3e1c725a85f: Pull complete
4daad8bdde31: Pull complete
63fe8c0068a8: Pull complete
4a70713c436f: Pull complete
bd842a2105a8: Pull complete
53ff8af3cf73: Pull complete
74a913562754: Verifying Checksum
74a913562754: Download complete
74a913562754: Pull complete
28fa543c565a: Pull complete
Digest: sha256:1379170cd2340c429b28184160c87c6a5e14ea277c67a7379b6750084ec51df3
Status: Downloaded newer image for cotdsa/docker-circlebuild:master
  using image cotdsa/docker-circlebuild@sha256:1379170cd2340c429b28184160c87c6a5e14ea277c67a7379b6750084ec51df3

Build # 18782 (the second build after the build image was modified):

Starting container cotdsa/docker-circlebuild:master
  image not cached, downloading cotdsa/docker-circlebuild:master
master: Pulling from cotdsa/docker-circlebuild
b3e1c725a85f: Pulling fs layer
4daad8bdde31: Pulling fs layer
63fe8c0068a8: Pulling fs layer
4a70713c436f: Pulling fs layer
bd842a2105a8: Pulling fs layer
53ff8af3cf73: Pulling fs layer
74a913562754: Pulling fs layer
28fa543c565a: Pulling fs layer
74a913562754: Waiting
bd842a2105a8: Waiting
53ff8af3cf73: Waiting
28fa543c565a: Waiting
4a70713c436f: Waiting
4daad8bdde31: Verifying Checksum
4daad8bdde31: Download complete
63fe8c0068a8: Download complete
4a70713c436f: Verifying Checksum
4a70713c436f: Download complete
bd842a2105a8: Download complete
53ff8af3cf73: Verifying Checksum
53ff8af3cf73: Download complete
b3e1c725a85f: Download complete
28fa543c565a: Verifying Checksum
28fa543c565a: Download complete
74a913562754: Verifying Checksum
74a913562754: Download complete
b3e1c725a85f: Pull complete
4daad8bdde31: Pull complete
63fe8c0068a8: Pull complete
4a70713c436f: Pull complete
bd842a2105a8: Pull complete
53ff8af3cf73: Pull complete
74a913562754: Pull complete
28fa543c565a: Pull complete
Digest: sha256:1379170cd2340c429b28184160c87c6a5e14ea277c67a7379b6750084ec51df3
Status: Downloaded newer image for cotdsa/docker-circlebuild:master
  using image cotdsa/docker-circlebuild@sha256:1379170cd2340c429b28184160c87c6a5e14ea277c67a7379b6750084ec51df3

Build # 18783 (the third build after the build image was modified):

Starting container cotdsa/docker-circlebuild:master
  using image cotdsa/docker-circlebuild@sha256:129526d4b972abcce0104a04238292ae743a4fd198bbd3a25c4ddd91f9b75fb3

In summary:

Build 1: 129526d4b972abcce0104a04238292ae743a4fd198bbd3a25c4ddd91f9b75fb3 (old)
Build 2: 1379170cd2340c429b28184160c87c6a5e14ea277c67a7379b6750084ec51df3 (new)
Build 3: 1379170cd2340c429b28184160c87c6a5e14ea277c67a7379b6750084ec51df3 (new)
Build 4: 129526d4b972abcce0104a04238292ae743a4fd198bbd3a25c4ddd91f9b75fb3 (old)


#6

Hi Ash,

So you raise an interesting point. We can’t simply suggest avoiding latest because it’s mutable–we have to recommend avoiding all mutable tags, as your master tag demonstrates. We haven’t mentioned this at all in our docs or onboarding, and I think you’re on to something.

Speaking of which, I wanted to verify that SHA based pulling is not a thing before making it a ticket internally. But, I was able to get a successful pull with the following:

version: 2
executorType: docker
containerInfo:
  - image: cotdsa/docker-circlebuild@sha256:129526d4b972abcce0104a04238292ae743a4fd198bbd3a25c4ddd91f9b75fb3

So it looks like you can pull images by SHA.

To explain what happened in the series of builds you ran, different physical machines ran your builds, at least for the last 2. One of them had an outdated cache of the “master” tag. We don’t synchronize the Docker cache across our fleet, nor do we plan on doing this.


Node:latest npm fails
#7

Ahh you’re right Eric, SHA256 images are working fine for me now, I must’ve inadvertently mashed the commits when I was trying to test the deployment stage at the same time. Thanks for your help! :slight_smile:


#8