Still need to add mechanism to update existing cache key

cache

#1

This original post wasn’t actually solved and is now locked ( Add Mechanism to Update Existing Cache Key ).

Adding epoch to create a new cache key doesn’t work because it will then build up an insane number of caches when building fixed tags like master and dev.


Why do tickets keep being locked?
#2

Could you explain more why this is a problem for your builds? This is caching strategy that CircleCI 1.0 has used for quite some time.


#3

Our docker images can be several GBs when dumped to the cache so if it adds a new chunk of GB files every time there’s a build on our dev/master/etc branch then that’s going to make the restoring of the cache be very slow.

It’s probably also worth noting, that we found that it was significantly faster in CircleCI 1.0 to pull the images from an external repo than from the cache in CircleCI. We are doing the steps described at Cache docker images that are pulled from a repository and it’s always struck us as odd that downloading over the internet is faster than using a local cache, but maybe this build up of caches explains that.


#4

it’s always struck us as odd that downloading over the internet is faster than using a local cache

CircleCI builds aren’t running on a single bare metal machine: they run on a fleet of AWS machines. You may be thinking of the cache as a local disk, but the cache is stored in S3. S3, while in the same data center as the build machines, transfers slower than from DockerHub. Writing additional, separate caches will not slow down cache fetching.

I see from your other post that you’re using the machine executor. Please see my reply in that thread regarding docker layer caching.


#5

That makes sense, but it still seems odd that downloading the same data from a completely different location is faster than downloading from the same server framer (AWS).


#6

I had the same initial thoughts as well, but DockerHub is most likely running on AWS the same EC2 region.


#7

So you’re saying that we should move to building in Docker Hub because CircleCI just can’t do caching more efficiently than user’s manually caching on an external service? :wink:


#8