Cache save/restore algorithm

Build Environment
1

Hello,

there’re some unclear paths in cache saving algorithm. Which cache is saved and how during parallel builds? The thing is there’re several parallel processes which first restore cache and then should save it somehow.

I’ve managed to find in the blog http://blog.circleci.com/our-first-postmortem/.

We also serialized cache restoration: in a parallel build, we had been piping the tarball contents into all containers in parallel. Switching this to one container at a time vastly improved things in production. It was still pretty broken, and much much slower, but we had a bit of much-needed breathing space.

This well clarifies restoration process, but the saving process is that what I don’t understand. So I’d like some help which could save me from debugging:

  1. Which cache is picked up during the save process?
  • first built
  • last built
  • caches of all the builds
  1. Is cache distinguished by $CIRCLE_NODE_INDEX?

Thank you,
Denis

2

The cache is collected from the first container only (container with the index 0). On the next build, the previously collected cache from the node 0 is distributed across all the machines in the parallel build. Does that make sense?

3

Thank you Alexey, it makes sense!

I’m going to find how to work this around, since in our build scenario cache is container specific(
We trigger different OS distro builds for wheezy, jessie, centos7 etc, and in each case cache differs…

That’s disappointing, but I do understand why there’s such implementation in circle ci, it’s valid for more than 80% use cases.

4

Sorry about not supporting your use case better. The only think I can offer you here is to capture and store the cache yourself, for example on S3.