Cache saving on multiple containers


#1

Hi all,

According an older thread, when running a parallel build, only the cache from the first container is collected. Is that still supposed to be true in Circle2.0 world? We’re seeing in our builds that the container that finishes first gets to save the cache, and further containers will not attempt to save (since cache already exists). Unfortunately this means any artifact we want to cache from deploy will never get saved, since the deployment always waits for other containers to finish first. Is there a workaround for this issue?

cheers,
Geoff

P.S. the thread mentioned: Cache save/restore algorithm


#2

You either need to save those specific artifacts under a different cache step & name, or you can store those artifacts with store_artifacts and pull them back down with the API.


#3

Hey Ryan,

Thanks for the reply. I’m afraid the “different cache step” approach won’t work. Our build steps are essentially:

  • checkout
  • restore_cache
  • build
  • deploy
  • save_cache

If I understand correctly, you’re suggesting we do the following:

  • checkout
  • restore_cache (build)
  • build
  • save_cache (build)
  • restore_cache (deploy)
  • deploy
  • save_cache (deploy)

The problem is that on containers that are not the primary container, the deploy steps are skipped, but the save_cache steps are not. So we end up always saving an empty deploy cache. Am I missing something obvious here? Feel like this should be a pretty common use-case.

Geoff


#4

Hmm that’s a good point. I definitely understand how that’s frustrating for you.

The current workaround would be to halt on the non-primary image.

- run: if [ $CIRCLE_NODE_INDEX != 0 ] ; then circleci step halt ; fi

Putting that between your build and deploy steps should solve it. :slight_smile:


#5

Ahhh, that would do. I’ve been looking for something like that actually. Thanks!


#6

This topic was automatically closed 41 days after the last reply. New replies are no longer allowed.