Is it possible to preserve cache entries for longer than 15 days?

Build Environment Caching Dependencies
1

Hi there,

My project makes use of a number of large dependencies via Vcpkg. After trying a number of options I’ve found that it takes about 1.5 hours on a Windows XLarge machine to build the dependencies which drains quite a large chunk of credits.

To avoid paying the cost repeatedly I’m caching the Vcpkg binary archive directory (which takes ~900MB). This all works well, but I’m looking at the maximum cache lifetime in the usage settings screen and as far as I can see the max value I can set for cache lifetime is 15 days. Is this correct? If the limit is indeed 15 days, is that from the moment that the cache is created, or from when an attempt was last made to write the cache? I don’t anticipate the Vcpkg dependencies will change that often so it seems a bit harsh to have to endure the cost (in both time and credits) of rebuilding them from scratch every 15 days. :stuck_out_tongue:

Any help or advice here would be much appreciated.

Regards

Alan

2

From all the docs I have read the indication is that the life of a cached object is calculated from its creation time/date, rather than its last access time/date.

As you seem to need to allocate a lot of processing time to generate a fairly small data set (900MB) have you considered just compressing the file set and placing it into your repo as part of your workflow so that it is available for future builds?

3

Thanks for the reply.

The built dependencies are ~900MB when compressed (that figure comes from what the caching step said it uploaded). But yeah I think you’re right in that it looks like I’m going to have to build some kind of extra caching mechanism myself in order to extend the 15 day limit.

Options I’m considering (in no particular order)

  • Manually build and check deps into repo (as you suggest)
  • Add in scripts to cache deps into an S3 bucket and to check there if not found in the CircleCI cache
  • Periodically make a non functional change to the vcpkg.json file

That last option, which I’m just thinking of and through now, would change the hash of the cache key, but allow the existing cache data to be loaded in and then resaved (I use the standard approach for restoring the cache shown below)

- restore_cache:
          name: Restore vcpkg Cache
          keys: 
            - vcpkg-cache-{{ checksum "vcpkg.json" }}-{{ checksum "vcpkg-configuration.json" }}
            - vcpkg-cache-

Actually, thinking about this further, I think that I could get the same effect by not touching vcpkg.json at all, but instead extending the key to be of the form

vcpkg-cache-{{ checksum "vcpkg.json" }}-{{ checksum "vcpkg-configuration.json" }}-{{ refresh_date }}

where refresh_date is a fixed date in the last 10 days (or some other time shorter than the cache lifetime limit). Now the question is just how would I calculate that date and get it injected into the hash key? Reading through the docs (Caching Dependencies - CircleCI) it looks like you can inject environment variables into cache keys so this feels doable. :slight_smile: I’ll give it a go.

Anyway, hopefully the above makes sense, and thanks again for the input.

Regards

Alan

1 Like
4

Ok, so that seems to work.

To clarify, what I’ve done is generated a file containing a ‘CACHE_DAY’ number early on in the build job

      # Puts a number that will increment by 1 every 10 days into the file CACHE_DAY
      # Can be used to refresh cache entries periodically before they're purged by
      # CircleCI's max cache entry age of 15 days
      - run:
          name: Set cache date
          command: echo $(( (($(date +%s) / 86400)/10)*10 )) > CACHE_DAY

Then I can use the checksum of this file in my cache key names

      - restore_cache:
          name: Restore vcpkg Cache
          keys: 
            - vcpkg-cache-{{ checksum "vcpkg.json" }}-{{ checksum "vcpkg-configuration.json" }}-{{ checksum "CACHE_DAY" }}
            - vcpkg-cache-

      ...

      - save_cache:
          name: Save vcpkg Cache
          key: vcpkg-cache-{{ checksum "vcpkg.json" }}-{{ checksum "vcpkg-configuration.json" }}-{{ checksum "CACHE_DAY" }}
          paths:
            - ...

That should refresh the cache entry every 10 days or so providing that I’m checking in changes on a regular basis. I think that I can also setup a periodic trigger build to keep the cache entry around during periods of less development.

5

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.