Caching file without checksum

cache

#1

I’m wanting to cache two Maxmind DB files that aren’t included inside my repository but are required for unit tests. The geoipupdate command below DLs two files into download-cache/maxmind2 which I then save into cache.

- restore_cache:
    keys:
      - maxmind-geolite2-v1
- run:
    name: 'Download GoeLite2 Maxmind DBs'
    command: |
      mkdir -p download-cache/maxmind2
      cp .circleci/Geoip.conf /etc/GeoIP.conf
      geoipupdate -v
- save_cache:
    key: maxmind-geolite2-v1
    paths:
      - "download-cache/maxmind2"

This method works, but means that I’ll effectively only have a single cache entry which isn’t ideal since it’ll quickly get out of date. I thought I could do something like key: maxmind-geolite2-v1-{{ checksum "download-cache/maxmind2/GeoLite2-City.mmdb" }} but since I can’t checksum this file at the start I have no way of restoring using this key.

Is there a nice way of caching (or any point at all?) this/achieve what I’m trying to do?


#2

If they’re unit tests, isn’t it better to download a copy and commit it to your repo? Make your tests generic enough that they can cope with a new file, which must be in exactly the same format.

I’d even go so far as to cut down the version you download, since once you’ve tested a few entries with different mathematical or geographical features, you’re done.


#3

I’m testing an API that sources data from these files so if data goes missing from them or changes I want to error out at this stage. The data that I do cherry pick however is generic enough that most changes should still cause everything to go okay! I don’t want to commit them since they’re pretty big binary files and distributing out of date ones isn’t ideal. I’m also not sure whether I’m allowed to given their liccence model.

I want to make sure I’ve got the latest version since thats what will be running in production, so if unit tests pass against whatever is in CI then production falls over since they’ve got latest, that isn’t ideal!


#4

This topic was automatically closed 41 days after the last reply. New replies are no longer allowed.