"No such file or directory" error for cached dependency; fine until yesterday

I am running a Flask app that needs to load a ~15 mb dataset to do geocoding. This has worked well for ages – specifying in config.yml to run a shell script that downloads it to a vendor directory, and then caches it with save_cache. As of yesterday, this stopped working, throwing the error:

IOError: [Errno 2] No such file or directory: 'vendor/GeoLiteCity.dat'
Exited with code 1

This was working fine a month ago, then stopped working yesterday, with no substantive changes to the code, and no changes to config.yml. The URL for curl works fine (at least when I test on my dev machine), and the tests all still work when I run them locally, so something is going wrong with the download, caching, or the coordination of paths. Did anything change in the way that config.yml / paths get interpreted? Or is it possible that this is a downstream error generated by another issue?

Here’s my config.yml. Test.py loads the Flask app and then hits all of the endpoints with requests:

version: 2
jobs:
  build:
    docker:
      - image: circleci/python:2.7-jessie-browsers
    steps:
      - checkout      
      - restore_cache:
          key: deps1-{{ .Branch }}-{{ checksum "requirements.txt" }}
      - restore_cache:
         keys:
           - vendor
      - run:
          name: Install Python deps in a venv
          command: |
            python -m virtualenv venv  
            source venv/bin/activate
            pip install -r requirements.txt            
      - run: 
          name: Get the PEM
          command: echo -e $APNS_PEM_CONTENTS > pushcert.pem
      - run: 
          name: Get the Firebase Token
          command: echo $FIREBASE_JSON > $FIREBASE_AUTH_TOKEN_PATH
      - run: 
          name: change permissions on geoLite download script
          command: chmod +x getGeoLite.sh
      - run:
          name: download GeoLite geocoding database
          command: ./getGeoLite.sh      
      - save_cache:
          key: vendor
          paths:
            - vendor
      - save_cache:
          key: deps1-{{ .Branch }}-{{ checksum "requirements.txt" }}
          paths:
            - "venv"      
      - run:
          command: |
            source venv/bin/activate
            python test.py      

What is the result of the cache restore in the build GUI?

What happens in getGeoLite.sh - does it skip of the dat file exists, and download if it does not? What was the result of this script in your broken build?

In getGeoLite.sh, the curl command runs only if the vendor/ directory doesn’t exist. The data file is not expected to change for the duration of this project.

The cache restores have all been successful, but I just found that the size of the cache restore varies between the unsuccessful runs and the last successful run.

Unsuccessful runs:

Found a cache from build 164 at vendor
Size: 1.4 MB
Cached paths:
  * /home/circleci/project/vendor

Downloading cache archive...
Validating cache...

Unarchiving cache...

vs. the last successful run:

Found a cache from build 95 at vendor
Size: 12 MB
Cached paths:
  * /home/circleci/project/vendor

Downloading cache archive...
Validating cache...

Unarchiving cache...

So I tried rebuilding without the cache.

Warning: skipping this step: disabled in configuration

But then the same error, that the data file is missing! So I tried Rebuild with SSH. This shows that the file is indeed missing, plain and simple. So for some reason that data file wasn’t downloaded (trouble reaching the server where it is hosted), and then the cache was updated. Changing the conditional structure of the download script (download if the specific file is missing) resolved the issue.

The takeaways for me with this are:

  1. Rebuild without cache and Rebuild with SSH are my friends
  2. I need to be careful / more granular with conditionals in bash scripts that add additional dependencies to be robust to network failures
1 Like

Great! I think CircleCI 2.1 is in pre-release availability, and that has conditional statements. That might be useful to avoid a cache save in the case of a network failure (but retries+failure is good as well, of course).

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.