Python pip dependencies are not cached, reinstalled every time

When I run our build script locally I see lines like:

Collecting requests==2.4.3 (from>-r requirements.txt (line 4))
  Using cached requests-2.4.3-py2.py3-none-any.whl

With no ‘ install’ step for the majority of packages (the ones which come from pypi rather than github)

However on CircleCI it downloads the package every time, for each one:

Collecting requests==2.4.3 (from>-r requirements.txt (line 4))
  Downloading requests-2.4.3-py2.py3-none-any.whl (459kB)
    16% |█████▏                          | 73kB 23.0MB/s eta 0:00:01    100% |████████████████████████████████| 462kB 1.2MB/s 


  Running bdist_wheel for requests-oauthlib ... - done
  Stored in directory: /home/ubuntu/.cache/pip/wheels/73/f4/27/df0ae9b4e83413bd9bb24d42a4286eb77a227661c0e2de8fc2

Sadly we are using a bunch of packages like Pandas, numpy etc which take FOREVER to install

I have tried adding:

    - /home/ubuntu/.cache/pip

but it makes no difference

We’re currently evaluating CircleCI at my company and it’s unusable unless this can be resolved

Ahhh!! I found out why this happens:

The cache is not saved until after all the dependencies steps are completed. In fact it is the next stage after dependencies.

In my case I was trying to start a docker container in the dependencies > post stage, which was failing, so I had not yet successfully managed to save my pip cache and every build it was having to install them from scratch.

Frustrating, but hopefully I can iterate faster now that it got through that stage successfully once.

I wish this was documented. I nearly gave up entirely.

I’m glad you were able to figuring this out. Looks like we have this documented right in the configuration doc:

Is there another place you were looking that’s missing this information? We’re constantly trying to improve our documentation.

I was reading here: trying to understand why it wasn’t caching

I would have found it useful to have clearly stated that nothing is cached until all dependency steps have completed successfully

Thank you for the feedback. I’ve opened a GitHub Issue on the doc here. We’ll work on making this clearer.

1 Like