Python pip dependencies are not cached, reinstalled every time

Feedback & Bug Reports
, ,
1

When I run our build script locally I see lines like:

Collecting requests==2.4.3 (from contentful.py==0.9.3->-r requirements.txt (line 4))
  Using cached requests-2.4.3-py2.py3-none-any.whl

With no ‘setup.py install’ step for the majority of packages (the ones which come from pypi rather than github)

However on CircleCI it downloads the package every time, for each one:

Collecting requests==2.4.3 (from contentful.py==0.9.3->-r requirements.txt (line 4))
  Downloading requests-2.4.3-py2.py3-none-any.whl (459kB)
    16% |█████▏                          | 73kB 23.0MB/s eta 0:00:01    100% |████████████████████████████████| 462kB 1.2MB/s 

...

  Running setup.py bdist_wheel for requests-oauthlib ... - done
  Stored in directory: /home/ubuntu/.cache/pip/wheels/73/f4/27/df0ae9b4e83413bd9bb24d42a4286eb77a227661c0e2de8fc2

Sadly we are using a bunch of packages like Pandas, numpy etc which take FOREVER to install

I have tried adding:

dependencies:
  cache_directories:
    - /home/ubuntu/.cache/pip

but it makes no difference

We’re currently evaluating CircleCI at my company and it’s unusable unless this can be resolved

2

Ahhh!! I found out why this happens:

The cache is not saved until after all the dependencies steps are completed. In fact it is the next stage after dependencies.

In my case I was trying to start a docker container in the dependencies > post stage, which was failing, so I had not yet successfully managed to save my pip cache and every build it was having to install them from scratch.

Frustrating, but hopefully I can iterate faster now that it got through that stage successfully once.

I wish this was documented. I nearly gave up entirely.

3

I’m glad you were able to figuring this out. Looks like we have this documented right in the configuration doc: https://circleci.com/docs/configuration/#cache-directories

Is there another place you were looking that’s missing this information? We’re constantly trying to improve our documentation.

4

I was reading here: https://circleci.com/docs/how-cache-works/ trying to understand why it wasn’t caching

I would have found it useful to have clearly stated that nothing is cached until all dependency steps have completed successfully

5

Thank you for the feedback. I’ve opened a GitHub Issue on the doc here. We’ll work on making this clearer.

1 Like