CircleCi caching of python dependencies not working

Build Environment Caching Dependencies
, ,
1

Hey, I am trying to implement caching to speed up the time for testing of my app.
I have gotten CircleCi to save and restore the cache successfully.

However, it does still reinstall all of the dependencies each time, is this a bug or am I doing something wrong in my code?

Please see the message from CircleCi and my code below

Restoring response from CircleCI:

Found a cache from build 13 at deps1-CircleCi-caching-integration-x442no1rWzQq7IjN88tbxpE0VT7AchlwkRloCNuuS0k=
Size: 32 B
Cached paths:
  * /home/circleci/project/venv

Downloading cache archive...
Validating cache...

Unarchiving cache...

Saving response from CircleCI:

Creating cache archive...
Uploading cache archive...
Stored Cache to deps1-CircleCi-caching-integration-x442no1rWzQq7IjN88tbxpE0VT7AchlwkRloCNuuS0k=
  * /home/circleci/project/venv

However, it does still reinstall dependencies

#!/bin/bash -eo pipefail
echo 'export PATH=~$PATH:~/.local/bin' >> $BASH_ENV && source $BASH_ENV
pip install --user -r requirements.txt

     |████████████████████████████████| 748.9MB 26kB/s s eta 0:00:0101
Collecting torchvision==0.4.2
  Downloading https://files.pythonhosted.org/packages/74/ee/d872c12ea508f9cca4cf0d7b91fc5a5e476cec77628a66795ec5a585a67f/torchvision-0.4.2-cp37-cp37m-manylinux1_x86_64.whl (10.2MB)
     |████████████████████████████████| 10.2MB 65.6MB/s eta 0:00:011
Collecting pylint==2.4.3
  Downloading https://files.pythonhosted.org/packages/ea/f1/758de486e46ea2b8717992704b0fdd968b7cbc2bc790b976fae4a35a212c/pylint-2.4.3-py3-none-any.whl (302kB)
     |████████████████████████████████| 307kB 84.3MB/s eta 0:00:01
Collecting pytest==5.2.2
  Downloading https://files.pythonhosted.org/packages/93/16/f6dec5178f5f4141e80dfc4812a9aba88f5f29ca881f174ab1851181d016/pytest-5.2.2-py3-none-any.whl (227kB)
     |████████████████████████████████| 235kB 77.1MB/s eta 0:00:01
Collecting pytest-html==2.0.0
  Downloading https://files.pythonhosted.org/packages/f7/05/7cc56180a9c4b00b4055ffdad3763884000082a471c2e7251e97ea055932/pytest_html-2.0.0-py2.py3-none-any.whl
Collecting pillow==6.2.0
  Downloading https://files.pythonhosted.org/packages/34/b7/fb24ad352b747340ec1975c84c30cca37bb93a0079e037d1f0930afd65ad/Pillow-6.2.0-cp37-cp37m-manylinux1_x86_64.whl (2.1MB)
     |████████████████████████████████| 2.1MB 37.4MB/s eta 0:00:01
Collecting pandas==1.0.1
  Downloading https://files.pythonhosted.org/packages/61/af/ceb7523e86753d5643ed00e8c17a42bdcfe819782c3449d9bbbf5d01867a/pandas-1.0.1-cp37-cp37m-manylinux1_x86_64.whl (10.1MB)
     |████████████████████████████████| 10.1MB 31.1MB/s eta 0:00:011
Requirement already satisfied: six>=1.11.0 in /usr/local/lib/python3.7/site-packages (from jsonschema==3.0.2->-r requirements.txt (line 2)) (1.12.0)
Requirement already satisfied: pyrsistent>=0.14.0 in /usr/local/lib/python3.7/site-packages (from jsonschema==3.0.2->-r requirements.txt (line 2)) (0.14.11)
Requirement already satisfied: setuptools in /usr/local/lib/python3.7/site-packages (from jsonschema==3.0.2->-r requirements.txt (line 2)) (41.4.0)
Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.7/site-packages (from jsonschema==3.0.2->-r requirements.txt (line 2)) (19.3.0)
Collecting python-dateutil>=2.1
  Downloading https://files.pythonhosted.org/packages/d4/70/d60450c3dd48ef87586924207ae8907090de0b306af2bce5d134d78615cb/python_dateutil-2.8.1-py2.py3-none-any.whl (227kB)
     |████████████████████████████████| 235kB 78.4MB/s eta 0:00:01
Collecting cycler>=0.10
  Downloading https://files.pythonhosted.org/packages/f7/d2/e07d3ebb2bd7af696440ce7e754c59dd546ffe1bbe732c8ab68b9c834e61/cycler-0.10.0-py2.py3-none-any.whl
Collecting kiwisolver>=1.0.1
  Downloading https://files.pythonhosted.org/packages/93/f8/518fb0bb89860eea6ff1b96483fbd9236d5ee991485d0f3eceff1770f654/kiwisolver-1.1.0-cp37-cp37m-manylinux1_x86_64.whl (90kB)
     |████████████████████████████████| 92kB 25.7MB/s eta 0:00:01
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/site-packages (from matplotlib==3.1.1->-r requirements.txt (line 3)) (2.4.2)
Collecting astroid<2.4,>=2.3.0
  Downloading https://files.pythonhosted.org/packages/ad/ae/86734823047962e7b8c8529186a1ac4a7ca19aaf1aa0c7713c022ef593fd/astroid-2.3.3-py3-none-any.whl (205kB)
     |████████████████████████████████| 215kB 46.1MB/s eta 0:00:01
Collecting mccabe<0.7,>=0.6
  Downloading https://files.pythonhosted.org/packages/87/89/479dc97e18549e21354893e4ee4ef36db1d237534982482c3681ee6e7b57/mccabe-0.6.1-py2.py3-none-any.whl
Collecting isort<5,>=4.2.5
  Downloading https://files.pythonhosted.org/packages/e5/b0/c121fd1fa3419ea9bfd55c7f9c4fedfec5143208d8c7ad3ce3db6c623c21/isort-4.3.21-py2.py3-none-any.whl (42kB)
     |████████████████████████████████| 51kB 16.2MB/s eta 0:00:01
Collecting atomicwrites>=1.0
  Downloading https://files.pythonhosted.org/packages/52/90/6155aa926f43f2b2a22b01be7241be3bfd1ceaf7d0b3267213e8127d41f4/atomicwrites-1.3.0-py2.py3-none-any.whl
Collecting wcwidth
  Downloading https://files.pythonhosted.org/packages/58/b4/4850a0ccc6f567cc0ebe7060d20ffd4258b8210efadc259da62dc6ed9c65/wcwidth-0.1.8-py2.py3-none-any.whl
Collecting py>=1.5.0
  Downloading https://files.pythonhosted.org/packages/99/8d/21e1767c009211a62a8e3067280bfce76e89c9f876180308515942304d2d/py-1.8.1-py2.py3-none-any.whl (83kB)
     |████████████████████████████████| 92kB 42.4MB/s eta 0:00:01
Collecting pluggy<1.0,>=0.12
  Downloading https://files.pythonhosted.org/packages/a0/28/85c7aa31b80d150b772fbe4a229487bc6644da9ccb7e427dd8cc60cb8a62/pluggy-0.13.1-py2.py3-none-any.whl
Requirement already satisfied: more-itertools>=4.0.0 in /usr/local/lib/python3.7/site-packages (from pytest==5.2.2->-r requirements.txt (line 9)) (7.2.0)
Collecting packaging
  Downloading https://files.pythonhosted.org/packages/62/0a/34641d2bf5c917c96db0ded85ae4da25b6cd922d6b794648d4e7e07c88e5/packaging-20.3-py2.py3-none-any.whl
Requirement already satisfied: importlib-metadata>=0.12; python_version < "3.8" in /usr/local/lib/python3.7/site-packages (from pytest==5.2.2->-r requirements.txt (line 9)) (0.23)
Collecting pytest-metadata
  Downloading https://files.pythonhosted.org/packages/ce/8f/d0542e1aa0e23d902ce6acce2790736473da94453a36bdc7829f25734199/pytest_metadata-1.8.0-py2.py3-none-any.whl
Collecting pytz>=2017.2
  Downloading https://files.pythonhosted.org/packages/e7/f9/f0b53f88060247251bf481fa6ea62cd0d25bf1b11a87888e53ce5b7c8ad2/pytz-2019.3-py2.py3-none-any.whl (509kB)
     |████████████████████████████████| 512kB 60.2MB/s eta 0:00:01
Collecting lazy-object-proxy==1.4.*
  Downloading https://files.pythonhosted.org/packages/23/f8/69df5a663b59512eb9f9b84e7f203c48c7a933e460316e9ebf4db2871ae0/lazy_object_proxy-1.4.3-cp37-cp37m-manylinux1_x86_64.whl (56kB)
     |████████████████████████████████| 61kB 17.2MB/s eta 0:00:01
Collecting wrapt==1.11.*
  Downloading https://files.pythonhosted.org/packages/23/84/323c2415280bc4fc880ac5050dddfb3c8062c2552b34c2e512eb4aa68f79/wrapt-1.11.2.tar.gz
Collecting typed-ast<1.5,>=1.4.0; implementation_name == "cpython" and python_version < "3.8"
  Downloading https://files.pythonhosted.org/packages/5d/10/0c1e8aa723a2b0c4032e048d8e511df82c8a1262f0e1df5e4c54eb2613e9/typed_ast-1.4.1-cp37-cp37m-manylinux1_x86_64.whl (737kB)
     |████████████████████████████████| 747kB 73.1MB/s eta 0:00:01
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/site-packages (from importlib-metadata>=0.12; python_version < "3.8"->pytest==5.2.2->-r requirements.txt (line 9)) (0.6.0)
Building wheels for collected packages: wrapt
  Building wheel for wrapt (setup.py) ... - \ done
  Created wheel for wrapt: filename=wrapt-1.11.2-cp37-cp37m-linux_x86_64.whl size=76137 sha256=199cb7b7348e923f011112773c7e2893ee298b0400ff7520b2e2d4a590fb161d
  Stored in directory: /home/circleci/.cache/pip/wheels/d7/de/2e/efa132238792efb6459a96e85916ef8597fcb3d2ae51590dfd
Successfully built wrapt
ERROR: torchvision 0.4.2 has requirement torch==1.3.1, but you'll have torch 1.2.0 which is incompatible.
Installing collected packages: json5, jsonschema, python-dateutil, numpy, cycler, kiwisolver, matplotlib, opencv-contrib-python, torch, pillow, torchvision, lazy-object-proxy, wrapt, typed-ast, astroid, mccabe, isort, pylint, atomicwrites, wcwidth, py, pluggy, packaging, pytest, pytest-metadata, pytest-html, pytz, pandas
Successfully installed astroid-2.3.3 atomicwrites-1.3.0 cycler-0.10.0 isort-4.3.21 json5-0.8.5 jsonschema-3.0.2 kiwisolver-1.1.0 lazy-object-proxy-1.4.3 matplotlib-3.1.1 mccabe-0.6.1 numpy-1.16.5 opencv-contrib-python-4.1.2.30 packaging-20.3 pandas-1.0.1 pillow-6.2.0 pluggy-0.13.1 py-1.8.1 pylint-2.4.3 pytest-5.2.2 pytest-html-2.0.0 pytest-metadata-1.8.0 python-dateutil-2.8.1 pytz-2019.3 torch-1.2.0 torchvision-0.4.2 typed-ast-1.4.1 wcwidth-0.1.8 wrapt-1.11.2
WARNING: You are using pip version 19.3; however, version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
CircleCI received exit code 0

And my code:

version: 2.1
jobs:
  run_tests:
    docker:
      - image: circleci/python:3.7.4
    steps:
      - checkout
      - restore_cache: # **restores saved dependency cache if the Branch key template or requirements.txt files have not changed since the previous run**
          key: deps1-{{ .Branch }}-{{ checksum "requirements.txt" }}
      - run: # install and activate virtual environment with pip
          name: Install Python Dependencies
          command: |
            echo 'export PATH=~$PATH:~/.local/bin' >> $BASH_ENV && source $BASH_ENV
            pip install --user -r requirements.txt
      - save_cache: # ** special step to save dependency cache **
          key: deps1-{{ .Branch }}-{{ checksum "requirements.txt" }}
          paths:
            - "venv"
      - run:
          name: Run Unit Tests
          command: |
            pytest --junitxml=test-reports/junit.xml --html=test-reports/pytest_report.html --self-contained-html
      - store_test_results: # saves test resoults
          path: test-reports
      - store_artifacts:
          path: test-reports
workflows:
  build_test:
    jobs:
      - run_tests
2

@RasmussenChristian I’m seeing the same issue. Were you able to resolve it?

3

Pip typically installs to ~/.local/bin and ~/.local/lib and I cannot see that being cached.
That being said when I’ve added

      - save_cache:
          key: deps1-{{ .Branch }}-{{ checksum "requirements.txt" }}
          paths: ~/.local

I cannot see it saving the cache

4

We found these things worked for us:

  1. Use & tell CI to activate virtual environments where project dependencies will live.
- run:
    command: |
         virtualenv .venv;
         echo "source .venv/bin/activate" >> $BASH_ENV;
  1. Save content of your virtual environment & pip caches to avoid re-building wheels.
- save_cache:
    key: pydeps-v01__{{ checksum "requirements.txt" }}
    paths:
        - ".venv"
        - "~/.cache/pip"    # (pip cache dir)