Cache CLI tools in /bin from pip install

pip
cache
python
cli

#1

Hi,

We’re currently caching our python packages as follows:

    - restore_cache:
          key: v1-python-requirements-{{ checksum "requirements.txt" }}
      - run:
          name: Install Python dependencies
          command: pip install -r requirements.txt
      - save_cache:
          key: v1-python-requirements-{{ checksum "requirements.txt" }}
          paths:
            - "~/.cache/pip"
            - "/usr/local/lib/python2.7/site-packages"
            - "/usr/local/lib/site-python"

Some python packages (like coverage and fabric) install CLI tools in /usr/local/bin, and by caching these directories but not /usr/local/bin, when we run pip install the packages don’t get reinstalled, and so the CLI tools don’t get reinstalled. As a result, we lose access to running fabric, etc. from the command line. What’s the best way to handle this? Should we be caching /usr/local/bin? (Seems a little extreme–plus, I wouldn’t know what to use as the cache key.


#2

Yes, you are correct, caching all of /usr/local/bin is going to end up in the weeds.

It looks like there is a way to install packages using pip to a specific directory, could you try using the --target option?
https://pip.pypa.io/en/latest/reference/pip_install/#cmdoption-t

From here, you will have to tell Python where to find these packages by modifying the PATHs.

Then finally, you can just cache this target directory and your packages should remain intact between builds.

Please let us know if you have any questions.

Best, Zak


#3

Issue though isn’t really the target of the builds, it’s that the build also generates the command-line utility, which gets put in /usr/local/bin.

Right now I’m getting around this by pre-installing the specific packages that require it (fabric, coverage, AWS CLI) before restoring the pip cache, but I’m sure others will have this issue as well.


#4

Hello @jdotjdot,

Glad that you found a working solution, could you share your circle.yml for those other users?


#5

I would definitely not call this a “good” solution by any means, but this is what we’re doing for now:

      # Python dependencies
      - run:
          name: Pre-install Python CLI packages
          # This is required since when we cache the output of pip, it doesn't also cache
          # /usr/local/bin, where the CLI gets installed
          command: pip install $(grep -i -e coverage -e fabric -e aws requirements.txt)
      - restore_cache:
          key: v1-python-requirements-{{ checksum "requirements.txt" }}
      - run:
          name: Install Python dependencies
          command: pip install -r requirements.txt
      - save_cache:
          key: v1-python-requirements-{{ checksum "requirements.txt" }}
          paths:
            - "~/.cache/pip"
            - "/usr/local/lib/python2.7/site-packages"
            - "/usr/local/lib/site-python"

#6

You can install coverage, fabric, and aws in your Dockerfile and only cache the requirements.txt installation.


#7