Latest circleci/python:3.7 container creates corrupt pipenv caches

Ecosystem CircleCI Images
1

Hello there,

We follow the the recommendations for using a pipenv cache with python projects nearly exactly to the letter, with the exception that we use pipenv sync instead of pipenv install.

This used to work like a charm, but since a few days caches seem to be corrupted, and we have to invalidate the cache each run. The error we’re getting this that pip cannot be found.

 in <module>', '    from pip._internal.cli.main import main', "ModuleNotFoundError: No module named pip

This is caused by pip somehow being symlinked to a location outside of the project directory. This location doesn’t get cached, and hence pipenv will fail when run from a restored cache. The symlink that pip points to:

$ ls -alhtr .venv/lib/python3.7/site-packages/pip
lrwxrwxrwx 1 circleci circleci 106 Feb 11 15:01 .venv/lib/python3.7/site-packages/pip -> /home/circleci/.local/share/virtualenv/seed-v1/3.7/image/SymlinkPipInstall/pip-20.0.2-py2.py3-none-any/pip

This does not happen on older image digests, where pip happily lives in .venv/lib/python3.7/site-packages/pip. The latest image digest (of tag python:3.7) that we find working is:

f82ee816b296960b042a1b1ca77ffa742a95147b02912b67f1df745d0d691496.

1 Like
2

I recently encountered this on circleci/python:3.6 image containers even without explicitly using pipenv. The virtual envs created by the official virtualenv package had symlinks to shared locations, making the env dir not-self-contained. These shared locations aren’t normally in paths included by caching directives.

It looks like this is a new behavior in virtualenv 20.x that was reverted in 20.0.2. If Circle CI’s pipenv uses the official virtualenv modules, there’s a good chance it would have been creating these bad virtualenvs prior to the inclusion of virtualenv 20.0.2+. The circleci/python:3.6 and circleci/python:3.7 images now seem to have the fixed virtualenv

To reproduce the issue locally using one of the “bad” images:
docker run circleci/python@sha256:5f1335f1ad01744fae0c38c6fcfd142dc5a9b1a611d314638290dd5589f4823f sh -c "virtualenv ~/venv && ls -l ~/venv/lib/python3.6/site-packages"