Cache not restoring properly

We have build caching of large subdirectories (~1-2GB) of $HOME that seem to be cached properly during build:

https://circleci.com/gh/mne-tools/mne-python/1428

But then only a subset get loaded later:

https://circleci.com/gh/mne-tools/mne-python/1432

We have tried clearing the cache (right before the top-link build occurred) but this did not solve the issue.

Any ideas? Is there a size limit on the directories that can be cached? (If so, it would help to add it to the caching docs.)