Unable to restore cache saved in Docker to machine type job


#1

My test job uses the docker executor type and one of its steps saves a cache. I then attempt to restore that cache in a build job that uses the machine executor type.

The test job:

  • save_cache: # don’t forget to save a build cache, too
    key: v3-build-cache-{{ .Branch }}
    paths: ~/app_name/_build

The build job:

  • restore_cache: # restores saved build cache
    keys:
    - v3-build-cache-{{ .Branch }}
    - v3-build-cache

The build job fails with errors like:

Unarchiving cache…
tar: opt/app: Cannot mkdir: Permission denied
tar: opt/app/app_name/deps: Cannot mkdir: No such file or directory
tar: opt/app: Cannot mkdir: Permission denied
tar: opt/app/app_name/deps/foo: Cannot mkdir: No such file or directory

I’m not sure if this is unsupported or If I’m misunderstanding something. I do not see any way to specify the restoration directory for a given cache. That would be helpful in this case as I would restore it to the home directory.

If this isn’t supported, I would recommend that if a “save_cache” uses a relative path, it should be restored as such across all executor types. If anyone has a recommended work-around that would be much appreciated.


#2

I also tried creating the desired path for my machine executor, build job, in the docker container, test job. Then I soft linked the real directories to those newly created paths. Unfortunately it looks like the tar command used to create the cache doesn’t follow symlinks.


#3

Work around:
Similarly, if within the docker container I create a path that is supported in the machine executor ("/home/circleci/app_name") I am able to copy the files I want into that dir, cache that path, and restore it without error on the machine executor. This appears to be an issue with restoring to absolute paths and is like the macOS issue I found here.

I also tried creating the desired path for my machine executor, build job, in the docker container, test job. Then I soft linked the real directories to those newly created paths. Unfortunately it looks like the tar command used to create the cache doesn’t follow symlinks.

Better solution:
So that’s a decent work around if you really need to have something cached and accessible across executor types. However, in my case I had to step back and ask myself if there was a better solution. Since I’m using this cache to restore build and dependency artifacts in another job further down the workflow I decided to use workspaces instead. I can persist what I need to the workspace in my test job, and restore it in my build job. I have had no issue with this between executor types.

Hopefully this will help others in the future.