Unable to restore bundle from cache using circleci/ruby docker images

[dominicsayers]

Using the Docker image circleci/ruby it’s impossible to restore a bundler cache saved to the default location of /usr/local/bundle.

The image already has a /usr/local/bundle directory pre-created by the root user.

$ ls -Al /usr/local
total 36
drwxrwsr-x 1 root staff 4096 Aug 11 00:06 bin
drwxrwsrwx 2 root staff 4096 Jul 21 08:36 bundle
drwxrwsr-x 1 root staff 4096 Jul 17 07:54 etc
drwxrwsr-x 2 root staff 4096 Jul 16 00:00 games
drwxrwsr-x 1 root staff 4096 Jul 21 08:36 include
drwxrwsr-x 1 root staff 4096 Jul 21 08:36 lib
lrwxrwxrwx 1 root staff    9 Jul 16 00:00 man -> share/man
drwxrwsr-x 2 root staff 4096 Jul 16 00:00 sbin
drwxrwsr-x 1 root staff 4096 Jul 17 03:16 share
drwxrwsr-x 2 root staff 4096 Jul 16 00:00 src

But the build tasks are run as the circleci user.

$ whoami
circleci

And the default bundler options are:

$ bundle config
Settings are listed in order of priority. The top value will be used.
app_config
Set via BUNDLE_APP_CONFIG: "/usr/local/bundle"

path
Set via BUNDLE_PATH: "/usr/local/bundle"

silence_root_warning
Set via BUNDLE_SILENCE_ROOT_WARNING: true

The usr/local/bundle directory has drwxrwsrwx permissions, so we can use a Circle 2.0 config.yml to bundle install and save the cache:

  - run: bundle install
  - save_cache:
      key: v1-bundle-{{ checksum "Gemfile.lock" }}
      paths:
        - /usr/local/bundle

But when we come to restore the cache for the next job we get errors:

Found a cache from build 45 at v1-bundle-37RyclRo4zIg9I2R_L+I72Po+eXbphREAXxknkNduy8=
Size: 60 MB
Cached paths:
  * /usr/local/bundle

Downloading cache archive...
Validating cache...

Unarchiving cache...
tar: usr/local/bundle: Cannot utime: Operation not permitted
tar: usr/local/bundle: Cannot change mode to rwxr-sr-x: Operation not permitted
tar: Exiting with failure status due to previous errors
Error untarring cache: exit status 2

I think this is because of the problem described by Dave Hay in this blog post. tar cannot set the file time for files in a folder that is owned by root.

$ mkdir ~/foo
$ chown root:root ~/foo
$ chmod a+w ~/foo
$ cd ~/foo
$ tar xvf /media/LenovoExternal/Products/DB297/db297linux32.tar
tar: .: Cannot utime: Operation not permitted
tar: Exiting with failure status due to previous errors

I cannot see a way round this with the circleci/ruby images. restore_cache cannot use tar in this folder.

My questions are:

1. Why does this folder exist in the Docker image? It is empty.
2. Why, if it must exist, can it not be owned by the circleci user?
3. Is your implementation of restore_cache visible anywhere (on Github maybe), so I can check out what it’s trying to do?

[dominicsayers]

I’m aware of the vendor/bundle workaround, I’m just interested why it’s done like this.

[dominicsayers]

The fact that BUNDLE_APP_CONFIG and BUNDLE_PATH are both set to usr/local/bundle in the image shows me that this is the folder we are supposed to use.

[avi]

I just encountered this as well.

[avi]

I’ve found a workaround: before the restore_cache step, I’ve got a run step with the command sudo chown -R circleci /usr/local/bundle — after I added this, the subsequent restore_cache step is now working.

[dominicsayers]

Hi Avi, this is helpful, thank you.

I tried variations of this workaround but my project has multiple steps in the workflow so I found myself having to repeatedly fix the permissions in every step.

It’s not possible to fix the permissions then cache the fix, because the problem exists in the underlying Docker image .

I hope someone from CircleCI sees this thread and fixes the underlying problem.

[Cormac]

Hi all, I’ll pass this on and see if we can get it looked at. I’ll update this thread when I know anything more.

[dominicsayers]

Hi Cormac. Thanks for passing this on.

I’ve created a Github repo to demonstrate the problem. This is the simplest config.yml I can make that exhibits this behavior: https://github.com/dominicsayers/circleci-2.0-cache-restore/pull/1

I’ve removed everything except saving and restoring the cache.

This config file works with the vanilla ruby image but fails with the circleci/ruby image because the build steps are run by the circleci user rather than root.