Circle Bug Error Restoring Cache

Build Environment
1

Last 24 hours, we have been seeing this error. Any body has similar issues?

CircleCI team, can you guys help us out ?

Looks like we had a bug in our infrastructure, or that of our providers (generally GitHub or AWS) We should have automatically retried this build. We’ve been alerted of the issue and are almost certainly looking into it, please contact us if you’re interested in the cause of the problem.

2

Hi there,

This bug typically occurs when directories outside of the $HOME directory have been cached. You should be able to get things working again by removing these directories from the cache_directories part of your circle.yml file. If you really need to cache things outside of $HOME, you can do so by following these steps:

  1. cp the files you’d like to cache into somewhere inside the $HOME directory before the cache is saved (e.g. dependencies: post: in your circle.yml).
  2. Use sudo cp to put the files back to where you need them, after the cache is restored (e.g. machine: post: in your circle.yml).
  3. Add the directory you created inside $HOME to your cache_directories.

Example circle.yml:

machine:
  post:
    - cp -r /usr/local/bin/git $HOME/git
dependencies:
  cache_directories:
    - $HOME/git
  post:
    - sudo cp -r $HOME/git /usr/local/bin/git

Hope this helps! If this solves your problem, please mark this thread as solved by clicking on the checkbox below.

Cheers,
Frank

2 Likes
3

Hi Frank - I work with Ravi – we’re still seeing an error even though we’ve removed the lines outside of /home (this really should be documented).

The issue being that when the system tries to restore the old cache, it still tries to restore the directories outside of /home. At this stage the only way we can get a green build is to manually “rebuild without cache” - but doing that doesn’t wipe out those old invalid cache entries.

Is there a way to wipe out the cache, so we can start with a blank slate?

4

Normally you will not have to clear the cache permanently, but if you
feel that’s what you need, you can just remove the necessary parts of
the cache anywhere in your circle.yml’s dependencies section,
before the cache is saved:

dependencies:
post:
- rm -r ~/.gradle

5

Folks, I think you’re missing what we’re saying here.

The error now is not in the saving of the cache (that now works fine, and we’ve removed non-home entries) - it’s in the restoration of the cache from previous builds (that had entries in /var and such) - meaning there’s nothing we can do prior to that step to make it work.

The issue being the cache still has entries to /var and such (from the older builds), so whenever we perform a new build, it errors out when it tries to restore the cache.

Is there a way to wipe out the cache entirely? “build without cache” doesn’t seem to do it for us - the next automated builds just fail with the same old restore error.

6

hubot circle clear / - Clears the cache for the specified repo

hubot circle clear all - Clears the cache for all projects. Requires HUBOT_GITHUB_ORG

7

Excellent - thanks for that tip - I forgot that cache management might be offered by your API. That looks to have solved the problems we were having.

Now on to making the build faster!

Thanks again,

-Dave

8

I just want to add a little more explanation to this post for users who are reading through this today.

Cache restore downloads a tarball of a directory you specied, and then attempts to un-tar it into the location you specify. When you specify a folder not owned by the ubuntu user, the untar will have a permissions error.

While it’s possible to hack around this by using sudo chown, that doesn’t always work. I believe this is due to your build containers being unprivileged lxc containers, so certain superuser commands will be ignored. That’s why it’s best to keep anything you want to cache in the one folder you’re guaranteed full permissions to: your home folder.