Caching on a machine executor

Build Environment
,
1

Is caching on the machine executor done as below:

      - restore_cache:
          keys:
            - make-382-{{ arch }}-{{ .Branch }}-{{ .Environment.CIRCLE_SHA1 }}
            - make-382-{{ arch }}-{{ .Branch }}
      - run:
          name: Build and install make via script
          command: ./build/x86_64/linux_macos/installMake.sh
          timeout: 1m
      - save_cache:
          key: make-382-{{ arch }}-{{ .Branch }}-{{ .Environment.CIRCLE_SHA1 }}
          paths:
            - /make-3.82/
            - /usr/bin/make
            - /usr/local/bin/make
            - /usr/share/man/man1/make.1.gz
            - /lib/

I tried doing this sometime back and got a lot of errors when trying to restore the cache on a new build - this could have to do with file permissions on the new instance.

Is there a way around this?

2

This portion effectively says to only cache/restore these files for the current commit. Try removing this last portion to allow your cache to apply to {{ arch }}-{{ .Branch }}, which scopes to this Arch type and this branch.

3

Thanks I have just applied it and will let you know what transpires from it.

4

When I mentioned about errors earlier if you take a look at https://circleci.com/gh/neomatrix369/awesome-graal/627 and https://circleci.com/gh/neomatrix369/awesome-graal/628 you will see what I mean.

I think the ones in https://circleci.com/gh/neomatrix369/awesome-graal/627 can be avoided by using unique names but we can still come across these unintentionally. For the ones in https://circleci.com/gh/neomatrix369/awesome-graal/628 I have had them in the past and it stopped me from using caches.

A side question, which folders are of importance when we install a linux package, are these sufficient:

    - "vendor/apt"
    - "vendor/apt/archives"
5

@KyleTryon any feedback for me on this issue?

6

It’s not clear if your post after this one gave the detail of your findings. Would you expand on how you acted on Kyle’s advice? :slight_smile:

7

If you look at this commit onwards - https://github.com/neomatrix369/awesome-graal/commit/0a73f25cc455300fb642f306fc532c1c4feb84d7, its what I applied after @KyleTryon’s advise.

And the results are here Caching on a machine executor (see post above starting with When I mentioned....).

Does this help to give a starting point?

Cheers
Mani

8

@halfer Any chance of an update for my last response?

9

Ah, apologies - I did not have anything further to add. I was asking for clarifications based on what I thought Kyle would ask you next, to save them having to ask.

If this issue really has been in flight for 24+ days, I would personally change track. That’s not to say it won’t work, just that it is sensible to timebox each approach, so you don’t spend more than a day or two on each approach. I would suggest getting an S3 bucket or an external SSH server, from/to where you can scp a cache tarball. Worth a try?

10

It possible both yourself and @KyleTryon might have got the problem the wrong end of the stick. If you see the config on https://circleci.com/gh/neomatrix369/awesome-graal/627 I have done as per the docs.

While I revert using the feature, could you please show the errors reported in the logs on https://circleci.com/gh/neomatrix369/awesome-graal/627 and https://circleci.com/gh/neomatrix369/awesome-graal/628 to your engineers to find out why this could be happening? Has the caching feature worked for everyone?

The logs show a mix of ‘File already exists’ and ‘Permission denied’ error - is this a known issue, appears to be a user permission issue:

Found a cache from build 627 at make-arch1-linux-amd64-6_63-build-graalvm-suite
Size: 36 MB
Cached paths:
  * /make-3.82
  * /usr/bin/make
  * /usr/local/bin/make
  * /usr/share/man/man1/make.1.gz
  * /lib

Downloading cache archive...
Validating cache...

Unarchiving cache...
tar: usr/bin/make: Cannot open: File exists
tar: usr/local/bin/make: Cannot open: Permission denied

I’d like to know if its not allowed to cache and restore files or folders lib and usr/bin/make - if so why?

11

I’ll leave that for Kyle - I don’t work for CircleCI, and I am only here in a volunteer capacity.

12

Np thanks for the support. I’ll add additional observations or queries to the bottom of this thread.

13

Found new errors happening even when not using caching, see https://circleci.com/gh/neomatrix369/awesome-graal/656

Linux/Unix specific steps
Running in a real OS environment

Reading package lists... Done

E: Could not get lock /var/lib/apt/lists/lock - open (11: Resource temporarily unavailable)
E: Unable to lock directory /var/lib/apt/lists/
Exited with code 100
14

Hey @neomatrix369,

This particular bug is one of the more annoying ones but unrelated to the original issue. This happens when a mirror from ubuntu has gone offline or is particularly slow for some reason. This also generally only affects the VM executor. The update process from the spinup step is still running (likely due to a mirror issue) and will need to be killed. This issue usually fixes itself within a few hours if it’s network related.

We’ve put together some options for avoiding this particular issue: https://support.circleci.com/hc/en-us/articles/360021256633-Apt-Get-Update-Is-Slow-Or-Locked

For the original issue you have posted. For a speedier response and for the nature of the issue, I would recommend opening up a ticket with the support team.

I see one of the builds above is attempting to pull a cache form a job that is over a month old. This cache likely no longer exists.

I would recommend rolling your cache keys. Could you take all of your keys and affix a number to them like this?

llvm-{{ arch }}-{{ .Branch }} :point_right:llvm-v2-{{ arch }}-{{ .Branch }}

And if there is already a number, try incrementing it up one just to generate a new cache. :+1: This should result at first in the next commit having no cache at all, call this build X. Your next commit (Y), you should see that it restores the cache from build X.

If there is still a permission issue at this point it will be easy for us to identify.

15

@KyleTryon Thanks for the feedback and looking deeper into the caching issue, I have reproduced this now https://circleci.com/gh/neomatrix369/awesome-graal/666

All of the keys have a -v2- in their names, and it has just about failed to restore the first cache make-v2-arch1-linux-amd64-6_63-build-graalvm-suite.

16

@KyleTryon any update for the above? We have an instance where the restoring of the cache does fail.