Skip a step on cache-restore


#1

possible to add a feature to skip a step if cache-restore was successful?


We’d like your feedback!
We’d like your feedback!
#2

No. Are you able to check for the presence of a file before executing a step? Can you elaborate on your use case? It would help to open and justify a feature request.


#3

that’s exactly it. I want to cache some dependencies if its definitions haven’t changed. simply restore cache and skip the install.

but can certainly use a shell conditional to accomplish this.


#4

also, what’s the verdict on AWS credential support?


#5

There is no verdict at this time.

We have a few conversations going about changing/improving the caching ability. That can be included :slight_smile:


#6

I second this request. I have a set up where I’m restoring the cache based on branch name, but want to restore from a shared (master) cache by default. This works now, but if I am building on master, I’ll have a redundant cache restore. Not a huge deal, but inefficient. This could be solved by a conditional on my second cache restore that is similar to if cache-name-{{ .Branch }} != "master" then <restore cache>.


#7

So the keys field on cache-restore might do exactly what you need. Instead of specifying a key, specify a yaml array:

          - type: cache-restore
            keys:
              - projectname-{{ .Branch }}-{{ checksum "project.clj" }}
              # Providing keys in decreasing specificity means it's more likely a new cache can be built from an existing one.
              - projectname-master-
              - projectname-

This is currently only mentioned briefly in our docs, but it really should be called out more clearly. I’ll submit this change to the rest of the team. Let me know if that solves your problem or not.


#8
  • restore_cache
  • if restored_cache === false { build deps }
  • save_cache

seems like a pretty common scenario?


#9

I can open a feature request for you, but can you elaborate on your use case for that?

A lot of use cases are like this, using a check first:

  • restore cache
  • bundle check || bundle install
  • save_cache

The save_cache step looks at the key you’re using and will not overwrite the cache if it already exists. If the cache existed initially, the install would never be hit because of the check.


#10

Hi and much thanks for the quick reply.

I assume that bundle check would place the burden at the application logic then? For me, it makes more sense for the logic to at least be controlled from the cache handler seeing how it created it [the cache] in the first place. For instance, exposing an environment variable with the cache key or something else that is easily testable against.

Also; streamlining the test would avoid bashisms or other “approaches” to verifying this key; so providing a consistent command would likely be preferable. Pardon my poor example:

 - restore_cache
 - run: if_cached($key) || command
 - save_cache

Edit: elaborating on consistent: the idea is that if you provide a simple to use command, your users will implement less buggy versions of it ultimately leading to less support.


#11

perhaps a fancier syntax would be:

- restore_cache
- run_if: arg:true cmd:..
- save_cache

#12

I would imagine something like that:

- restore_cache
- run:
    name: Launch migrations if needed
    command: rake db:migrate
    if: cached("rails-{{ checksum "migrations-manifest" }}")
- save_cache

#13

I would like to see the ability to skip any number of steps. For example, I’d like to avoid duplicating the check for "${CIRCLE_BRANCH}" == "master" and just skip all 3 of these steps in the case that it’s false.

      - run:
          name:
          command: bash .circleci/setup-heroku.sh

      - add_ssh_keys:
          fingerprints:
            - "fingeprint"

      - deploy:
          name: Deploy Master to Heroku
          command: |
            if [ "${CIRCLE_BRANCH}" == "master" ]; then
              git push heroku master
            fi

#14

I have the case:

  - restore_cache:
      keys:
        - v1-node-{{ checksum "package.json" }}
        # if cache for exact version of `package.json` is not present then load any most recent one
        - v1-node-

  # set up docker on a remote machine, where the images are cached.
  - setup_remote_docker:
      reusable: true

  - run:
      name: Start docker container to build the project
      command: |
        docker rm -f webbuild
        docker run -d --name webbuild repo/my-maven-image:v123 

  - run:
      name: Copy code and nodejs cache inside the build container
      command: |
        # It is not possible to use a volume as the webbuild container is remote.
        docker cp $(pwd) webbuild:/usr/src/app

  - run:
      name: Build .ear package
      command: |
        docker exec -it webbuild mvn clean package

  - run:
      name: Copy nodejs cache on the local/CI container
      command: |
        # This allows to save and restore this cache for the next build
        rm -rf node_modules
        docker cp webbuild:/usr/src/app/node_modules node_modules

  - save_cache:
      key: v1-node-{{ checksum "package.json" }}
      paths:
        - node_modules/

The step, Copy nodejs cache on the local/CI container, takes 15 seconds, and it unnecessary most of the time.
If would be nice to be able to skip it when a specific version of the cache is present.
Here, in the restore_cache step, we can’t know if
"v1-node-{{ checksum “package.json” }}"
or
"v1-node-"
was restored.

Having a function like

  • cached("v1-node-{{ checksum "package.json" }")
  • cache_key_exists("v1-node-{{ checksum "package.json" }")
    returning true or false would be nice.

Then the step: Copy nodejs cache on the local/CI container could become:

  - run:
      name: Copy nodejs cache on the local/CI container
      command: |
        # This allows to save and restore this cache for the next build
        if [[ cache_key_exists("v1-node-{{ checksum 'package.json' }") eq 1 ]]; then
          docker cp webbuild:/usr/src/app/node_modules node_modules
        fi

#15

so 1 year later - any update on this feature? i can write some hacks that verify some files exist, but it seems like it’d be much cleaner if this feature was supported at the platform level.


#16

Is there any progress on this feature request?

I’m working on a Rails project, using that uses sprockets and would like to skip asset precompilation depending on the availability of a cache. I’m using CircleCi’s workflows and have the asset precompilation happening on one job and all other jobs referencing the cache (which is great). However, I’d like to skip the precompilation if there’s any cache available — and this is proving to be overly difficult.

Per the documentation there’s an alternative recommendation to do:

ls -laR your-deps-dir > deps_checksum

and then reference it with {{ checksum "deps_checksum" }}. This works, but it’s not clear how to create and reference this key without committing it to version control. I’ve attempted to leverage workspaces to save this file which does work across jobs but it doesn’t seem to work across separate builds (e.g., {{ checksum "/tmp/workspace/deps_checksum" }}).

Is there a better approach or convention for doing this type of check within CircleCI?


#17

Maybe I am abusing the caching system here, but I found this thread while trying to do what I think most people are describing. But for a different use case (a firmware project).
To do a conditional step of downloading the toolchain & SDK from a fixed location if the restore_cache step did not do that already, I used a bash script get_assets.sh, like this:

#!/bin/bash
if [ -e ti ]
then
    echo "Assets restored via CircleCI cache"
else
    set -e
    echo "Downloading and unpacking assets"
    wget http://domain.org/assets.tbz2
    tar xjf assets.tbz2
fi

And added it after the restore_cache step in the YML file, as:

-run:
   command: |
      ./get_assets.sh

It seems to work as I needed it to.


#18

@apullin: this forum offers Markdown formatting, which you can use to preserve indentation in your code/config samples. Your unformatted YAML has been reformatted incorrectly as an unordered list - if you edit your post, you’ll see the code formatting device </> in the toolbar.


#19

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.