Skip job if cache exists

Build Environment
#1

I want to skip a job if a particular cache already exists.

In my case, the first job that runs builds the cache. The cache rarely changes. So each run of the workflow runs the job “cache_rebuild”. The job “cache_rebuild” tries to save that cache. Most often the cache already exists (Circle CI reports “Skipping cache generation, cache already exists”). The job “cache_rebuild” isn’t necessary most of the time.

How do I skip the job “cache_rebuild” if a specific cache already exists?

This is similar to this post regarding skipping a build.

#2

There is no way to skip a build conditionally like this, but what you could do is wrap the steps up in a script that first checks if the cache has changed, and if it has not return an exit code of 0 rather than continuing to regenerate the cache.

#3

@levlaz how would I check if the cache has changed from within a script?

#4

That depends on your cache key. If you are are using a checksum you would check that to see if it has changed from what it was when you restored it.

Can you share your config.yml file?

1 Like
#5

Can you share your config.yml file?

#6

Hi there, I would like to do the same.
Did you manage to do something @jtmoon79 ?

#7

Nope. :frowning:

#8

It looks like you are using cache-{{ .Environment.CACHE_VERSION }}-{{ .Branch }}-{{ arch }}-{{ checksum "Pipfile.lock" }}. I assume that if you do a cache restore and the files do not appear, does that not mean that the hash has changed? In other words, the key lookup failed and your dependency fetch has to do an operation from scratch?

#9

I had to skip several jobs in cache cache exists.
So I did this trick:

lint_code:
executor: my-executor
steps:
  - skip_if_cache_exists:
      skiptype: "linting"
  - run: do job stuff
  - save_cache_flag:
      skiptype: "linting"

and the commands that Im using above: skip_if_cache_exists, save_cache_flag

commands:
  skip_if_cache_exists:
description: |
  a command to exit the job for selected branch
parameters:
  skiptype:
    description: type of job to skip
    type: string
steps:
  - restore_cache:
      key: skipcheck-<<parameters.skiptype>>-{{ .Environment.CIRCLE_BRANCH }}-{{ .Environment.CIRCLE_SHA1 }}
  - run: 
      name: if cache exists exit
      command: |
        FILE=~/cachedflags/job.<<parameters.skiptype>>.flag
        if test -f "$FILE"; then
            echo "$FILE exist"
            circleci step halt
        else
            echo "$FILE doesnt exist"
        fi
save_cache_flag:
  description: |
  a command that will create the cache
  parameters:
    skiptype:
      description: type of job to skip
      type: string
  steps:
    - run:
        name: create job flag file
        command: mkdir -p ~/cachedflags/ && touch ~/cachedflags/job.<<parameters.skiptype>>.flag
  - save_cache:
      key: skipcheck-<<parameters.skiptype>>-{{ .Environment.CIRCLE_BRANCH }}-{{ .Environment.CIRCLE_SHA1 }}
      paths:
        - ~/cachedflags/job.<<parameters.skiptype>>.flag
1 Like
#10

Hi @tzookb - thanks for sharing - I found your commands in the last few days when trying to check if a cache already existed. Here’s my adjustment to allow for:

  1. Category (e.g. src, bundle, node_modules)
  2. Original cache key used as part of the “marker” cache key, rather than a new key. Explicit matching.
  3. Output for each command now includes the category and more descriptive text so that you can separate the original cache from the marker cache quickly on the UI. This especially helps as not everyone I work with will need to know about the underlying mechanism.
  4. Renamed cache commands

I couldn’t get code formatting to play nice today in the forum, so here it is in a gist: