Caching Ruby Gems

Build Environment
1

I’m trying to cache Ruby gems as part of a large project, but it doesn’t seem to be working. It seems to always install the bundler gems and takes a bunch of time. I do see inside of CircleCI that it’s saving the cache and restoring from it, but it doesn’t seem to work. I’m able to SSH into the build and see the gems in that directory. I feel like I’m making a simple mistake.

Here is my task with CircleCI. I have to use machine as this is only part of a longer task.

  visual-regression-test:
    machine: true
    steps:  
      - run:
          # Installs PhantomJS
          name: Install phantomjs
          command: |
            if ! [ $(which phantomjs) ]; then                
              curl --output /usr/local/bin/phantomjs https://s3.amazonaws.com/circle-downloads/phantomjs-2.1.1
            fi
      - checkout
      - restore_cache:
          keys:
            - v1-gem-cache-{{ arch }}-{{ .Branch }}-{{ checksum "Gemfile.lock" }}
      - run:
          name: "Install Dependencies"
          command: |
            cd ~/project
            gem install bundler
            bundle install
      - save_cache:
          key: v1-gem-cache-{{ arch }}-{{ .Branch }}-{{ checksum "Gemfile.lock" }}
          paths:
            - /opt/circleci/.rvm/gems/ruby-2.3.1/gems/
      - run:
          name: "Run  Snapshots"
          command: |
            cd ~/project
            yarn build
            cd ~/project/mocks/visual
            bundle exec ruby snapshots.rb

Any help would be greatly appreciated.

3

Woops, sorry about that.

1 Like
4

There’s one error and one worry:

The error is your restore_cache doesn’t specify a path to cache, so I guess this would be ignored.

The worry is that you’ve used keys for one and key for the other. I would imagine it would be good to be consistent, even if what you have would work.

Edit

Ha! I just realised you and I must have copied this from the docs, since I have the key/keys thing too. Not spotted that before…

5

I do see the restored caching occuring in CircleCI, but bundler always re-downloads the gems instead of detecting them as installed.

Found a cache from build 359 at v1-gem-cache-arch1-linux-amd64-6_63-63-ujAeh3KHETUukkuPFbJL2UMLKEjHvsJ4rLigsss7nng=
Size: 8.1 MB
Cached paths:
  * /opt/circleci/.rvm/gems/ruby-2.3.1/gems

Downloading cache archive...
Validating cache...

Unarchiving cache...

Do I need to cache the bundler install too? (And if so how?)

6

Hmm, odd. I can’t comment on Bundler (since I barely know what it is, I assume it’s your dependency manager) but that looks like it is nearly working. Perhaps not specifying a path means that all cached paths are restored? That said, I wonder if trying to add an explicit path would be worth a go, just in case?

Are the Gems used from the location you have specified? I don’t use Ruby, but in PHP they are brought inside the project itself.

7

I’ve tried that too but end up with the issue. The folder /opt/circleci/.rvm/gems/ruby-2.3.1/gems/ contains everything I need for my project.

addressable-2.5.2  capybara-2.18.0  faraday-0.15.0	 multipart-post-2.0.0  percy-client-1.13.8  public_suffix-3.0.2  websocket-driver-0.7.0
bundler-1.15.4	   cliver-0.3.2     mini_mime-1.0.0	 nokogiri-1.8.2        phantomjs-2.1.1.0    rack-2.0.4		 websocket-extensions-0.1.3
bundler-1.16.1	   excon-0.62.0     mini_portile2-2.3.0  poltergeist-1.17.0   rack-test-1.0.0	 xpath-3.0.0

If I login via SSH and run bundle install it detects that they are already installed, so it’s defo something to do with how it’s restoring the cache. At a loss :frowning_face:

8

Can you run your bundle install in the build in verbose mode, so it can show you in console output why it is installing stuff that is already installed?

Also, prior to doing the install, try doing an ls -R /opt/circleci/.rvm/gems/ruby-2.3.1/gems/ in the build, to make sure the cache really has been restored.

9

It’s definitely present prior to running bundle install. Unless bundler is installing these gems elsewhere, or it just doesn’t link the two together. Running the gem location commands reveals the directory I’m caching.

Fetching gem metadata from https://rubygems.org/..........
Resolving dependencies...
Using public_suffix 3.0.2
Using addressable 2.5.2
Using bundler 1.16.1
Using mini_mime 1.0.0
Using mini_portile2 2.3.0
Fetching nokogiri 1.8.2
Installing nokogiri 1.8.2 with native extensions # This is where it hangs
10

I wonder if this might be a Stack Overflow question in the making: “How does Bundler determine whether to re-install something or use the currently installed version?” :smiley:

11

I’ll try there. Thanks!

1 Like
12

I do caching of Ruby gems with the follow snippet:

- restore_cache:
    key: circlev2-{{ checksum "Gemfile.lock" }}
- run: bundle check --path=vendor/bundle || bundle install --path=vendor/bundle --jobs=4 --retry=3
# Store bundle cache
- save_cache:
    key: circlev2-{{ checksum "Gemfile.lock" }}
    paths:
      - vendor/bundle
1 Like
13

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.