Failing build Ruby mysql2 error

stephen.chen · May 30, 2018, 5:53pm

My CircleCI builds are failing intermittently with the following error:

If I rebuild without cache for the individual builds, they succeed. However it fails pretty frequently when new builds are created and it is frustrating to have to manually re-build every time. I’ve also included my config below.

Failing command: bundle exec rake db:create 
Exit code: 1 
Output:
rake aborted!
LoadError: libmysqlclient.so.18: cannot open shared object file: No such file or directory - /home/circleci/parting-pro/vendor/bundle/ruby/2.4.0/gems/mysql2-0.4.8/lib/mysql2/mysql2.so
/home/circleci/parting-pro/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.2/lib/active_support/dependencies.rb:292:in `require'
/home/circleci/parting-pro/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.2/lib/active_support/dependencies.rb:292:in `block in require'
/home/circleci/parting-pro/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.2/lib/active_support/dependencies.rb:258:in `load_dependency'
/home/circleci/parting-pro/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.2/lib/active_support/dependencies.rb:292:in `require'
/home/circleci/parting-pro/vendor/bundle/ruby/2.4.0/gems/mysql2-0.4.8/lib/mysql2.rb:31:in `<top (required)>'
/home/circleci/parting-pro/config/application.rb:7:in `<top (required)>'
/home/circleci/parting-pro/Rakefile:4:in `require_relative'
/home/circleci/parting-pro/Rakefile:4:in `<top (required)>'
/home/circleci/parting-pro/vendor/bundle/ruby/2.4.0/gems/rake-12.3.1/exe/rake:27:in `<top (required)>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
(See full trace by running task with --trace)
Exited with code 1

# Ruby CircleCI 2.0 configuration file
#
# Check https://circleci.com/docs/2.0/language-ruby/ for more details
#
version: 2
jobs:
  build:
    docker:
      # specify the version you desire here
      - image: circleci/ruby:2.4.4-node-browsers
        environment:
          RAILS_ENV=test
      - image: library/mysql:5.7.19
        environment:
          MYSQL_ALLOW_EMPTY_PASSWORD=true
          MYSQL_ROOT_HOST=%
      - image: redis

      # Specify service dependencies here if necessary
      # CircleCI maintains a library of pre-built images
      # documented at https://circleci.com/docs/2.0/circleci-images/
      # - image: circleci/postgres:9.4

    working_directory: ~/removed

    steps:
      - checkout

      # Download and cache dependencies
      - restore_cache:
          keys:
          - v1-dependencies-{{ checksum "Gemfile.lock" }}
          # fallback to using the latest cache if no exact match is found
          - v1-dependencies-

      - run:
          name: install dependencies
          command: |
            bundle install --jobs=4 --retry=3 --path vendor/bundle

      - save_cache:
          paths:
            - ./vendor/bundle
          key: v1-dependencies-{{ checksum "Gemfile.lock" }}

      - restore_cache:
          keys:
          - v1-yarn-dependencies-{{ checksum "yarn.lock" }}
          - v1-yarn-dependencies-

      - run:
          name: install package.json dependencies
          command: |
            yarn install

      - save_cache:
          paths:
            - node_modules
          key: v1-yarn-dependencies-{{ checksum "yarn.lock" }}

      # Database setup
      - run: bundle exec rake db:create
      - run: bundle exec rake db:schema:load

      # run tests!
      - run:
          name: run tests
          command: |
            mkdir /tmp/test-results
            TEST_FILES="$(circleci tests glob "spec/**/*_spec.rb")"

            bundle exec rspec --format progress \
                            --out /tmp/test-results/rspec.xml \
                            --format progress \
                            $TEST_FILES

      # collect reports
      - store_test_results:
          path: /tmp/test-results
      - store_artifacts:
          path: /tmp/test-results
          destination: test-results

stephen.chen · June 5, 2018, 7:01pm

When i SSH into this build, the file in question does exist:

circleci@1145de4ad865:~/parting-pro/vendor/bundle/ruby/2.4.0/gems/mysql2-0.4.8/lib/mysql2$ ls -al
total 500
drwxr-xr-x 2 circleci circleci   4096 May 14 03:42 .
drwxr-xr-x 3 circleci circleci   4096 May 14 03:41 ..
-rw-r--r-- 1 circleci circleci   5934 May 14 03:41 client.rb
-rw-r--r-- 1 circleci circleci    121 May 14 03:41 console.rb
-rw-r--r-- 1 circleci circleci   1132 May 14 03:41 em.rb
-rw-r--r-- 1 circleci circleci   2346 May 14 03:41 error.rb
-rw-r--r-- 1 circleci circleci     53 May 14 03:41 field.rb
-rwxr-xr-x 1 circleci circleci 465696 May 14 03:42 mysql2.so

mazikwyry · June 8, 2018, 9:14am

I have exactly the same problem.

version: 2
jobs:
  build:
    docker:
      - image: circleci/ruby:2.4-node
        environment:
          PGHOST: 127.0.0.1
          PGUSER: dfgdfgdfg
          MYSQL_URL: mysql2://root@127.0.0.1:3306
          RAILS_ENV: test
      - image: circleci/postgres:9.5-alpine
        environment:
          POSTGRES_USER: dfgdfgdfg
          POSTGRES_DB: tmr
          POSTGRES_PASSWORD: ""
      - image: docker.elastic.co/elasticsearch/elasticsearch:6.2.2
        environment:
          ELASTIC_PASSWORD: 'dfgdfgdfgdfg'
      - image: circleci/mysql:latest
      - image: redis:latest

halfer · June 8, 2018, 10:29am

Prior to the Rake command, could you list the directory involved to see what is in it? You’d just need a step of:

ls -l /home/circleci/parting-pro/vendor/bundle/ruby/2.4.0/gems/mysql2-0.4.8/lib/mysql2/

If that fails too (because the directory does not exist) then back up one or more levels, and use -R. Hopefully that will help find out what state this folder is in prior to the Rake command needing it.

yukimura1227 · June 8, 2018, 3:25pm

I have exactly the same problem.
The cause of the details has not been investigated, but it was solved by the following method.
(reinstal mysql2 gem)

      - restore_cache:
          keys:
            - v2-dependencies-{{ checksum "Gemfile.lock" }}
            - v2-dependencies-

      - run:
          name: install dependencies
          command: |
            bundle install --jobs=3 --retry=3 --path vendor/bundle
      # adding the 2 steps below 
      - run:
          name: remove mysql2 temporary
          command: |
            bundle exec gem uninstall mysql2

      - run:
          name: reinstall mysql2 temporary
          command: |
            bundle install --jobs=3 --retry=3 --path vendor/bundle

      - save_cache:
          key: v2-dependencies-{{ checksum "Gemfile.lock" }}
          paths:
            - vendor/bundle

stephen.chen · June 8, 2018, 5:05pm

Still no solution, but here are some additional details I discovered. Will try the temporary uninstall/reinstall as a workaround.

On a fresh build + SSH into box, here is the PATH and ldd lookup of mysql2.so. Note that libmariadbclient.so.18 is part of the shared library lookup. This is an example of the initial build that succeeds.

$PATH is /usr/local/bundle/bin:/usr/local/bundle/gems/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

cd /home/circleci/parting-pro/vendor/bundle/ruby/2.4.0/gems/mysql2-0.4.8/lib/mysql2/
ldd mysql2.so
	linux-vdso.so.1 (0x00007ffc6bfd4000)
	libruby.so.2.4 => /usr/local/lib/libruby.so.2.4 (0x00007fd77df91000)
	libmariadbclient.so.18 => /usr/lib/x86_64-linux-gnu/libmariadbclient.so.18 (0x00007fd77d98f000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fd77d772000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fd77d558000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fd77d254000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fd77d050000)
	libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007fd77ce18000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd77ca79000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fd77e6ac000)
	libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fd77c6f7000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fd77c4e0000)

On a rebuild these are the results instead:

ldd mysql2.so
	linux-vdso.so.1 (0x00007ffc5e16e000)
	libruby.so.2.4 => /usr/local/lib/libruby.so.2.4 (0x00007fe2febe1000)
	libmysqlclient.so.18 => not found
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fe2fe9c4000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fe2fe7aa000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fe2fe4a6000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fe2fe2a2000)
	libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007fe2fe06a000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fe2fdccb000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fe2ff2fc000)

Note that libmysqlclient.so.18 cannot be found, which I believe is the source of the failure. Unfortunately I have not been able to debug any further. Any help is appreciated.

Topic		Replies	Views
Builds now failing with no such file error (mysql2.so) Feedback & Bug Reports ruby	2	2349	August 15, 2019
Sometimes error occurred when execute rake db:create db:migrate Build Environment rails	7	3849	October 31, 2017
Mysql2::Error: Can't connect to MySQL server on '127.0.0.1' (111) Databases rails , ruby , rspec , mysql	2	11854	October 14, 2017
Formerly passing build now fails Feedback & Bug Reports rails , ruby , rubygems	5	3401	April 8, 2017
Building Rails environment fails because of mysql2-0.4.1 gem error Build Environment	0	840	December 31, 2015

Failing build Ruby mysql2 error

Related topics