Performing the following works every time:
git clone https://github.com/***/deliv (a private repo)
cd deliv
git checkout dockerize-platform
docker pull deliv/platform:base
docker-compose up --build
Great, now all I want is to get this thing to build and run in circleci so I can test it and push the image to ECR.
So my first attempt is to at least get it building by doing the following:
- replacing mysql2://root:@host.docker.internal/deliv_development?pool=5 with mysql2://root:burrito@127.0.0.1:3306/deliv_test?pool=5 in the docker-compose file
- Creating and seeding a mysql database image and redis image in circleci that we can use
- Adding to our current config.yaml a docker-compose build
My issue is this when it hits the docker-compose build step
Mysql2::Error::ConnectionError: Canāt connect to MySQL server on ā127.0.0.1ā (111 āConnection refusedā)
I have no idea why Iām getting this error. Iāve tried to make sure that the ports are open and that the URL is correct.
Here is my full config.yml
version: 2
jobs:
build:
working_directory: ~/deliv
parallelism: 8
# let's not build any of the feature or release branches.
branches:
ignore:
- /feature-.*/
- /release-.*/
# The basic container setup is a debian linux, ruby 2.3 + node.
docker:
- image: circleci/ruby:2.3.6-node
environment:
BUNDLE_JOBS: 3
BUNDLE_RETRY: 3
BUNDLE_PATH: ./vendor/bundle
RAILS_ENV: test
NODE_ENV: test
RUBY_GC_HEAP_INIT_SLOTS: 2000000
RUBY_HEAP_FREE_MIN: 20000
RUBY_GC_MALLOC_LIMIT: 100000000
PARALLEL_TEST_PROCESSORS: 2
BUNDLE_GEMS__CONTRIBSYS__COM: 7aed07e4:b9302537
# This variant of mysql is supposed to set the tables to read and write
# from a ram-disk. Also of note, we can't have a password-less root
# account in circle.
- image: circleci/mysql:5.7-ram
environment:
- MYSQL_ALLOW_EMPTY_PASSWORD=true
- MYSQL_ROOT_HOST=%
- MYSQL_ROOT_PASSWORD=burritos
- image: redis:2.8.21
steps:
- setup_remote_docker
# install the mysql client for the SQL dumps we do later
- run:
name: Install mysql client
command: sudo apt update && sudo apt install -y mysql-client libdigest-sha-perl
# Restore the git cache if we have it.
# Circle says this _may_ be slower than a git checkout, but in my
# testing, it seems to be far faster given our repo size/age
- restore_cache:
keys:
- source-v1-{{ .Branch }}-{{ .Revision }}
- source-v1-{{ .Branch }}-
- source-v1-
# checkout from github
- checkout
# save the git checkout into circle cache.
- save_cache:
key: source-v1-{{ .Branch }}-{{ .Revision }}
paths:
- ".git"
# where we store the output
- run:
name: Create output directories
command: |
mkdir -p ~/deliv/rspec
mkdir -p ~/deliv/output
# since we can't use the same deliv/nopass user we do on local
# development/test, let's copy over a bespoke db yml file with the
# circle specific info.
- run:
name: Copy database config file
command: cp config/database.yml.ci config/database.yml
# gem/bundler cache
- restore_cache:
keys:
- gem-cache-v1-{{ arch }}-{{ .Branch }}-{{ checksum "Gemfile.lock" }}
- gem-cache-v1-{{ arch }}-{{ .Branch }}
- gem-cache-v1-
- run:
name: Bundle Install
command: bundle install --path ./vendor/bundle
- save_cache:
key: gem-cache-v1-{{ arch }}-{{ .Branch }}-{{ checksum "Gemfile.lock" }}
paths:
- ./vendor/bundle
# webpacker or yarn cache
- restore_cache:
keys:
- deliv-yarn-v2-{{ checksum "yarn.lock" }}
- deliv-yarn-v2-
- run:
name: Make sure yarn is in the path
command: echo 'export PATH=/opt/yarn/yarn-v1.5.1/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$PATH' >> $BASH_ENV
- run:
name: Yarn Install
command: yarn install --cache-folder ~/.cache/yarn
# Used to bind the parallel_test output to a single xml file.
# Placed here, so it can be picked up in the node cache.
- run:
name: Install junit-merge
command: sudo npm install -g junit-merge
- save_cache:
key: deliv-yarn-v2-{{ checksum "yarn.lock" }}
paths:
- ~/.cache/yarn
- ./node_modules
# ensure that redis and mysql are up...
- run:
name: Wait for DB
command: dockerize -wait tcp://localhost:3306 -timeout 1m
- run:
name: Wait for Redis
command: dockerize -wait tcp://localhost:6379 -timeout 1m
- run:
name: Checksum the DB migrations and Schema files
command: shasum db/schema.rb db/migrate/* db/seeds/* > db/checksum.tmp
# restore the cache if we have it. This is such a huge improvement. From
# minutes to seconds.
- restore_cache:
keys:
- mysql-dump-v7-{{ checksum "db/checksum.tmp" }}
# the magic bits. If we were able to restore the sql dump, just shove it
# in. If not, we'll do the work to actually generate the tables and run
# rake:test:populate. On average that takes about 3 minutes.
- run:
name: Preparing databases
command: |
if [ -e ~/mysql_dump.sql ]
then
echo "Restoring databases dump"
mysql -h 127.0.0.1 -u root -pburritos < ~/mysql_dump.sql
else
echo "Creating and Seeding databases..."
bundle exec rake parallel:create
bundle exec rake parallel:rake[db:test:populate]
echo "Dumping databases..."
mysqldump --databases $(mysql -h 127.0.0.1 -u root -pburritos -Bse 'show databases like "deliv_test%"') -u root -pburritos -h 127.0.0.1 > ~/mysql_dump.sql
fi
- save_cache:
key: mysql-dump-v7-{{ checksum "db/checksum.tmp" }}
paths:
- ~/mysql_dump.sql
- run:
name: Build application Docker image for web
command: |
docker build --build-arg RAILS_ENV=production --build-arg REDIS_URL=redis://127.0.0.1:6379 --build-arg DATABASE_URL=mysql2://root:burrito@127.0.0.1:3306/deliv_test?pool=5 -f ./docker/app/Dockerfile -t app .
# NPM test. TODO: get this into parallelization
- run:
name: Run NPM test...
command: npm test
# Using parallel_test we can speed our build time even further...
# NOTE: parallel_tests is pulling it's configuration info from the
# .rspec_parallel
- run:
name: Run rspec tests (in parallel)...
command: |
TESTFILES=$(circleci tests glob "spec/**/*_spec.rb" | circleci tests split --split-by=timings --timings-type=filename)
bundle exec $(bundle show parallel_test)/bin/parallel_rspec ${TESTFILES} --verbose
# Combine the multiple output XML files to a single entity for analyzing.
- run:
name: Bundle Junit test results
command: junit-merge -d rspec -o output/output.xml
- store_test_results:
path: ~/deliv/output
- store_artifacts:
path: ~/deliv/output
Here is my full app Dockerfile:
### INSTALL NODE MODULES FOR ASSET COMPILATION
# Lock those nasty node_modules away in their own stage
FROM node:10-slim as node
# INSTALL PYTHON FOR ASSET COMPILATION
RUN apt-get update && apt-get install -y python-pip \
&& rm -rf /var/lib/apt/lists/*
COPY package.json yarn.lock ./
ENV NODE_ENV production
RUN yarn --pure-lockfile --ignore-optional
### BUILD GEMS & ASSETS FOR PRODUCTION
FROM deliv/platform:base as build
# INSTALL DEVELOPMENT PACKAGES
RUN apt-get update && apt-get install -y build-essential \
libxml2-dev libxslt1-dev libmagickcore-dev libmagickwand-dev \
libpq-dev default-libmysqlclient-dev \
&& rm -rf /var/lib/apt/lists/*
COPY Gemfile* ./
RUN bundle install -j20 -r5 --without development test
RUN rm -rf root/.bundle \
&& rm -rf /usr/local/bundle/cache/*.gem \
&& find /usr/local/bundle/gems/ -name "*.c" -delete \
&& find /usr/local/bundle/gems/ -name "*.o" -delete
COPY . .
COPY --from=node node_modules ./node_modules
ENV NODE_ENV production
ARG RAILS_ENV
ARG DATABASE_URL
ARG REDIS_URL
RUN RAILS_ENV=${RAILS_ENV} DATABASE_URL=${DATABASE_URL} REDIS_HOST=${REDIS_URL} \
DEVISE_SECRET_KEY=test \
bundle exec rake assets:precompile && \
rm -rf node_modules
### BUILD THE FINAL IMAGE
FROM deliv/platform:base
COPY --chown=app . .
COPY --from=build /usr/local/bundle/ /usr/local/bundle/
COPY --from=build --chown=app $RAILS_ROOT/public ./public
RUN mkdir -p log tmp/pids && \
chown -R $APP_USER log tmp/pids
RUN rm -rf .git spec
EXPOSE 80
Here is my full base Dockerfile:
FROM starefossen/ruby-node:slim
## INSTALL REQUIRED PACKAGES
RUN apt-get update && apt-get install -y wget git-core libxml2 \
imagemagick mysql-client \
&& rm -rf /var/lib/apt/lists/*
ENV RAILS_ROOT /var/www/platform
ENV APP_USER app
RUN mkdir -p $RAILS_ROOT
WORKDIR $RAILS_ROOT
RUN useradd --user-group $APP_USER && \
chown -R $APP_USER $RAILS_ROOT $GEM_HOME
This is the full stacktrace:
/usr/local/bundle/gems/httpi-2.4.2/lib/httpi/auth/ssl.rb:13: warning: constant OpenSSL::SSL::SSLContext::METHODS is deprecated
** Invoke assets:precompile (first_time)
** Invoke assets:environment (first_time)
** Execute assets:environment
** Invoke environment (first_time)
** Execute environment
rake aborted!
Mysql2::Error::ConnectionError: Can't connect to MySQL server on '127.0.0.1' (111 "Connection refused")
/usr/local/bundle/gems/mysql2-0.5.2/lib/mysql2/client.rb:90:in `connect'
/usr/local/bundle/gems/mysql2-0.5.2/lib/mysql2/client.rb:90:in `initialize'
/usr/local/bundle/gems/activerecord-4.2.11/lib/active_record/connection_adapters/mysql2_adapter.rb:18:in `new'
/usr/local/bundle/gems/activerecord-4.2.11/lib/active_record/connection_adapters/mysql2_adapter.rb:18:in `mysql2_connection'
/usr/local/bundle/gems/activerecord-4.2.11/lib/active_record/connection_adapters/abstract/connection_pool.rb:438:in `new_connection'
/usr/local/bundle/gems/activerecord-4.2.11/lib/active_record/connection_adapters/abstract/connection_pool.rb:448:in `checkout_new_connection'
/usr/local/bundle/gems/activerecord-4.2.11/lib/active_record/connection_adapters/abstract/connection_pool.rb:422:in `acquire_connection'
/usr/local/bundle/gems/activerecord-4.2.11/lib/active_record/connection_adapters/abstract/connection_pool.rb:349:in `block in checkout'
/usr/local/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize'
/usr/local/bundle/gems/activerecord-4.2.11/lib/active_record/connection_adapters/abstract/connection_pool.rb:348:in `checkout'
/usr/local/bundle/gems/activerecord-4.2.11/lib/active_record/connection_adapters/abstract/connection_pool.rb:263:in `block in connection'
/usr/local/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize'
/usr/local/bundle/gems/activerecord-4.2.11/lib/active_record/connection_adapters/abstract/connection_pool.rb:262:in `connection'
/usr/local/bundle/gems/activerecord-4.2.11/lib/active_record/connection_adapters/abstract/connection_pool.rb:292:in `with_connection'
/var/www/platform/config/initializers/activeuuid.rb:4:in `<main>'
/usr/local/bundle/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:50:in `load'
/usr/local/bundle/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:50:in `load'
/usr/local/bundle/gems/activesupport-4.2.11/lib/active_support/dependencies.rb:268:in `block in load'
/usr/local/bundle/gems/activesupport-4.2.11/lib/active_support/dependencies.rb:240:in `load_dependency'
/usr/local/bundle/gems/activesupport-4.2.11/lib/active_support/dependencies.rb:268:in `load'
/usr/local/bundle/gems/railties-4.2.11/lib/rails/engine.rb:652:in `block in load_config_initializer'
/usr/local/bundle/gems/activesupport-4.2.11/lib/active_support/notifications.rb:166:in `instrument'
/usr/local/bundle/gems/railties-4.2.11/lib/rails/engine.rb:651:in `load_config_initializer'
/usr/local/bundle/gems/railties-4.2.11/lib/rails/engine.rb:616:in `block (2 levels) in <class:Engine>'
/usr/local/bundle/gems/railties-4.2.11/lib/rails/engine.rb:615:in `each'
/usr/local/bundle/gems/railties-4.2.11/lib/rails/engine.rb:615:in `block in <class:Engine>'
/usr/local/bundle/gems/railties-4.2.11/lib/rails/initializable.rb:30:in `instance_exec'
/usr/local/bundle/gems/railties-4.2.11/lib/rails/initializable.rb:30:in `run'
/usr/local/bundle/gems/railties-4.2.11/lib/rails/initializable.rb:55:in `block in run_initializers'
/usr/local/lib/ruby/2.5.0/tsort.rb:228:in `block in tsort_each'
/usr/local/lib/ruby/2.5.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
/usr/local/lib/ruby/2.5.0/tsort.rb:422:in `block (2 levels) in each_strongly_connected_component_from'
/usr/local/lib/ruby/2.5.0/tsort.rb:431:in `each_strongly_connected_component_from'
/usr/local/lib/ruby/2.5.0/tsort.rb:421:in `block in each_strongly_connected_component_from'
/usr/local/bundle/gems/railties-4.2.11/lib/rails/initializable.rb:44:in `each'
/usr/local/bundle/gems/railties-4.2.11/lib/rails/initializable.rb:44:in `tsort_each_child'
/usr/local/lib/ruby/2.5.0/tsort.rb:415:in `call'
/usr/local/lib/ruby/2.5.0/tsort.rb:415:in `each_strongly_connected_component_from'
/usr/local/lib/ruby/2.5.0/tsort.rb:349:in `block in each_strongly_connected_component'
/usr/local/lib/ruby/2.5.0/tsort.rb:347:in `each'
/usr/local/lib/ruby/2.5.0/tsort.rb:347:in `call'
/usr/local/lib/ruby/2.5.0/tsort.rb:347:in `each_strongly_connected_component'
/usr/local/lib/ruby/2.5.0/tsort.rb:226:in `tsort_each'
/usr/local/lib/ruby/2.5.0/tsort.rb:205:in `tsort_each'
/usr/local/bundle/gems/railties-4.2.11/lib/rails/initializable.rb:54:in `run_initializers'
/usr/local/bundle/gems/railties-4.2.11/lib/rails/application.rb:352:in `initialize!'
/var/www/platform/config/environment.rb:5:in `<main>'
/usr/local/bundle/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in `require'
/usr/local/bundle/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in `block in require_with_bootsnap_lfi'
/usr/local/bundle/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/loaded_features_index.rb:65:in `register'
/usr/local/bundle/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:20:in `require_with_bootsnap_lfi'
/usr/local/bundle/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:29:in `require'
/usr/local/bundle/gems/activesupport-4.2.11/lib/active_support/dependencies.rb:274:in `block in require'
/usr/local/bundle/gems/activesupport-4.2.11/lib/active_support/dependencies.rb:240:in `load_dependency'
/usr/local/bundle/gems/activesupport-4.2.11/lib/active_support/dependencies.rb:274:in `require'
/usr/local/bundle/gems/railties-4.2.11/lib/rails/application.rb:328:in `require_environment!'
/usr/local/bundle/gems/railties-4.2.11/lib/rails/application.rb:457:in `block in run_tasks_blocks'
/usr/local/bundle/gems/rake-12.3.2/lib/rake/task.rb:273:in `block in execute'
/usr/local/bundle/gems/rake-12.3.2/lib/rake/task.rb:273:in `each'
/usr/local/bundle/gems/rake-12.3.2/lib/rake/task.rb:273:in `execute'
/usr/local/bundle/gems/rake-12.3.2/lib/rake/task.rb:214:in `block in invoke_with_call_chain'
/usr/local/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize'
/usr/local/bundle/gems/rake-12.3.2/lib/rake/task.rb:194:in `invoke_with_call_chain'
/usr/local/bundle/gems/rake-12.3.2/lib/rake/task.rb:183:in `invoke'
/usr/local/bundle/gems/sprockets-rails-2.3.3/lib/sprockets/rails/task.rb:64:in `block (2 levels) in define'
/usr/local/bundle/gems/rake-12.3.2/lib/rake/task.rb:273:in `block in execute'
/usr/local/bundle/gems/rake-12.3.2/lib/rake/task.rb:273:in `each'
/usr/local/bundle/gems/rake-12.3.2/lib/rake/task.rb:273:in `execute'
/usr/local/bundle/gems/rake-12.3.2/lib/rake/task.rb:214:in `block in invoke_with_call_chain'
/usr/local/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize'
/usr/local/bundle/gems/rake-12.3.2/lib/rake/task.rb:194:in `invoke_with_call_chain'
/usr/local/bundle/gems/rake-12.3.2/lib/rake/task.rb:238:in `block in invoke_prerequisites'
/usr/local/bundle/gems/rake-12.3.2/lib/rake/task.rb:236:in `each'
/usr/local/bundle/gems/rake-12.3.2/lib/rake/task.rb:236:in `invoke_prerequisites'
/usr/local/bundle/gems/rake-12.3.2/lib/rake/task.rb:213:in `block in invoke_with_call_chain'
/usr/local/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize'
/usr/local/bundle/gems/rake-12.3.2/lib/rake/task.rb:194:in `invoke_with_call_chain'
/usr/local/bundle/gems/rake-12.3.2/lib/rake/task.rb:183:in `invoke'
/usr/local/bundle/gems/rake-12.3.2/lib/rake/application.rb:160:in `invoke_task'
/usr/local/bundle/gems/rake-12.3.2/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/usr/local/bundle/gems/rake-12.3.2/lib/rake/application.rb:116:in `each'
/usr/local/bundle/gems/rake-12.3.2/lib/rake/application.rb:116:in `block in top_level'
/usr/local/bundle/gems/rake-12.3.2/lib/rake/application.rb:125:in `run_with_threads'
/usr/local/bundle/gems/rake-12.3.2/lib/rake/application.rb:110:in `top_level'
/usr/local/bundle/gems/rake-12.3.2/lib/rake/application.rb:83:in `block in run'
/usr/local/bundle/gems/rake-12.3.2/lib/rake/application.rb:186:in `standard_exception_handling'
/usr/local/bundle/gems/rake-12.3.2/lib/rake/application.rb:80:in `run'
/usr/local/bundle/gems/rake-12.3.2/exe/rake:27:in `<top (required)>'
/usr/local/bundle/bin/rake:23:in `load'
/usr/local/bundle/bin/rake:23:in `<top (required)>'
/usr/local/lib/ruby/gems/2.5.0/gems/bundler-1.16.3/lib/bundler/cli/exec.rb:74:in `load'
/usr/local/lib/ruby/gems/2.5.0/gems/bundler-1.16.3/lib/bundler/cli/exec.rb:74:in `kernel_load'
/usr/local/lib/ruby/gems/2.5.0/gems/bundler-1.16.3/lib/bundler/cli/exec.rb:28:in `run'
/usr/local/lib/ruby/gems/2.5.0/gems/bundler-1.16.3/lib/bundler/cli.rb:424:in `exec'
/usr/local/lib/ruby/gems/2.5.0/gems/bundler-1.16.3/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/usr/local/lib/ruby/gems/2.5.0/gems/bundler-1.16.3/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/usr/local/lib/ruby/gems/2.5.0/gems/bundler-1.16.3/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
/usr/local/lib/ruby/gems/2.5.0/gems/bundler-1.16.3/lib/bundler/cli.rb:27:in `dispatch'
/usr/local/lib/ruby/gems/2.5.0/gems/bundler-1.16.3/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
/usr/local/lib/ruby/gems/2.5.0/gems/bundler-1.16.3/lib/bundler/cli.rb:18:in `start'
/usr/local/lib/ruby/gems/2.5.0/gems/bundler-1.16.3/exe/bundle:30:in `block in <top (required)>'
/usr/local/lib/ruby/gems/2.5.0/gems/bundler-1.16.3/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'
/usr/local/lib/ruby/gems/2.5.0/gems/bundler-1.16.3/exe/bundle:22:in `<top (required)>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
Tasks: TOP => environment
The command '/bin/sh -c RAILS_ENV=${RAILS_ENV} DATABASE_URL=${DATABASE_URL} REDIS_HOST=${REDIS_URL} DEVISE_SECRET_KEY=test bundle exec rake --trace assets:precompile && rm -rf node_modules' returned a non-zero code: 1
Exited with code 1