Migration to 2.0 Guidance

ruby
docker
circle.yml

#1

I’m trying to migrate to 2.0 however I’ve been struggling quite a bit with the legacy parts: Ruby 2.2 and Mysql 5.6. I’m still wrapping myself around docker concepts, but reading things makes me think I’m making this far more complicated than it needs to be.

Attempting the automatic config translation has the issue that the Ubuntu image has Mysql 5.7 automatically started which causes our migrations to fail.
https://circleci.com/docs/2.0/config-translation/

I’ve tried to make my own docker image from CircleCI’s Ubuntu image, just removing 5.7, however it seems to be defaulting to Ruby 1.9 instead of the 2.2 that it says it should be the default here: https://circleci.com/docs/1.0/build-image-trusty/

My docker file:

FROM circleci/build-image:ubuntu-14.04-XXL-1327-3401d20

MAINTAINER fir3chi3f "fir3chi3f@asdf.com"

RUN ruby --version

Output:

Step 3/3 : RUN ruby --version
 ---> Running in 033fb3b122dd
ruby 1.9.3p484 (2013-11-22 revision 43786) [x86_64-linux]

Let me know if there is a better way I should be trying. :slight_smile:


#2

How are they failing?


#3

Migrations fail with the following error:

rake aborted!
StandardError: An error has occurred, all later migrations canceled:

Mysql2::Error: All parts of a PRIMARY KEY must be NOT NULL; if you need NULL in a key, use UNIQUE instead: CREATE TABLE `companies` (`id` int(11) DEFAULT NULL auto_increment PRIMARY KEY, `name` varchar(255), `carrier` tinyint(1), `vendor` tinyint(1), `master` tinyint(1), `created_at` datetime NOT NULL, `updated_at` datetime NOT NULL) ENGINE=InnoDB

#4

I can’t see how that might be a CircleCI problem. Somehow your migrations have gotten into a state in which they are trying to define a PK as having a DEFAULT NULL clause. As per the error message you’re getting, that’s not allowed in MySQL (and I can’t imagine it would be useful in any database).

Change your migrations so that either the PK is DEFAULT NOT NULL or the clause is omitted, which should amount to the same thing in practice (since it is auto-inc).

Edit: or should it be DEFAULT auto_increment? I forget the syntax here :smile_cat:


#5

The migration issue isn’t CircleCI’s problem, but nor is it the problem I’m asking for help. I would love to be able to resolve this issue by updating MySQL however that is turning out to be it’s own can of worms just like trying to update to a newer version of Ruby is turning out to be.

1.0 had a downgrade path: https://circleci.com/docs/1.0/downgrading-mysql-to-5.6/
I didn’t see it in the 2.0 doc’s but as far as I can tell the docker image is the same, so I’ll try digging into that a bit.


#6

MySQL allows nulls in a primary key in a later version? That doesn’t sound like a good idea :scream: :laughing:

What Docker image are you using? If your project is sensitive to MySQL version, you can use a custom Docker image containing the exact version of whatever software you want. You can do that in Docker Compose (pull images from your own registry, which is what I do) or Circle supports custom images natively.


#7

Check my original post for the circleci docker image


#8

Gotcha. So you’re building on stock Ubuntu, and will get the MySQL specified in that repo.

I suggest you build the version you want on a local Docker set up, and then you can push that image and use it as per the link I added in my last post. You may be able to find this predefined in the Docker Store, or if you get stuck, use stock Ubuntu and compile the one you want from source - it’s usually pretty easy.


#9

I went back and tried my hand at modifying the auto-generated circleci.yml file and got quite a bit further after going through a docker tutorial. Right now it looks something like this:

# This configuration was automatically generated from a CircleCI 1.0 config.
# It should include any build commands you had along with commands that CircleCI
# inferred from your project structure. We strongly recommend you read all the
# comments in this file to understand the structure of CircleCI 2.0, as the idiom
# for configuration has changed substantially in 2.0 to allow arbitrary jobs rather
# than the prescribed lifecycle of 1.0. In general, we recommend using this generated
# configuration as a reference rather than using it in production, though in most
# cases it should duplicate the execution of your original 1.0 config.
version: 2
jobs:
  build:
    working_directory: ~/myapp
    parallelism: 2
    shell: /bin/bash --login
    # CircleCI 2.0 does not support environment variables that refer to each other the same way as 1.0 did.
    # If any of these refer to each other, rewrite them so that they don't or see https://circleci.com/docs/2.0/env-vars/#interpolating-environment-variables-to-set-other-environment-variables .
    environment:
      CIRCLE_ARTIFACTS: /tmp/circleci-artifacts
      CIRCLE_TEST_REPORTS: /tmp/circleci-test-results
    # In CircleCI 1.0 we used a pre-configured image with a large number of languages and other packages.
    # In CircleCI 2.0 you can now specify your own image, or use one of our pre-configured images.
    # The following configuration line tells CircleCI to use the specified docker image as the runtime environment for you job.
    # We have selected a pre-built image that mirrors the build environment we use on
    # the 1.0 platform, but we recommend you choose an image more tailored to the needs
    # of each job. For more information on choosing an image (or alternatively using a
    # VM instead of a container) see https://circleci.com/docs/2.0/executor-types/
    # To see the list of pre-built images that CircleCI provides for most common languages see
    # https://circleci.com/docs/2.0/circleci-images/
    docker:
    - image: circleci/ruby:2.2.9-browsers
      command: sudo apt-get update && sudo apt-get install mysql-client
      environment:
        RAILS_ENV: test
        DB_HOST: 127.0.0.1
        REDIS_HOST: 127.0.0.1
      command: /sbin/init

    - image: circleci/mysql:5.6
      command: mysqld --character-set-server=utf8mb4 --collation-server=utf8mb4_bin --innodb-large-prefix=true --innodb-file-format=Barracuda

    - image: redis
steps:
# Machine Setup
#   If you break your build into multiple jobs with workflows, you will probably want to do the parts of this that are relevant in each
# The following `checkout` command checks out your code to your working directory. In 1.0 we did this implicitly. In 2.0 you can choose where in the course of a job your code should be checked out.
- checkout
# Prepare for artifact and test results  collection equivalent to how it was done on 1.0.
# In many cases you can simplify this from what is generated here.
# 'See docs on artifact collection here https://circleci.com/docs/2.0/artifacts/'
- run: mkdir -p $CIRCLE_ARTIFACTS $CIRCLE_TEST_REPORTS
# This is based on your 1.0 configuration file or project settings
- run: mkdir ./tmp
- run: mkdir ./spec/support/tmp
# The following line was run implicitly in your 1.0 builds based on what CircleCI inferred about the structure of your project. In 2.0 you need to be explicit about which commands should be run. In some cases you can discard inferred commands if they are not relevant to your project.
- run: echo -e "export RAILS_ENV=test\nexport RACK_ENV=test" >> $BASH_ENV
- run: 'bundle check --path=vendor/bundle || bundle install --path=vendor/bundle
    --jobs=4 --retry=3 '
# The following line was run implicitly in your 1.0 builds based on what CircleCI inferred about the structure of your project. In 2.0 you need to be explicit about which commands should be run. In some cases you can discard inferred commands if they are not relevant to your project.
- run: |-
    mkdir -p config && echo 'test:
      adapter: mysql2
      database: circle_ruby_test
      username: root
      host: 127.0.0.1
    ' > config/database.yml
- run: |-
    export RAILS_ENV="test"
    export RACK_ENV="test"
    bundle exec rake db:create db:migrate --trace
# Test
#   This would typically be a build job when using workflows, possibly combined with build
# The following line was run implicitly in your 1.0 builds based on what CircleCI inferred about the structure of your project. In 2.0 you need to be explicit about which commands should be run. In some cases you can discard inferred commands if they are not relevant to your project.
- run: mkdir -p $CIRCLE_TEST_REPORTS/rspec
- run: |-
    export RAILS_ENV="test"
    export RACK_ENV="test"
    bundle exec rspec --colour --require spec_helper --format RspecJunitFormatter --out $CIRCLE_TEST_REPORTS/rspec/rspec.xml --format progress $(circleci tests glob spec/**/*.rb | circleci tests split)
# This is based on your 1.0 configuration file or project settings
- run:
    name: bundle exec rake spec:javascript
    command: |-
      export RAILS_ENV="test"
      if [ "$CIRCLE_NODE_INDEX" == "0" ]; then bundle exec rake spec:javascript; fi
# Teardown
#   If you break your build into multiple jobs with workflows, you will probably want to do the parts of this that are relevant in each
# Save test results
- store_test_results:
    path: /tmp/circleci-test-results
# Save artifacts
- store_artifacts:
    path: /tmp/circleci-artifacts
- store_artifacts:
    path: /tmp/circleci-test-results

Which leaves me with one last issue during the rspec run:

Failures:

  1) SearchSetup values can utilize all 'Security Filing' core module model fields in a SearchQuery
     Failure/Error: rs = ActiveRecord::Base.connection.execute to_sql opts
     
     ActiveRecord::StatementInvalid:
       Mysql2::Error: Out of sort memory, consider increasing server sort buffer size: SELECT DISTINCT BlablaMyReallyLongQuery...
     # ./app/models/search_query.rb:30:in `execute'
     # ./spec/models/search_setup_spec.rb:174:in `block (5 levels) in <top (required)>'
     # ./spec/models/search_setup_spec.rb:163:in `block (4 levels) in <top (required)>'
     # ------------------
     # --- Caused by: ---
     # Mysql2::Error:
     #   Out of sort memory, consider increasing server sort buffer size
     #   ./app/models/search_query.rb:30:in `execute'

Hopefully that’s not too much text all at once :slight_smile:


#10

Progress, nice one! If you are on a paid plan, you can increase the RAM in your machine.

However, it may just be a MySQL or Ruby setting, so I would try reconfiguring that first.


#11

You probably want to specify two more properties in your mysqld command:

https://dev.mysql.com/doc/refman/5.6/en/innodb-parameters.html#sysvar_innodb_sort_buffer_size

https://dev.mysql.com/doc/refman/5.6/en/server-system-variables.html#sysvar_sort_buffer_size

These are the values I’m using for our application:

--innodb_sort_buffer_size=67108864 --sort_buffer_size=268435456


#12

Thank you Jason! That actually was the solution.

Note: It’s not innodb_sort_buffer_size like I kept trying to use prior to your post


#13

This topic was automatically closed 41 days after the last reply. New replies are no longer allowed.