Flaky Circle CI Tests

Build Environment
1

Ruby: 2.6.3
Rails: 5.2.4.4
Rspec: 3.8.2

Our tests sweet passes locally, but when in our circle ci container we are seeing two main types of failures (although there are more):

  • Persistent data through tests:
    • One of our tests creates a user, and check that User.count is 1. For some reason User.count in circle is 4.
  • Odd format changes
expected: ([{:amount=>0.8215e2, :description=>"Ach Discount - 12/16/2020 - 12/31/2020", :item_code=>15, :type=>"credit_memo"}], {:account_id=>120})
         got: ([{:amount=>82.15, :description=>"Ach Discount - 12/16/2020 - 12/31/2020", :item_code=>15, :type=>"credit_memo"}], {:account_id=>120})

This is only happening to one particular branch, and all of these tests are passing locally. Any ideas as to what is causing this nonsense? This may very well be user error but after inspecting diffs over and over and not finding anything significant, I’m wondering if I could benefit from help from the community.

Circle config:

version: 2
jobs:
  build:
    docker:
      - image: circleci/ruby:2.6.3-node
      - image: redis:3.2.9
      - image: circleci/postgres:10.3
    environment:
      RACK_ENV: test
      RAILS_ENV: test
      CIRCLE_TEST_REPORTS: test-reports
    parallelism: 8
    steps:
      - checkout
      - run:
          name: Switch timezone to Mountain
          command: sudo rm /etc/localtime && sudo ln -s /usr/share/zoneinfo/America/Boise /etc/localtime && date
      - run: sudo apt-get update
      - run: sudo apt install postgresql-client
      - run: mkdir -p tmp/cache $CIRCLE_TEST_REPORTS
      - run: npm install
      - run: gem install bundler
      - restore_cache:
          keys:
            - v1-dep-bundle-{{ checksum "Gemfile.lock" }}
            - v1-dep-bundle-
      - run: bundle check --path=vendor/bundle || bundle install --path=vendor/bundle --jobs=4 --retry=3
      - save_cache:
          key: v1-dep-bundle-{{ checksum "Gemfile.lock" }}
          paths:
            - vendor/bundle
      - run:
          name: Database
          command: |
            mkdir -p config
            echo 'test:
              database: circle_ruby_test
              adapter: postgresql
              encoding: unicode
              pool: 25
              prepared_statements: false
              username: root
              host: localhost
            ' > config/database.yml
            bundle exec rake db:create db:structure:load --trace
      - run:
          name: Tests
          command: |
            TESTFILES=$(circleci tests glob "spec/requests/**/*.rb" "spec/controllers/**/*.rb" "spec/integration/**/*.rb" "spec/concerns/**/*.rb" "spec/services/**/*.rb" "spec/helpers/**/*.rb" "spec/lib/**/*.rb" "spec/mailers/**/*.rb" "spec/models/**/*.rb" "spec/workers/**/*.rb"| circleci tests split --split-by=timings)
            echo ${TESTFILES}
            bundle exec rspec -r rspec_junit_formatter --format RspecJunitFormatter -o $CIRCLE_TEST_REPORTS/rspec/junit.xml -- ${TESTFILES}
          no_output_timeout: 15m
      - run: if [ $CIRCLE_NODE_INDEX == 0 ] ; then bundle exec bundle-audit check --update --ignore CVE-<PRIVATE> ; fi
      - run: if [ $CIRCLE_NODE_INDEX == 1 ] ; then bash ./script/syntax_check.sh ; fi
      - run: if [ $CIRCLE_NODE_INDEX == 2 ] ; then bundle exec brakeman --exit-on-warn ; fi
      - add_ssh_keys
      - store_test_results:
          path: test-reports

Thanks for reviewing

2

I don’t have a good answer for your formatting issues unfortunately. But for the user count issue, I think I know.

Parallelism on jobs runs all of the steps on every executor, not just the testing step. You’ll either want to make your user creation idempotent, or rewrite the tests to account for multiple executors hitting the same database. From the documentation I wouldn’t have guessed this, but I’m pretty sure only the primary executor is duplicated for parallel jobs. Maybe someone else can chime in more authoritatively.

3

A recent pr introduced odd factory bot syntax that was somehow creating records rspec wasn’t tearing down :man_shrugging:.

In one of our factories:

lead { FactoryBot.create(:lead) }

Should’ve been:

association :lead

All is fixed thank you for your response.