Tests fail but build passes

Build Environment
1

The job is a “success” even though the Rspec test fails. Also, my jobs are designated as “workflows” even though I have not used the “workflows” key in my config file. Any ideas on how to remedy either of these things? It only needs to be a simple single container job. Thanks!

# Ruby CircleCI 2.0 configuration file
#
# Check https://circleci.com/docs/2.0/language-ruby/ for more details
#
version: 2
jobs:
  build:
    docker:
      # specify the version you desire here
      - image: circleci/ruby:2.6.0-node-browsers
        environment:
          PGHOST: 127.0.0.1
          PGUSER: ubuntu
          RAILS_ENV: test
      - image: redis:2.8.19
      - image: circleci/postgres:9.6
        environment:
          POSTGRES_USER: ubuntu
          POSTGRES_PASSWORD: ubuntu
          POSTGRES_DB: circle_ruby_test

      # Specify service dependencies here if necessary
      # CircleCI maintains a library of pre-built images
      # documented at https://circleci.com/docs/2.0/circleci-images/
      # - image: circleci/postgres:9.4

    working_directory: ~/repo

    steps:
      - checkout

      # Download and cache dependencies
      - restore_cache:
          keys:
          - v1-dependencies-{{ checksum "Gemfile.lock" }}
          # fallback to using the latest cache if no exact match is found
          - v1-dependencies-

      - run:
          name: install dependencies
          command: |
            bundle install --jobs=4 --retry=3 --path vendor/bundle

      - save_cache:
          paths:
            - ./vendor/bundle
          key: v1-dependencies-{{ checksum "Gemfile.lock" }}

      # Database setup
      - run: bundle exec rake db:create
      - run: bundle exec rake db:schema:load

      # run tests!
      - run:
          name: run tests
          command: |
            set -e
            mkdir /tmp/test-results
            TEST_FILES="$(circleci tests glob "spec/**/*_spec.rb" | circleci tests split --split-by=timings)"

            bundle exec rspec --format progress \
                              --format RspecJunitFormatter \
                              --out /tmp/test-results/rspec.xml \
                              --format progress \
                              $TEST_FILES

      # collect reports
      - store_test_results:
          path: /tmp/test-results
      - store_artifacts:
          path: /tmp/test-results
          destination: test-results
2

After your rspec command, try echo $? to get the Unix return code. If this is 0 even in a failure case then rspec or bundle is at fault.

I believe all builds on CircleCI are in a workflow - if you have no need for them then just ignore that tab.

3

This ended up being the problem. In our case there was an exit in a rake task being tested which returned 0.

4

You are most welcome :wink:

5

:+1: Thanks!
Another team member reached the same conclusion and fixed it just before you commented actually haha. But thanks a lot for your insight! This was a tricky one.

1 Like
6

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.