Tests passing but build passing

Build Environment
1

We have 2 commands to run our tests,

bundle exec rake spec:unit
and
bundle exec rake spec:integration

We have these separated so we can run our faster unit tests a lot more while developing, and only run our integration tests on CI Servers. Well When I drill down, the bundle exec rake spec:integration tests are all failing, but the build still passes.

How do I fix?

2

the build will pass unless you return an exit code of 1. Can you check the exit code on your build, if it comes back as 0 then that is your problem, you’ll need to find a way to return exit code 1 when your tests fail. Not sure how to do that with rake. But this might help https://circleci.com/docs/parallel-manual-setup/#manual-balancing

We use an adapted version of that balancing script.

#!/bin/bash

# We set the ex var to 0 in the beginning
ex=0

i=0
files=()
for testfile in $(find ./test -name "*.py" | sort); do
  if [ $(($i % $CIRCLE_NODE_TOTAL)) -eq $CIRCLE_NODE_INDEX ]
  then
    files+=" $testfile"

# We put this extra piece in here to set $ex=1 when the tests fail.
    if [ $? = 1 ]; then
        ex=1
    fi 

  fi
  ((i=i+1))
done

test-runner ${files[@]}

# Then at the end we exit with the $ex, if any tests have failed, then $ex will be 1, and this will trigger the build to fail

exit $ex

This script is to run tests on multiple containers, but maybe you can adapt this script to exit the tests properly.
Good luck

2 Likes