Build failed but it's green (used to be red few days ago)

Build Environment
1

Hi,

I use CircleCI to build (with docker) an environement to run my symfony 3.4 application with nginx and mysql.

The main goal is to execute my unit and functionnal tests with PHPUnit and Behat.

I just noticed than the builds are green even if my tests have failed. It was not the case 30 days ago.

Did I miss a recent update ? should I change my circleci config file ?

# PHP CircleCI 2.0 configuration file
#
# Check https://circleci.com/docs/2.0/language-php/ for more details
#
version: 2
jobs:
    build:
        # docker:
        #- image: circleci/php:7.1-cli-jessie-node-browsers-legacy
        machine: true
        working_directory: ~/repo

        steps:
            - checkout
            - run: mkdir ~/repo/features/fail-screenshots

            - run: cp .circleci/.env.dist .env
            - run: cp .circleci/parameters.yml.dist app/config/parameters.yml
            - run: docker-compose up -d

            # Download and cache dependencies
            - restore_cache:
                keys:
                    - v1-dependencies-{{ checksum "composer.json" }}
                    # fallback to using the latest cache if no exact match is found
                    - v1-dependencies-

            - run: docker-compose exec --user 1001:1002 php composer install -n --prefer-dist

            - save_cache:
                paths:
                    - ./vendor
                key: v1-dependencies-{{ checksum "composer.json" }}

            - run: docker-compose run --rm front yarn install
            - run: docker-compose run --rm front yarn build

            # run tests!

            # Circleci uses a user with id 1001 and a group with id 1002
            # We use this user instead of the default one in our dockerfile (1000)
            # to avoid permissions issues
            - run: docker-compose exec --user 1001:1002 php ./bin/console c:c --env=test
            # We use the root user here to change logs and cache permissions
            - run: docker-compose exec --user="root" php chmod -R 777 var/logs
            - run: docker-compose exec --user="root" php chmod -R 777 var/cache
            - run: docker-compose exec --user="root" php chmod -R 777 var/sessions
            - run: docker-compose exec --user="root" php chmod -R 777 var/uploads

            # Fixtures
            - run: docker-compose exec php console doctrine:database:drop --if-exists --force --env=test
            - run: docker-compose exec php console doctrine:database:cre --env=test
            - run: docker-compose exec php console doc:sch:cre --env=test
            - run: docker-compose exec php console doctrine:fixtures:load --env=test -n --fixtures=src/Infrastructure/Doctrine/DataFixtures/ORM

            # BEHAT
            # Here we use 1001:1002 to launch behat cause it can create screenshots
            # so it needs to write in a directory owned by 1001:1002 (circleci user)
            - run: docker-compose exec --user 1001:1002 php ./vendor/bin/behat -v --suite=back
            - run: docker-compose exec --user 1001:1002 php ./vendor/bin/behat -v --suite=api

            # This line is to be able to access failing screenshots in the
            # artifacts section in CircleCI
            - store_artifacts:
                path: ~/repo/features/fail-screenshots

            - store_artifacts:
                path: ~/repo/var/logs

            - store_artifacts:
                path: ~/repo/features/spooled-emails
2

Success and failure depends strictly on exit code.

Exit code 0 = success (green)
Any other exit code = failure (red)

It looks like you are executing tests with:

run: docker-compose exec --user 1001:1002 php ./vendor/bin/behat -v --suite=back

You need to make sure that failures cause the test runner to exit with a non-zero exit code.