Test Failures tab reports false "0 failures" for Cucumber

Build Environment
1

Even though Container 14 shows a failure, the tab itself says “Your build ran 7163 tests in RSpec, Cucumber with 0 failures”…

https://circleci.com/gh/spreemo/chewy/11170.

2

It failed in just one container? Would it be possible to share the relevant part of your circleci.yml file?

Are you doing anything differently in that container?

3

That’s right, only one container failed with that single failure.

We aren’t doing anything different, far as I can tell, but here’s the complete config file:

dependencies:
  pre:
    - clamscan -V || sudo apt-get install clamav
    - mysql_tzinfo_to_sql /usr/share/zoneinfo | mysql -u root mysql
    - bash ./headless-chrome.bash
  post:
    - sudo pip install awscli
    - ruby lib/cucumber_rerun_shim.rb
test:
  pre:
    - "[[ ! -s \"$(git rev-parse --git-dir)/shallow\" ]] || git fetch --unshallow":
        parallel: true
  post:
    - COVERALLS_PARALLEL=true bundle exec rake coveralls:push:
        parallel: true
    - case $CIRCLE_NODE_INDEX in 0) bundle exec pronto run -f text -c origin/master --exit-code ;; 1) bundle exec pronto run -f github -c origin/master ;; esac:
        parallel: true
    - case $CIRCLE_NODE_INDEX in 0) bundle exec brakeman --exit-on-warn ;; 1) bundle exec rubocop -D --format offenses --format progress ;; 2) bundle exec rails_best_practices --features --spec ;; esac:
        parallel: true
deployment:
  master:
    branch: master
    commands:
      - 'aws --region="us-east-1" opsworks create-deployment --stack-id="25e2e3d1-2c0f-471d-b8d9-b54d7a6700cd" --app-id="b8b8857d-2646-4620-9d12-c365b9acb25e" --comment="`git log --pretty=format:"%h %ad %an: %s" -1 --date=short | tr -c "[:alnum:]-_():+@?/."  " " | tr -s "[:space:]" " "`" --command="{\"Name\": \"deploy\", \"Args\":{\"migrate\":[\"true\"]}}"':
          timeout: 600
  demo:
    branch: demo
    commands:
      - 'aws --region="us-east-1" opsworks create-deployment --stack-id="4e4f5024-a679-4a07-83bf-415ef8e69d2a" --app-id="e924ea06-fa5e-4a56-b813-b3ca3b4402a5" --comment="`git log --pretty=format:"%h %ad %an: %s" -1 --date=short | tr -c "[:alnum:]-_():+@?/."  " " | tr -s "[:space:]" " "`" --command="{\"Name\": \"deploy\", \"Args\":{\"migrate\":[\"true\"]}}"':
          timeout: 600
notify:
  webhooks:
    - url: https://coveralls.io/webhook?repo_token=0xxLY843awMJ1FfMAxo2MHGujlVBVrCjK
general:
  artifacts:
    - 'coverage'
    - 'log/test.log'
    - 'tmp/capybara'
4

Hm. Not seeing anything odd. Is it reproducible, if you retry the build? Wonder if it was just a fluke with the container.

5

Looking back, I think it happens intermittently. Haven’t noticed any pattern as the failures themselves are sometimes intermittent too.

Would tracking down example CI URLs help?

6

It should. However I would have to defer to staff as I can’t see private builds :frowning:

7

We’re seeing a similar situation

8

Additionally, here is our implementation code (sorry for double posting, Discourse says new users can only post 2 links in a given post :grin:).

9