Hi,
We’re using Meteor.js (Node.js) with Chimp and ran into a peculiar issue.
When we’re running our test suite and getting more than 75 failures the CircleCI environment yields a success (notice the green bar in the attached image below).
We could make the following truth table based on three example exit codes, where true positive is returned success when there is no failed test cases.
True positive: Chimp CircleCi exited with code 0
True negative: Chimp CircleCi exited with code 2
False positive: Chimp CircleCi exited with code null
False negative: We haven’t seen any yet.
The core issue of null is most likely due to a crash in Chimp caused by limited amount of memory in CircleCI.
As anything but 0 should yield a failure according to the documentation, we guess that null is evaluated as 0 and thus yielding a success.
Failing commands (those with a non-zero exit code) will cause the whole build to fail, and you’ll receive a notification.
Is this intentional behavior or a bug?
If this is intentional behavior, does anyone have any suggestion on how we can re-configure or otherwise circumvent the issue?
Thanks in advance!
Cheers,
Christian
