Incomplete PHPUnit test error build

Running Tests
#1

We’re running phpunit tests, some of which we’re marked incomplete. https://phpunit.de/manual/6.5/en/incomplete-and-skipped-tests.html
What we want, is to let the build process continue, but instead it fails: Exited with code 255, which I think means the exit status is out of range (or -1).

I think a possible solution is to get php to properly report errors, I’m just not sure how to do that. I have tried this in my phpunti.xml:

<php>
<ini name="display_errors" value="true"/>
</php>

However, I’m getting no output on the console from the tests.
(When it’s properly set up phpunit spits out some feedback like: .....I.....F..... )

#2

I don’t think incomplete PHPUnit tests are an error condition in CircleCI. I use them all the time, and our tests always return an exit status of 0 (successful).

The best approach might be to get an SSH session to run tests by file or by class, to narrow down which one is causing the failure.

#3

That’s good advice, but the SSH sessions aren’t working! I have a ticket in for that issue.

Question: when you run phpunit tests, are you getting output like you might on a local install (ie. the test output?) I’m getting nothing, and I’m wondering if that’s the issue, like php is outputting something that unix is seeing as “out of range,” thus the 255 error.

I’ll be able to trace it back better when I have SSH back, I’m just wondering if you’ve had that output to the console.

#4

Do you mean errors for failed tests? Yes, it does stack traces. For a recent failed build:

#!/bin/sh -eo pipefail
docker-compose \
    exec \
    webserver \
    php vendor/bin/phpunit

^@^@PHPUnit 7.5.2 by Sebastian Bergmann and contributors.

EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE  63 / 224 ( 28%)
EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE.................... 126 / 224 ( 56%)
.............................................I................. 189 / 224 ( 84%)
......................II...........                             224 / 224 (100%)

Time: 3.23 seconds, Memory: 16.00MB

There were 106 errors:

1) Maverick\Test\Api\AutoCompleteDiffTest::testOneWordCompletion
Dotenv\Exception\InvalidPathException: Unable to read the environment file at /var/www/html/.env.

/var/www/html/vendor/vlucas/phpdotenv/src/Loader.php:105
/var/www/html/vendor/vlucas/phpdotenv/src/Loader.php:82
/var/www/html/vendor/vlucas/phpdotenv/src/Dotenv.php:106
/var/www/html/vendor/vlucas/phpdotenv/src/Dotenv.php:50
/var/www/html/bootstrap/classes/AppBootstrap.php:80
/var/www/html/bootstrap/classes/AppBootstrap.php:47
/var/www/html/tests/base/ApiTestCase.php:22
/var/www/html/vendor/laravel/lumen-framework/src/Testing/TestCase.php:56
/var/www/html/vendor/laravel/lumen-framework/src/Testing/TestCase.php:73
/var/www/html/tests/api/AutoCompleteDiffTest.php:18

(many more lines)

However there is no output for successful, skipped or incomplete tests. A good run looks like this:

#!/bin/sh -eo pipefail
docker-compose \
    exec \
    webserver \
    php vendor/bin/phpunit

^@^@PHPUnit 7.5.2 by Sebastian Bergmann and contributors.

...............................................................  63 / 224 ( 28%)
..........................SS..S..........I..................... 126 / 224 ( 56%)
.............................................I................. 189 / 224 ( 84%)
......................II...........                             224 / 224 (100%)

Time: 9.92 seconds, Memory: 18.00MB

OK, but incomplete, skipped, or risky tests!
Tests: 224, Assertions: 586, Skipped: 3, Incomplete: 4.

If you are without SSH, try branching your work, cut your tests down to a single case, and push that. Get a successful build, then re-enable ten tests at a time, until you can isolate the problematic test.

It may be worth stripping out any funky options in phpunit.xml and running it as bare-bones as you can until you know what the cause of the issue is.

#5

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.