Inconsistent execution between automated test and SSH execution of same command (affecting db access)

My database access seems to be rather sporatic. When my fixture loading, and behat tests are executed by the CI process, they fail with a database error. When I SSH into the machine and run the same testing command, it works.

Test step:

- php bin/behat @CoreBundle

Error message:

[Doctrine\DBAL\Exception\ConnectionException]
An exception occured in driver: SQLSTATE[HY000] [2002] No such file or directory

Circle.yml

machine:
php:
version: 5.6.14
hosts:
www.local.dev: 127.0.0.1

dependencies:
pre:
- sudo apt-get install ant;

cache_directories:
- ~/.composer/cache

post:
- wget http://selenium-release.storage.googleapis.com/2.44/selenium-server-standalone-2.44.0.jar
- java -jar selenium-server-standalone-2.44.0.jar:
background: true
- sleep 10
- sudo service apache2 restart

test:
override:
- php scripts/dbtest.php
- ant setup-config-files
- php app/console system:config:dbdump
- ant load-fixtures
- php bin/behat @CoreBundle
- php app/console doctrine:query:sql “select id from user limit 10”
- php app/console system:config:dbdump

When I ssh into the box. Change directory the PWD of the test script, and execute the same command… my database connection is fine. I have a number of other test steps that access the DB. Strangely, it seems like the first script to access the DB fails, and the 2nd passes.

Is anyone else experiencing anything like this?

Looks like the issue has been mentioned in this StackOverflow question, could you please check if the solution offered there could make sense for you?

The fact that the command runs normally through SSH might mean that there is a timing issue present here, such as database not being ready for some reason when the tests are first run but already running properly when you log in through SSH.

Unfortunately, that link is only about the issues of connecting via socket. I’m connecting to 127.0.0.1.

Either way, my issue is different. As you mentioned it seems to be related to timing, or storage errors.

In the dependencies section I’m creating a file with my db credentials. That file is being erased by the CI process somehow. I’ve tested this by putting in a simple “cat” of the db settings between test steps. Sure enough… the file is disappearing before running my last test step. I tried to upload a screenshot, but it’s not working. Either way, I can clearly see the credentials file is just gone.

The reason the database error is ‘no such file or directory’ is because the code is falling back on trying to connect to the database via unix_socket, instead of credentials.

Can someone elaborate on what happens between test steps?

BTW… there were not reported system outages when this happened.

I don’t think this is related to the outage.

We shouldn’t be erasing the file on our side. Does the database step have any inferred commands in your build? Maybe one of those is doing something unexpected.

Does the CI system do anything between test steps?

My next troubleshooting step is to compare all environment variables that differ between between running tests from an SSH session, and when the automated test suite is running.

Any other recommendations?