Step fails but shows as passing

Build Environment Running Tests
,
1

I run integration tests against our nodejs api within a docker container. I just had two separate branches pass all tests even though the integration step actually failed due to db connection issues.

Output says:

integration-1  | Error: unable to connect to database at mongodb://mongo/api-local
integration-1  |     at NativeConnection.<anonymous> (/app/app.js:60:9)
integration-1  |     at NativeConnection.emit (node:events:517:28)
integration-1  |     at /app/node_modules/mongoose/lib/connection.js:835:30
integration-1  |     at process.processTicksAndRejections (node:internal/process/task_queues:77:11)
integration-1  | 
integration-1  | Node.js v18.20.3
integration-1  | npm notice
integration-1  | npm notice New minor version of npm available! 10.7.0 -> 10.8.3
integration-1  | npm notice Changelog: https://github.com/npm/cli/releases/tag/v10.8.3
integration-1  | npm notice To update run: npm install -g npm@10.8.3
integration-1  | npm notice
integration-1 exited with code 7

Step shows as green.

How do I diagnose this? wouldn’t have even noticed if a dev hadn’t been looking at the tests manually.

2

Seems like something is reporting, but swallowing, the exit code. How is the test being run? i.e., what exact command is being run (all the way down, for example, if it’s an npm script target, what is in that line, and so on).

My guess (could be way off base) is that something is wrapping it which either doesn’t have errexit set, has an || true after it, or is otherwise not following the underlying exit status.

By default, any direct command run in a CircleCI run step should fail if the command it’s running exits non-0. However, you do have to take steps to make sure any additional scripts or tools you call are not evergreen themselves.

3

CircleCI config:

version: 2
jobs:  
  test:
    steps:
      - run: make test_integration

Makefile:

test_integration:
	@docker-compose up integration

docker-compose.yml:

integration:
    image: api_server
    entrypoint: /bin/sh
    command: -c 'npm run test_integration'

package.json:

"scripts": {
    "test_integration": "mocha ./test/integration",
4

If you induce a failure (for example, updating an expectation so it fails) locally and run docker-compose up integration, and then run echo $? immediately after, what exit code do you get there?

If you change the entrypoint to ["npm", "run", "test_integration"], and remove the command, does that change anything?

Are there other containers in the integration test docker-compose image? The docs suggest that docker-compose will exit 1 if the underlying command fails (and my quick test indicates that sh -c should also exit with the status code of the command it ran), but I would double-check locally and see if something needs to be adjusted.

Since Circle pipelines already are running in Docker, sometimes there’s a penalty for building / running Docker again. I don’t know how complicated your integration test setup is, but if it’s feasible, one other option to consider might be using accessory containers for any dependencies you have on things like databases, and run the npm command directly as part of your pipeline.