Jobs complete but the CircleCI status times out

Feedback & Bug Reports
,
1

I’m having issues with CircleCI jobs running to completion, but Circle doesn’t seem to know they’re finished. Here’s an example: https://circleci.com/gh/PrefectHQ/prefect/18557

This seems to only occur for my Python 3.7 job, but I don’t want to read too much into that.

As you can see in the output, pytest runs and reports its test summary, but for some reason CircleCI is still “waiting for output”. This seems to be happening more and more frequently for us, and I’m not sure how to even begin debugging this.

Any insights / help would be appreciated - thanks!

2

Perhaps the tests create new processes in memory, and CircleCI is waiting for them to close down?

You could temporarily force your tests to look like they have passed pytest --cov=prefect . || true and then have a separate run step ps aux to see what processes are in memory.

Is it intermittent with Python 3.7, or does it happen every time with that version?

3

Thanks for the response! It is intermittent, but seems to happen about 50% of the time. The ps aux idea is a good one, I’ll give that a shot!

4

It appears that adding || true prevents this behavior from happening… it looks weird but that makes me wonder if I should just do pytest --cov=prefect . || false…

Oh and for the record, this was the output of ps aux everytime:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0   4280   716 ?        Ss+  18:07   0:00 /bin/sh
root         7  0.5  0.0 123848 20048 ?        Ssl  18:07   0:00 /bin/circleci-a
root      3442  0.0  0.0  38388  3116 pts/0    Rs+  18:10   0:00 ps aux
5

Update: doing pytest --cov=prefect . || false didn’t work, job still times out (e.g., https://circleci.com/gh/PrefectHQ/prefect/18752?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link).

It seems then that pytest is failing to register an exit code for some reason related to CircleCI (I can’t reproduce this inside a python:3.7 docker container).