I’m having issues with CircleCI jobs running to completion, but Circle doesn’t seem to know they’re finished. Here’s an example: https://circleci.com/gh/PrefectHQ/prefect/18557
This seems to only occur for my Python 3.7 job, but I don’t want to read too much into that.
As you can see in the output,
pytest runs and reports its test summary, but for some reason CircleCI is still “waiting for output”. This seems to be happening more and more frequently for us, and I’m not sure how to even begin debugging this.
Any insights / help would be appreciated - thanks!
Perhaps the tests create new processes in memory, and CircleCI is waiting for them to close down?
You could temporarily force your tests to look like they have passed
pytest --cov=prefect . || true and then have a separate run step
ps aux to see what processes are in memory.
Is it intermittent with Python 3.7, or does it happen every time with that version?
Thanks for the response! It is intermittent, but seems to happen about 50% of the time. The
ps aux idea is a good one, I’ll give that a shot!
It appears that adding
|| true prevents this behavior from happening… it looks weird but that makes me wonder if I should just do
pytest --cov=prefect . || false…
Oh and for the record, this was the output of
ps aux everytime:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 4280 716 ? Ss+ 18:07 0:00 /bin/sh
root 7 0.5 0.0 123848 20048 ? Ssl 18:07 0:00 /bin/circleci-a
root 3442 0.0 0.0 38388 3116 pts/0 Rs+ 18:10 0:00 ps aux
pytest --cov=prefect . || false didn’t work, job still times out (e.g., https://circleci.com/gh/PrefectHQ/prefect/18752?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link).
It seems then that
pytest is failing to register an exit code for some reason related to CircleCI (I can’t reproduce this inside a
python:3.7 docker container).