We are running tests in parallel using the circleci cli. This is working great as long as the tests complete without crashing. But if there is a syntax error in any of the tests we simply get the output “Exited with code exit status 2”. But when I run pytest locally I get much more useful output that looks very similar to pytests usual output, saying that test failed and printing the error message. I’m guessing that “circleci tests run” is not gracefully catching the error and output, where as pytest run locally does. How can I get more useful output when a test crashes? Below is some code:
in config.yaml:
- run:
name: Run tests
command: |
cd python
# gather test filenames
TEST_FILES=$(poetry run pytest --collect-only -p no:warnings -q tests/unit 2> /dev/null | awk -F '::' '{print $1}' | sort | uniq | tail -n +3)
# Below we're generating junit xml to feed to circle for test timing, which is how
# the test splitting can do so by timing. We use the -n 4 because the auto detect
# "logical" arg doesn't work in CircleCI infra.
# We use the "circleci tests run" to allow us to rerun failed tests, but this appears to slow it down by about 1 min.
echo "$TEST_FILES" | circleci tests run --command="xargs poetry run pytest --disable-socket --junitxml=test-results/junit.xml --override-ini junit_family=xunit1 --cov=merq --cov-report= -n 4" --verbose --split-by=timings
[ -f .coverage ] && mv .coverage .coverage.unit.$CIRCLE_NODE_INDEX
The command we use locally looks more like this:
poetry run pytest tests/unit
Thanks in advance!