Test output never expands to show full log


#1

On most of the builds for my project, Hypothesis, if the tests fail then the test entry never expands so I can’t actually see what the failure was. Consider e.g. this one, but it seems to be consistent across builds for the project.

The relevant section is scripts/run-circle.sh. Attempting to expand it just results in an endless loading logo.

Additional notes:

  • I’d swear this used to work, but I couldn’t tell you how long ago that was.
  • I’ve verified this running under both Chrome and Edge, so it’s probably not a brbowser specific bug.
  • The test output is probably quite large.

#2

Mmm. I smell browser bug.

I just tried to open this build log in macOS on two browsers. In Safari, it opened almost immediately. In Chrome, I’m getting the spinning grey circle.

Full test output is here, all 3582 lines of it: https://gist.github.com/alexwlchan/040c8a4bad0ca9a91635fed26cecde2d


#3

I’ve added a quick tally of characters in the output to that Gist. Mostly vanilla ASCII, but then a small number of special characters. I wonder if the \x08 is throwing off Chrome and Edge?


#4

Could be!

As a data point, I think if it is a browser bug it’s a performance one. Sometimes on the same build I see it opening eventually (after say about a minute), sometimes it keeps spinning for ever.


#5

Hi David,

It looks like pytest is not outputting junit XML, so out tests results tab has no data to work with (we don’t parse command output, we reply on test metadata grok test results).

I think you need to specify --junitxml= when running pytest in your tox.ini, but I’m not too familiar with python. Essentially you need to put junit-style XML into $CIRCLE_TEST_REPORTS, and we will parse it. Ideally you want to put XML files into $CIRCLE_TEST_REPORTS/<test-suite-name>/<tests-name>.xml, so that we can show the <test-suite-name> in the UI.

Let me know if this helps! We are big fans of Hypothosis here, so we really want to keep you using CircleCI!

Marc


#6

I don’t think we’ve ever intentionally output junit XML! The command being run is just a bash script where we expect to see the normal console output from the script.

It certainly used to work that it would just display the output in that tab, and sometimes it still does, but most of the time it just hangs as described.

We could try switching over to junit XML, but I don’t actually know how well that interacts with pytest’s more general reporting mechanisms. I’d rather keep this consistent with the way we do other CI if we can.


#7

Looking at your builds, the only time I see you having test metadata for that project was when you were using the inferred nosetests command at the very beginning of your time with us.

If you can provide a build that displays, we can look into it, but if you check the Collecting test summery step at the end of the build you will see it has no results, which is why that tab doesn’t expand, we have nothing to parse.

You can read more about what we require to generate that summery here https://circleci.com/docs/1.0/test-metadata/


#8

Hi @drazisil, I think maybe we’re talking at slight cross purposes.

I agree we’ve never had test summaries, but that’s not what the issue I’m asking about is - we might look in to getting test summary output now that we know it’s a thing, but it’s not something we ever particularly expected.

The issue we’re having is instead that we can’t see the output log from scripts/run-circle.sh.


#9

Update: A friend reported that he’d experienced the same problem when his circle login was somehow broken, and that a diagnostic feature of that was that his avatar was also not showing correctly, and that logging out and logging back in again seemed to fix the problem for him.

It turned out, my avatar was also not displaying correctly (it showed what looked like a github default avatar) so I thought I’d try that, and it did indeed seem to fix things. So for the moment I’m unblocked and can see the output again, yay!

This still seems like there’s a bug of some sort there though.


#10