When running dotnet test the suite hangs and continuously prints out “Test run in progress.”
Historically the testing suite has run in under a minute. One specific test job from 6 days ago took 17 second to complete. I reran that same workflow via the Ui and the test job is now taking north of 4 minutes.
If I’m not on the Ui the testing suite has a tendency to timeout, but if I am actively watching the job it generally succeeds after 4 minutes.
Edit: Changed the verbosity of the suite and that “Test run in progress” line comes when actually running a test. However, the time to load a test project up is taking a considerable amount of time.
2 Likes
Hi Andrew. That’s very strange. Can you open a ticket with the job urls and reference this post so we can take a look?
Thanks @drazisil. I’ve submitted a ticket.
1 Like
I’ve just noticed the same issue. In one example,dotnet test runs 427 tests in ~3s on my computer, and until yesterday it took <5s to run in CircleCI. Today I’m seeing 14 minutes in the job step with “Test run in progress.” repeatedly. Hundreds of times.
To make things even more interesting, the dotnet test output reports a run time of 5.3961s, and the CircleCI job step reports a time of 14:49.
This is happening across multiple microservices running dotnet core.
I’ve just confirmed that a complete build of one service (dotnet core still) from 13 days ago that ran in 35s is now exhibiting the same symptoms, doing a rebuild of the previous run: 3:37.
A quick estimate puts it at 1 “Test run in progress” printed every 3s.
Given there have been no changes to these branches that should cause this, or no changes at all, it does appear to be a CircleCI bug.
+1 for this issue.
- netcoreapp2.0
- NUnit 3.10.1, NUnit.Portable.Agent 3.4.0-beta-3, NUnit3TestAdapter 3.10.0
- microsoft/dotnet:2.1-sdk image
- Very simple tests, all POCOs, no external dependencies at all.
Tests complete in 3.3727 seconds locally, same command in Circle CI flooding the build output with “Test run in progress” messages.
I’m not aware of any changes that we made that could cause this, can you please open a ticket so we can track and investigate?
Thanks @drazisil - I’ve opened a ticket.
1 Like
I opened a support ticket yesterday and have yet to hear back. I’ve setup the same commands in the same docker image and it’s working as expected here.
Is there a chance that it’s whatever is probing/piping the bash output from the docker container to the web UI causing the issue?
…does something like that even exist?
More info. I’m able to reproduce the issue when I ssh into a CircleCI build machine.
Example:
root@1d2d03c966cf:/REDACTED# dotnet test tests/REDACTED.csproj
Test run for /REDACTEDi/tests/bin/Debug/netcoreapp2.2/REDACTED.Tests.dll(.NETCoreApp,Version=v2.2)
Microsoft ® Test Execution Command Line Tool Version 16.1.0
Copyright © Microsoft Corporation. All rights reserved.
Starting test execution, please wait…
Test run in progress.
It prints “Test run in progress.” once, stalls for about 45s, then completes with positive results but inaccurate timing.
Test Run Successful.
Total tests: 427
Passed: 427
Total time: 2.4371 Seconds
-=UPDATE=-
The problem is very clearly related to screen output and it’s unique to the CircleCI VM (as far as I can tell).
I can get the tests to execute in ~1 minute when SSH’d in to the box with this:
dotnet test -v n --no-build --no-restore
I can get the tests to execute in ~5s (normal time!) with this:
x=$(dotnet test -v -n); echo $x;
The output is terrible, but it works. Difference in total execution time is ~50s (using -v n to get total exec time).
So… If anyone needs a workaround, that’s it. Hide the output from CircleCI by executing the tests in a subshell.
-=Final Update=-
This is the bash script I’m using for a workaround. Adjust to suit your testing needs… hope it helps.
#!/bin/bash
hack=(dotnet test -v m --no-build --no-restore)
if [ ? -ne 0 ]; then
echo “$hack”
exit 1
fi
echo “$hack”
@JohnJKerr did you get a resolution to this? I’m having the same issue as of a few hours ago.
FYI this is has started happening to my builds as well. I’d open a ticket, but there seems to be a bit of a pattern emerging…
Not had a reply to my ticket yet @nagytech
+1 - I started seeing this last night. It’s happening on my dotnet builds.
The test runner is reporting ~3s but the circleci step is reporting over 2m
This looks like a possible change to the dotnet image, be we are still investigating. Can those affected create tickets if you haven’t so we can link them together and get a scope of the issue?
Hello I am having similar issues for 2 days. Can’t deploy any dotnet app. I tried fiddling with different base images but the build always stuck in the test phase.
@drazisil are you talking about a separate thread? where do I create a ticket?
Hey @mustakim,
I’ve been working with the Circle Team on this and it seems the issue is Microsoft updated their underlying SDK without updating the tags on the docker image.
I was able to fix this by referencing an older build via sha. Hope that helps!
1 Like
It seems to be a problem with the newest image from Microsoft (mcr.microsoft.com/dotnet/core/sdk:2.2.300).
I was able to fix this by referring to the preceding build (2.2.204).
2 Likes
Using an older version of the image fixed the issue for me. Thanks.
Posting for those following this thread.
Has been reported as fixed in 2.2.301 for TravisCI (https://github.com/microsoft/vstest/issues/2078), however, the issue still remains for CircleCI with 2.2.401 (https://github.com/microsoft/vstest/issues/2120)