This is a great feature. Is there any way to determine within the job that it was started via “Re-run failed tests only”? i.e. an environment variable?
1 Like
@bbrinck no official way at this time. We’ve had a couple of users leverage this workaround: Detect Rerun of a job
This is a huge feature for us! I implemented it according to Re-run failed tests only (preview) - CircleCI but it didn’t work. That page doesn’t mention closed beta. Is there a way to get into the beta?
There’s a small issue with the docs for Cypress
circleci tests glob "cypress/**/*.cy.js" | circleci tests run --command="xargs npx cypress run --reporter cypress-circleci-reporter --spec" --verbose --split-by=timings"
That closing " shouldn’t be there
Also, should the tests be comma separated as before for test splitting?
Good spot on the docs typo. Opened a PR to make that change, thank you.
The test files should be delimited by a space, that’s the default output of the circleci tests glob command. Is that what you’re asking?
If you email me your org name, I can add you to the closed beta in the next couple of business days. sebastian@circleci.com
Thanks @sebastian-lerner, emailed you.
If we don’t do comma separation for the tests we get the following warning
This will work, but it's not recommended.
If you are trying to pass multiple arguments, separate them with commas instead:
cypress run --spec arg1,arg2,arg3
Should we just ignore this warning?
Email received, thank you.
We’ve been okay just ignoring the warnings from Cypress when we were doing internal testing. A user above has also mentioned that it works fine for them with ignoring the warnings.
I’ll look more deeply with my team to see if there’s anything we can do in the long-run to avoid that warning, but for now I think, it’ll be okay.
See this reply, with some bash magic it can be converted to comma separated list: [Product Launch] Re-run Failed Tests Only (circleci tests run) - #18 by chadchabot
Have you got any ideas brewing about how you’d use that information, @bbrinck?
If you can share those use cases, that could help inform how we’re building out the tool.
Our intention is that the workflow/job config should be pretty agnostic about whether an initial run or rerun with failed is happening.
That is, a rerun would have the same test runner and options used as the initial run, only with a subset of tests.
@villelahdenvuo that’s what we currently do but because new examples didn’t have it I was wondering if having commas is actually wrong as far as Circle is concerned.
Found two possible issues -
If you have parallelization enabled, every server will spin up even though you might have a single broken test.
If you trigger ‘rerun from failed’ on one test job it will trigger other test jobs with failed tests even if those jobs aren’t set up for it according to the docs.
@vlad for your first one, this is indeed expected behavior at this time as noted in the FAQs. We’re looking at ways to avoid spinning up additional VMs/containers, however for the vast majority of users we still see a substantial reduction in runtime & credits compared to rerunning all tests.
For the second item, can you send over an example workflow where you’re seeing this behavior? Are the other test jobs downstream in the workflow or run concurrently?
Apologies, didn’t see the FAQ. Yes, there’s definitely savings even with extra containers running. It’s just wasteful. If we have a long restore cache job and have a bunch of scaffolding then we spend a few minutes for nothing. Is there a way to detect what tests will run in the current instance? If there is then we can short-circuit the flow to exit if there are no tests.
About the second item, the jobs run concurrently, they are not downstream. They do have a common ancestor though. Here’s what it looks like. If I go to Cypress job, click ‘Rerun failed tests only’ then I expect it to only run the Cypress job but it also runs the playwright-e2e job.
In theory, I think yes. I don’t have any sample code yet on how to actually achieve that. You can probably write a step that is the first thing that executes that calls circleci tests run and instead of actually passing in the test runner to the --command to actually execute tests, you could echo or redirect that output to a file. And then check to see if that file is empty, don’t run any other steps.
Thanks for the workflow diagram, super helpful. We’re iterating on the behavior for this exact scenario actually. “Rerun failed tests only” is meant to be effectively equivalent to “Rerun workflow from failed”. So if there is a test failure in cypress-build, I would expect the failed tests from cypress-build to be re-run in the newly generated workflow. At the same time, playwright-e2e would also be re-run concurrently in the newly generated workflow, but if it’s not set up with circleci tests run, it should re-run all tests. The workflow would then proceed to downstream jobs if applicable. Concerns with that design?
It’s not urgent, but our primary use case is for code coverage. We save code coverage data when we run specs. If we run only a small subset of specs, that will save code coverage just for those specs, which messes up downstream tooling.
If there was, say, an environment variable that we could inspect, we’d just skip saving code coverage for these runs.
Right now, we’ve worked around this by adjusting our downstream tooling to notice code coverage data that is well below what we normally expect.
No issues at all except that ‘rerun from failed’ command is at the job level instead of workflow level which makes it confusing. If it was at the workflow level then rerunning all bad tests over all jobs makes sense. If I have to go to a job and click ‘rerun’ then it definitely seems like I’m just doing that for that one particular job.
I just tried to do something like this but interestingly enough on a full test run nothings is returned on that step
#!/bin/bash -eo pipefail
circleci tests glob "integration/**/*.cy.js" | circleci tests run --command "xargs echo" --verbose --split-by=timings > /tmp/test.output cat /tmp/test.output
DEBU[2023-05-31T01:46:53Z] Attempting to read from stdin. This will hang if no input is provided.
WARN[2023-05-31T01:46:53Z] No test names found in input source. If you were expecting test names, please check your input source.
INFO[2023-05-31T01:46:53Z] starting execution
DEBU[2023-05-31T01:46:53Z] There were no tests found in the bucket for node 8
INFO[2023-05-31T01:46:53Z] ending execution
CircleCI received exit code 0
on a ‘re-run failed tests’ we do get the tests
#!/bin/bash -eo pipefail
circleci tests glob "integration/**/*.cy.js" | circleci tests run --command "xargs echo" --verbose --split-by=timings > /tmp/test.output cat /tmp/test.output
DEBU[2023-05-31T01:56:55Z] Attempting to read from stdin. This will hang if no input is provided.
WARN[2023-05-31T01:56:55Z] No test names found in input source. If you were expecting test names, please check your input source.
DEBU[2023-05-31T01:56:55Z] rerunning failed tests
INFO[2023-05-31T01:56:55Z] starting execution
DEBU[2023-05-31T01:56:55Z] received test names: integration/retail/dashboard_manager.cy.js
DEBU[2023-05-31T01:56:55Z] Completed tests
INFO[2023-05-31T01:56:55Z] ending execution
CircleCI received exit code 0
Is there more documentation on how this all works? I don’t quite understand why this is happening.
@vlad I don’t have a diagram unfortunately, but that is something I can work on publishing, good suggestion.
I’ve just pushed a PR to the docs (it might take a few hours to load) that shows an example of how to do this which we’ve been able to use successfully on a Ruby project. It’s very similar to what you have. Is it possible that when you see:
“DEBU[2023-05-31T01:46:53Z] There were no tests found in the bucket for node 8”
you are actually looking at a parallel run that was not distributed any tests? From the UI, you can click on the boxes that represent the different parallel runs. Are you seeing the same “there were no tests found” in all of the parallel runs? When you use --split-by=timings, it’ll split tests across as many nodes as it needs. So if you’re getting a node that’s outputting “no tests found in the bucket for node 8”, it’s possible you can reduce the parallelism level.
@sebastian-lerner not sure what you mean by the diagram.
In the new doc PR you are saying to run this command for a list of tests -
circleci tests run --command "xargs echo" --verbose --split-by=timings > files.txt #split-by=timings is optional
As far as I understand that command just filters and runs the tests passed to it. It returns nothing by itself. It can’t know what tests to run.
For example, this is what we would do for rspec
circleci tests glob "spec/**/*_spec.rb" | circleci tests run --command "xargs echo" --verbose --split-by=timings > /tmp/tests.txt
First command would glob the files and then test run would filter and run them.
I did get that working but the problem is that when you run circleci tests command for the first time it outputs the following
Installing circleci-tests-plugin-cli plugin.
circleci-tests-plugin-cli plugin Installed. Version: 1.0.6154-5ff1072
That will go into your piped file.
I thought I would be clever and ‘pre-install’ it first by running a command and then running it again with a pipe but the command outputs
Skipping installation. circleci-tests-plugin-cli plugin already installed.
Is there any way to disable that output to get a clean file of tests or an empty file if there are no tests?
Another thing I noticed is that you have to wait until the workflow finishes to re-run failed tests. If that’s the case then why have re-run at the job level instead of a workflow level? You can’t re-run failed tests in a single job nor can you trigger them until the whole workflow fails. If you can only re-run from failed for the whole workflow and you have to wait for it to finish why not have it in the workflow dropdown?
Appreciate all your help!
A clarification, circleci tests run does not actually run tests by itself. All it does is execute what is in the --command parameter. So if you don’t include a command to run tests in --command, no tests are actually run when you invoke circleci tests run.
I’m checking with the team as to why you’re getting that output piped to the file…we weren’t seeing that in our internal testing. Let me get back to you.
Regarding the placement of the button in the UI to actually re-run the workflow, we are planning on adding the option to the workflow dropdown. You have a fair point, good feedback thank you.
@vlad can you try moving the piping inside of the --command.
circleci tests glob "spec/**/*_spec.rb" | --command ">files.txt xargs echo" --verbose --split-by=timings
--verbose is optional btw. But the above command will give you the test file names in files.txt without any of the “cli installed” output. We’re seeing that behavior in our internal testing.