Hey, it seems to work nicely, I can re-run my failed Cypress tests, however I see this error in the logs right after Cypress results:
WARN[2023-05-16T12:10:36Z] There was an error when uploading test results from with batchID 3c53b247-4b13-4b63-ba9d-2f41dca214e3: no path given
DEBU[2023-05-16T12:10:36Z] Error encountered with test batch 3c53b247-4b13-4b63-ba9d-2f41dca214e3
INFO[2023-05-16T12:10:36Z] ending execution
hey @villelahdenvuo I am trying to get this working for our cypress tests but hitting some issues.
We currently use the circleci test split command but when trying to change that to use tests run instead as per docs we get errors. Can't run because no spec files were found.
Do you mind sharing an example of your setup?
If you’d be wiling to share it, @villelahdenvuo, can you paste the command you’re using with the circleci tests run tool? (obscure whatever path or project name information you’d like)
I think you could have the result you’re looking for (the list of values sent to the -spec flag comma separated) by using the tr translate tool, like so:
Removing the initial echo command, and having the contents of the --command flag be --command="tr ' ' ',' | xargs -I {} cypress run -spec {}".
The tool currently expects input to the tool to be space or newline delimited, but how it’s assembled or arranged for the specific test runner you’re using is very flexible, depending on your comfort and familiarity with built-in bash tools.
This is a great feature. Is there any way to determine within the job that it was started via “Re-run failed tests only”? i.e. an environment variable?
This is a huge feature for us! I implemented it according to Re-run failed tests only (preview) - CircleCI but it didn’t work. That page doesn’t mention closed beta. Is there a way to get into the beta?
There’s a small issue with the docs for Cypress
circleci tests glob "cypress/**/*.cy.js" | circleci tests run --command="xargs npx cypress run --reporter cypress-circleci-reporter --spec" --verbose --split-by=timings"
That closing " shouldn’t be there
Also, should the tests be comma separated as before for test splitting?
Thanks @sebastian-lerner, emailed you.
If we don’t do comma separation for the tests we get the following warning
This will work, but it's not recommended.
If you are trying to pass multiple arguments, separate them with commas instead:
cypress run --spec arg1,arg2,arg3
We’ve been okay just ignoring the warnings from Cypress when we were doing internal testing. A user above has also mentioned that it works fine for them with ignoring the warnings.
I’ll look more deeply with my team to see if there’s anything we can do in the long-run to avoid that warning, but for now I think, it’ll be okay.
Have you got any ideas brewing about how you’d use that information, @bbrinck?
If you can share those use cases, that could help inform how we’re building out the tool.
Our intention is that the workflow/job config should be pretty agnostic about whether an initial run or rerun with failed is happening.
That is, a rerun would have the same test runner and options used as the initial run, only with a subset of tests.
@villelahdenvuo that’s what we currently do but because new examples didn’t have it I was wondering if having commas is actually wrong as far as Circle is concerned.
If you have parallelization enabled, every server will spin up even though you might have a single broken test.
If you trigger ‘rerun from failed’ on one test job it will trigger other test jobs with failed tests even if those jobs aren’t set up for it according to the docs.
@vlad for your first one, this is indeed expected behavior at this time as noted in the FAQs. We’re looking at ways to avoid spinning up additional VMs/containers, however for the vast majority of users we still see a substantial reduction in runtime & credits compared to rerunning all tests.
For the second item, can you send over an example workflow where you’re seeing this behavior? Are the other test jobs downstream in the workflow or run concurrently?