We’ve been successfully using this feature for our Jest workflow for the past month or so, but in the last couple of days have started running into one of our machines in a parallel workflow successfully testing all the Jest tests, but then failing with a 123 status code. It was intermittent at first, but is now happening on every run.
Command: yarn --silent test --listTests | circleci tests run --command="xargs yarn test-ci"
Where: "test-ci": "yarn test --ci --runInBand" and "test": "cross-env NODE_ENV=test NODE_NO_WARNINGS=1 TZ=UTC jest"
Result:
Test Suites: 120 passed, 120 total
Tests: 1043 passed, 1043 total
Snapshots: 0 total
Time: 297.126 s
Ran all test suites matching [redacted test paths] in 2 projects.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
Error: exit status 123
Error: exit status 123
Exited with code exit status 1
Can you advise on how we could debug this further to determine exactly what the error is?
Is there a way to add a rerun button on the main dashboard? Now it’s required to open every workflow to rerun failed tests while rerun doesn’t do it by default.
Submitted a feature request for this - ideas .circleci .com/cloud-feature-requests/p/test-rerun-ui-button-on-job-dashboard
@benedfit I believe an exit status of 123 just means a non-zero exit code was returned, which is expected if there are legitimate test failures. Are you seeing this exit code even when there are no test failures?
@unickq Thanks for submitting the feature request. No timeline to share at this time for when we’ll add the re-run option to the main dashboard, but it’s something that is under consideration.
I am trying to rerun the failed test cases but it seems to be running all the test cases again. I have tried every troubleshooting step but it doesn’t seems to be working. below is the command that I am using in my config.yml file to rerun the failed test cases .
My project is generating the junit.xml report in which I can see the className attribute and it seems to be reading the right info as mentioned below from my console output:
INFO[2023-09-08T13:19:55-04:00] received failed tests from workflow 8065b347-f218-4105-bf28-9ca7963bf324
INFO[2023-09-08T13:19:55-04:00] 2 test(s) failed out of 3 total tests. Rerunning tests from 1 classname
DEBU[2023-09-08T13:19:55-04:00] if all tests are being run instead of only failed tests, ensure your JUnit XML has a file or classname attribute.
ERRO[2023-09-08T13:19:55-04:00] There was an error reading historical timing data: file does not exist
ERRO[2023-09-08T13:19:55-04:00] Requested weighting by historical based timing, but they are not present. Falling back to weighting by name.
DEBU[2023-09-08T13:19:55-04:00] starting execution
DEBU[2023-09-08T13:19:55-04:00] Received: test afterSignInPage
It’s receiving the right feature file in which my test scenario is getting failed but scenarios from other feature files are also running . please help asap. any leads are appreciated. I have tried storing test results as well as uploading junit.xml report as an artifact . still doesn’t work.
And please include whether your job has ever used CircleCI’s test splitting before or not. If yes, please include a link to a job that used circleci tests split
Hey, this feature is really great and important for us, so thank you!.
I made config changes but on our jest test for the backend, if a test failing we are getting the following message:
Installing circleci-tests-plugin-cli plugin.
circleci-tests-plugin-cli plugin Installed. Version: 1.0.6868-921e28c
INFO[2023-09-19T09:47:47Z] received failed tests from workflow 51202872-e133-406f-85fb-ff38f645f8ee
INFO[2023-09-19T09:47:47Z] 2 test(s) failed out of 6600 total tests. Rerunning tests from 1 classname
DEBU[2023-09-19T09:47:47Z] if all tests are being run instead of only failed tests, ensure your JUnit XML has a file or classname attribute.
DEBU[2023-09-19T09:47:47Z] There was an error when sorting the test names: could not find size of file sync should create a provider update to update a user for the matching organisation if an eid matches with changes: stat sync should create a provider update to update a user for the matching organisation if an eid matches with changes: no such file or directory
Error: could not find size of file sync should create a provider update to update a user for the matching organisation if an eid matches with changes: stat sync should create a provider update to update a user for the matching organisation if an eid matches with changes: no such file or directory
ERRO[2023-09-19T09:47:47Z] could not find size of file sync should create a provider update to update a user for the matching organisation if an eid matches with changes: stat sync should create a provider update to update a user for the matching organisation if an eid matches with changes: no such file or directory
Error: exit status 1
and this is the setup of the step:
npx jest --listTests | circleci tests run --command="xargs yarn test:ci" --verbose --split-by=filesize
What i found strange that the first run everything is working as expected ( tests are running) then we have failure on flaky tests and then on the rerun this happens.
Hi! We have the same Jest setup step and the same issue that @ODVault is having. As in the first run works run and everything, but it cannot find the re-run test file(s) to work with.
Installing circleci-tests-plugin-cli plugin.
circleci-tests-plugin-cli plugin Installed. Version: 1.0.6868-921e28c
INFO[2023-09-19T09:49:31Z] received failed tests from workflow 05b3aa92-3593-4439-9d49-a5feba0e37cd
INFO[2023-09-19T09:49:31Z] 1 test(s) failed out of 25487 total tests. Rerunning tests from 1 file
DEBU[2023-09-19T09:49:31Z] if all tests are being run instead of only failed tests, ensure your JUnit XML has a file or classname attribute.
DEBU[2023-09-19T09:49:31Z] There was an error when sorting the test names: could not find size of file src/modules/name/components/__tests__/integration.spec.tsx: stat src/modules/name/components/__tests__/integration.spec.tsx: no such file or directory
Error: could not find size of file src/modules/name/components/__tests__/integration.spec.tsx: stat src/modules/name/components/__tests__/integration.spec.tsx: no such file or directory
ERRO[2023-09-19T09:49:31Z] could not find size of file src/modules/name/components/__tests__/integration.spec.tsx: stat src/modules/name/components/__tests__/integration.spec.tsx: no such file or directory
Error: exit status 1
Exited with code exit status 1
and the command for the test job is:
npx jest --listTests | circleci tests run --command="xargs yarn test:ci" --verbose --split-by=filesize
My personal speculation is it has the re-run path of
@ngvuthanhnhan can you make sure that for Jest the JUnit XML is being populated with the file attribute? Rerun failed tests - CircleCI shows a couple of options on how to do that. The feature just reads directly from the JUnit XML so if the JUnit XML is accurate, then the feature should work (and vice versa).
@ODVault can you check to make sure that there is indeed a file size being populated in your JUnit XML? You can use a store_artifacts command to upload the XML to CircleCI and inspect it manually.
@ngvuthanhnhan i think actually you may have the same root cause as ODVault below. In order to split by filesize, the XML needs to have the file attribute I believe. Can you make sure that you’re using the mechanism to add in the file attribute for Jest like is shown in the docs: Rerun failed tests - CircleCI
and it successfully stored the test results correctly in the first run:
Archiving the following test results
* /home/circleci/app/apps/packages/test_reports/jest/results-0.xml
Total size uploaded: 52 KiB
Having said that, CircleCI still cannot the size of the file(s) it is supposed to work with.
Installing circleci-tests-plugin-cli plugin.
circleci-tests-plugin-cli plugin Installed. Version: 1.0.6868-921e28c
INFO[2023-09-19T09:49:31Z] received failed tests from workflow 05b3aa92-3593-4439-9d49-a5feba0e37cd
INFO[2023-09-19T09:49:31Z] 1 test(s) failed out of 25487 total tests. Rerunning tests from 1 file
DEBU[2023-09-19T09:49:31Z] if all tests are being run instead of only failed tests, ensure your JUnit XML has a file or classname attribute.
DEBU[2023-09-19T09:49:31Z] There was an error when sorting the test names: could not find size of file src/modules/name/components/__tests__/integration.spec.tsx: stat src/modules/name/components/__tests__/integration.spec.tsx: no such file or directory
Error: could not find size of file src/modules/name/components/__tests__/integration.spec.tsx: stat src/modules/name/components/__tests__/integration.spec.tsx: no such file or directory
ERRO[2023-09-19T09:49:31Z] could not find size of file src/modules/name/components/__tests__/integration.spec.tsx: stat src/modules/name/components/__tests__/integration.spec.tsx: no such file or directory
Error: exit status 1
Exited with code exit status 1
Probs I’m missing a store_artifacts command to upload the XML to CircleCI or for me to inspect it in case it has this errors again, so fingers crossed…
Hi @sebastian-lerner , thanks for the speedy reply!
So this is the XML output example (suite and case aren’t related):
<testsuite name="Test Routing component" errors="0" failures="0" skipped="0" timestamp="2023-09-19T17:08:53" time="33.139" tests="24">
<testcase classname="Upload Component Integration › invites includes users with a termination date in the past" name="Upload Component Integration › invites includes users with a termination date in the past" time="0.022" file="src/test/components/UploadIntegration.test.tsx">
</testcase>`
So if I understand correctly a filesize attribute should be filled in order to use the --split-by=filesize?
if so, how does one do that? I didnt find on the jest-junit documentation
@sebastian-lerner So from the debug line seems you expecting a file size
DEBU[2023-09-19T16:16:17Z] There was an error when sorting the test names: could not find size of file shows correct placeholder for non-assignee: stat shows correct placeholder for non-assignee: no such file or directory
But adding the addFileAttribute just add a file attribute ( which does exists in the XML) and it works correctly.
Hey @benedfit - did you ever find a fix for your issue? We’re seeing something similar on the workflows for our Rails app. Tests all pass, but the step says it exited with a non-zero exit code and it fails.
@sebastian-lerner configuration it shows in the circleci documentation to rerun the failed test cases matches the output in my console as mentioned below:
Installing circleci-tests-plugin-cli plugin.
circleci-tests-plugin-cli plugin Installed. Version: 1.0.6868-921e28c
INFO[2023-09-20T18:05:38-04:00] received failed tests from workflow 12922603-926c-4d1a-aa72-edde510cd8de
INFO[2023-09-20T18:05:38-04:00] 1 test(s) failed out of 2 total tests. Rerunning tests from 1 classname
DEBU[2023-09-20T18:05:38-04:00] if all tests are being run instead of only failed tests, ensure your JUnit XML has a file or classname attribute.
DEBU[2023-09-20T18:05:38-04:00] starting execution
DEBU[2023-09-20T18:05:38-04:00] Received: test afterSignInPage
but it wont run just the failed test cases . please help me find a solution. I tried everything.
@ngvuthanhnhan sorry for the delayed response, I think I know what might be going on. In your --command="", can you add an xargs at the beginning please? It’s easy to miss in the the Jest docs, but we need xargs present. I’ll try to clean up the docs. Let me know the results after that.
@ODVault interesting…I talked with my team and I was wrong, I don’t think you need a file size in the JUnit XML. I believe we calculate the size on our end and then do the splitting…I think your original speculation of the paths being incorrect is what is causing this. I haven’t seen this issue with a jest set-up before, is there anything else going on in your config.yml where you change the directories or environments? Maybe could you send me a link to a job at sebastian@circleci.com as well please?
@bcroesch Is this happening intermittently or consistently? If you are seeing it regularly, could you send me an example job link to sebastian@circleci.com so I can make sure we don’t have an underlying bug with the feature.
@pratiksandhu let’s keep iterating over the email thread that we have.
I attempted to integrate re-run failed test feature into my workflow, but I have not achieved success yet. The error code I encountered is similar to the one described in this post. Here is my configuration:
@ducminhleeh What happens if you move --reporters=default --reporters=jest-junit into your yarn test:ci command? So that yarn test:ci resolves to:
jest --reporters=default --reporters=jest-junit --ci --maxWorkers=2 --silent --forceExit
In your linked post, the solution was to move the --reporters arguments to the start of the jest command. I’m wondering if doing something similar with your command might help?