The fail-fast feature is deprecated as of June 1, 2023.
Please reach out to sebastian @ circleci.com with any questions.
2 Likes
Thank you for introducing this new feature 
I am trying to get this working with cypress tests but I am getting this error:
ERRO[2023-02-17T17:25:29Z] the --command flag and script is required for this plugin to do work
Error: exit status 1
I tried running the command with --verbose flag but no other input is shown on the logs.
I am splitting the tests using the glob command and passing the results via a variable to cypress run command.
I know the test splitting works as this has been running for a long time on our pipeline. We use --split-by=timings on our current pipeline but to try and make this work I have tried --split-by=name as suggested by the docs but no luck still.
Any help on this would be much appreciated.
Hi @sebastian-lerner it’s my first time using this forum (newbie alert ). How can I share something as a DM
Click in the user’s name in the message they have posted (in this case sebastian-lerner) and you should be shown a drop down menu with ‘message’. as an option.
I can’t see that option or any dropdown menu available 
It’s probably related to my permissions since I am new user to the forum.
@sebastian-lerner Here’s the command I am trying to run. Thank you for taking a look.
- run:
name: Run e2e tests
no_output_timeout: 15m
command: |
# spin up the app
npm run start:ci:e2e
# find all the test files
TESTS=$(circleci tests glob "dir/dir/src/endToEnd/**/*.spec.ts")
echo $TESTS
# run the tests
circleci tests run
--command="xargs cypress run --headless --browser chrome --config-file dir/dir/cypress.json --spec "$TESTS" --"
--verbose
--split-by=name
--batch-count=3
--fail-fast
--test-results-path="dir/dir/report"
Oh, it seems that the forum system has been configured to only grant the right to message another user after you have made a number of posts. I guess Sebastian is not aware of this limitation when he asked you to message him.
Sorry about that @drilon241, I thought all Discuss users could send DMs. Thanks for sending that over.
One thing that stuck out from your command, can you try including "" at the end of each line to make sure that the circleci tests run command knows that all of those flags are for it to parse? So something like:
command: |
npm run start:ci:e2e
TESTS=$(circleci tests glob "dir/dir/src/endToEnd/**/*.spec.ts")
echo $TESTS
circleci tests run \
--command="xargs cypress run --headless --browser chrome --config-file dir/dir/cypress.json --spec "$TESTS" --" \
--verbose \
--split-by=name \
--batch-count=3 \
--fail-fast \
--test-results-path="dir/dir/report"
and let me know what the output is after that? Once we get to the bottom of it, I’ll make sure to update the docs to make things more clear.
Thank you @sebastian-lerner - it got a bit further this time. It was able to run the command but wasn’t able to read the file names from stdin
See error message below
Installing circleci-tests-plugin-cli plugin.
circleci-tests-plugin-cli plugin Installed. Version: 1.0.3349-470dac4
DEBU[2023-02-22T10:14:24Z] Attempting to read from stdin. This will hang if no input is provided.
WARN[2023-02-22T10:14:24Z] No test names found in input source. If you were expecting test names, please check your input source.
INFO[2023-02-22T10:14:24Z] starting batch execution batch_count=3 batches_processed=0 total_batches_for_job=12
DEBU[2023-02-22T10:14:24Z] There were no tests found in bucket 0
DEBU[2023-02-22T10:14:24Z] There were no tests found in bucket 0
DEBU[2023-02-22T10:14:24Z] There were no tests found in bucket 0
DEBU[2023-02-22T10:14:24Z] Processed 3 of 3 batches of tests
INFO[2023-02-22T10:14:24Z] ending batch execution batch_count=3 batches_processed=3 total_batches_for_job=12
1 Like
Thanks @drilon241, one more thing that I think will do the trick. We need to make sure that the circleci tests run command is getting the list of tests piped in via stdin properly. Can you try adjusting to the following:
TESTS=$(circleci tests glob "dir/dir/src/endToEnd/**/*.spec.ts")
echo $TESTS | circleci tests run \
--command="xargs cypress run --headless --browser chrome --config-file dir/dir/cypress.json --spec" \
--verbose \
--split-by=name \
--batch-count=3 \
--fail-fast \
--test-results-path="dir/dir/report"
The formatting on Discuss makes it look like the slashes ("") are commenting out the commands, but you can ignore that, I couldn’t figure out how to make it not look like it’s being commented out sorry. You still need the back slashes "" if you put the commands on different lines.
2 Likes
Installing circleci-tests-plugin-cli plugin. circleci-tests-plugin-cli plugin Installed. Version: 1.0.3349-470dac4 DEBU[2023-02-22T10:14:24Z] Attempting to read from stdin. This will hang if no input is provided. WARN[2023-02-22T10:14:24Z] No test names found in input source. If you were expecting test names, please check your input source. INFO[2023-02-22T10:14:24Z] starting batch execution batch_count=3 batches_processed=0 total_batches_for_job=12 DEBU[2023-02-22T10:14:24Z] There were no tests found in bucket 0 DEBU[2023-02-22T10:14:24Z] There were no tests found in bucket 0 DEBU[2023-02-22T10:14:24Z] There were no tests found in bucket 0 DEBU[2023-02-22T10:14:24Z] Processed 3 of 3 batches of tests INFO[2023-02-22T10:14:24Z] ending batch execution batch_count=3 batches_processed=3 total_batches_for_job=12ข้อความก่อนจัดรูปแบบ
hey @sebastian-lerner I tried your suggestion but the cypress was complaining about having the --spec arg without a value.
So I tried a few other options
-
removed the
--specfrom the cypress command
Cypress test started running but it was running all spec folders instead of running only the specs found in this directoryendToEnd/**/*.spec.ts
We also have cypress integration tests on another folder which we run separately, so this option wouldn’t work for us.
I also noticed that it wasn’t actually splitting the tests as all 4 parallel runs were trying to run the same specs -
tried to pass the test list directly to the
circleci tests run
TESTS=$(circleci tests glob "dir/dir/src/endToEnd/**/*.spec.ts")
$TESTS | circleci tests run \
This failed as it wasn’t able to find the input files
No test names found in input source. If you were expecting test names, please check your input source
- tried to echo and also use cypress
--specoption
TESTS=$(circleci tests glob "dir/dir/src/endToEnd/**/*.spec.ts")
echo $TESTS | circleci tests run \
--command="xargs cypress run --headless --browser chrome --config-file dir/dir/cypress.json ----spec "$TESTS" --" \
This failed same as attempt 1 as it was running all spec files and not splitting them.
- tried specifying the test path using the cypress
--specoption
circleci tests run \
--command="xargs cypress run --headless --browser chrome --config-file dir/dir/cypress.json --spec "dir/dir/src/endToEnd/**/*.spec.ts" --"
This failed as it wasn’t able to find the spec files
No test names found in input source. If you were expecting test names, please check your input source.
- last attempt was to use glob command and specify directory using cypress --spec option
This failed with with
TESTS=$(circleci tests glob "dir/dir/src/endToEnd/**/*.spec.ts") | circleci tests run \
--command="xargs cypress run --headless --browser chrome --config-file apps/app-e2e/cypress.json --env automationPassword="${CI_AUTOMATION_USER_PASSWORD}" --spec "apps/app-e2e/src/endToEnd/**/*.spec.ts" --"
Not sure what else to try but I am very keen to get this working. I feel we are so close 
@drilon241 working with the team on this. Sorry about the challenges. Hope to have an update relatively soon.
1 Like
i posted something earlier that I don’t think will actually work…deleting it and investigating more. Will update when I have more
@drilon241 a couple more questions to help us with diagnose the issue:
- What execution environment are you using? Even better, if you could send a link to the job to sebastian@circleci.com
- Can you try the following and let us know the output?
echo $TESTS | circleci tests run --command="xargs echo TEST GROUP" --batch-count=2
Hey folks I’m trying to add the fast-fail as well however I got an error saying: Unrecognized option "fail-fast"
I have added the fail fast here and did not work
name: Unit tests
command: |
TESTS=$(circleci tests glob "./src/**/*.test.js" "./src/**/*.test.tsx" "./src/**/*.test.ts" | circleci tests split --split-by=timings --fail-fast --batch-count=3)
echo "Tests to be run: $TESTS"
yarn test $TESTS --watchAll=false --coverage --runInBand --verbose --reporters=default --reporters=jest-junit --coverageThreshold '{}'
Also on this way with the same result
name: Unit tests
command: |
TESTS=$(circleci tests glob "./src/**/*.test.js" "./src/**/*.test.tsx" "./src/**/*.test.ts" | circleci tests split --split-by=timings )
echo "Tests to be run: $TESTS"
yarn test $TESTS --watchAll=false --coverage --runInBand --fail-fast --batch-count=3 --verbose --reporters=default --reporters=jest-junit --coverageThreshold '{}'
@byleto7 the command should be circleci tests run instead of circleci tests split
See config example:
Thank you for this feature, yet I’m a bit disappointed with how it works with parallelism :
(for us) parallelism is useful to speed up the execution of several tasks at the same time (each task also runs parallel tests), but to use the fail-fast option, we have to introduce batches in these parallel tasks and (when no failure) it becomes slower…
I was hoping this feature would immediately stop all running parallel executions on failure, without introducing those batches.
Is this something planned?
thanks !
@jeremiewtd Unfortunately, this is the expected behavior and how it will continue to work for the foreseeable future. Does reducing the batch size to something small like 2 still introduce too much overhead and slow things down?