We’re excited to announce that Smarter Testing is now available in beta for all Cloud customers. This includes test impact analysis, dynamic test splitting, and automatic rerun of failed tests — all designed to help you run fewer tests, run them faster, and get signal sooner, while maintaining full confidence.
What beta means
Beta means the product is in early stages and you may encounter bugs, unexpected behaviour, or incomplete features. When the feature is made generally available, there will be a cost associated with access and usage.
If you get stuck or run into unexpected behavior, let us know by dropping your questions/concerns right here in this thread or by reacing out to me directly at minjun.seong@circleci.com — your feedback directly shapes what Smarter Testing looks like at GA.
Have ideas or feature requests? Submit them on our Ideas board where you can also see existing feature requests and vote on them.
Is it true that it only compares with the default branch? Would it compare with a commit that has a tag and is on the default branch? For most of our tests, we don’t run them directly on the default branch. And testing is tough because it’s doesn’t seem to care about tests run previously on a particular feature branch.
You can set it up with any test runner as long as it outputs coverage data, so SimpleCov works well here. Additionally, we offer built-in coverage support for certain test runners, and RSpec is one of those supported options.
Test Impact Analysis compares your changes against the impact map generated during the analysis phase. For most teams, this map is built from the default branch.
When you push to another code change, we compare the changes in that branch against the existing impact map and only run the necessary tests.
So in practice:
It doesn’t strictly “compare branches” directly
It compares your changes to the baseline map (usually from the default branch)
I think there are 2 things to consider. One is the impact map for deciding whether a test needs to run, and the other is the set of changes to check against the impact map. I don’t really get where it is getting either. of those in our environment. Is there a way I can have visibility into that? If I run the tests on a tag that is on a commit and then that commit is in the history of a feature branch I am working on, will it use the impact map from that run and the changes from that commit?
The impact map is built during analysis runs. During analysis, each test runs with coverage instrumentation, and we record which source files each test touches along with the content hashes of those files. You can inspect the impact data locally by running analysis with a local run — see this section of the docs.
When using test selection, we hash your current source files and compare those hashes against the hashes stored in the impact map — it’s purely a content-level comparison against the latest impact map. Smarter Testing does not compare the contents of commits or tags, it compares the checked out file system with the impact data from the last time analysis was run.
OK. That seems very good. It’s not about the git diffs. It’s about the file hashes since the impact analysis was run. I think we can manage that effectively. Thank you very much! My mind was too connected to git. You all do such great work!
Been testing this, with good results so far. Really interested in some of the other things that I suspect this will unlock down the road (but also curious / nervous to see how the pricing model will be).
I know it’s possible to use mustache templating to do different parallel numbers on the default branch vs. other branches, but one thing that I do think customers with high parallelism (at the CircleCI test splitting level) will really benefit from would be some way to do parallelism a bit more dynamically depending on the number of changes… I can see a scenario where most of our branches have relatively few tests to run, but there will also be scenarios (where many files are changed, or where we’ve changed one of the files that the suite says should trigger a full run) where setting lower parallelism on PR branches would then take forever [with the reduced parallelism].
Thanks for the detailed feedback—this is a great point. We’re not building dynamic parallelism directly, but we do have related work in our backlog that would help enable use cases like this. Your example is really helpful context as well.
We’re excited to share that the doctor CLI flag is now available for Smarter Testing. If you’ve been holding off on setup or running into configuration issues, this should make things significantly easier.
The doctor flag inspects your current Smarter Testing configuration and flags any issues — so instead of working through documentation to figure out what’s wrong, you can run a single command and get direct feedback on what needs to be fixed and how.
We’ve also updated our documentation to incorporate the doctor flag into the setup flow, so whether you’re getting started for the first time or troubleshooting an existing setup, the docs will walk you through using it.
Note: Make sure you’re on the latest version of the CLI
brew upgrade circleci/tap/circleci-testsuite
If you run into any confusion or challenges, let us know in this thread — we’re continuing to iterate based on your feedback.
and this command: circleci run testsuite “TypeScript” --verbose --doctor
I receive the following output:
11:44:48.116 ==> Running test-suite-subcommand version "1.0.31123-6f0a497" built "2026-04-14T11:14:28Z"
11:44:48.117 ==> Testsuite timeout: 4h40m0s
11:44:48.119 ==> Running test suite 'TypeScript'
11:44:48.120 ==> Suite Configuration:
name: TypeScript
discover:
command: vitest list --filesOnly
shell: /bin/bash --login -eo pipefail
run:
command: vitest run --reporter=junit --outputFile="<< outputs.junit >>" --bail 0 << test.atoms >>
shell: /bin/bash --login -eo pipefail
outputs:
junit: test-reports/tests.xml
options:
timeout: 10m0s
test-impact-analysis: false
full-test-run-paths:
- .circleci/config.yml
- .circleci/test-suites.yml
- package.json
dynamic-test-splitting: false
test-selection-rules: []
impact-key: default
max-auto-rerun: 0
auto-rerun-duration: ""
11:44:48.120 ==> Discovering...
⎯⎯⎯⎯⎯⎯⎯ Collect Error ⎯⎯⎯⎯⎯⎯⎯⎯
file:///<<my-repo>>/pos/node_modules/.pnpm/vite@7.3.1_@types+node@24.10.12_jiti@2.6.1_lightningcss@1.31.1_sass@1.97.3_terser@5.46.0/node_modules/vite/dist/node/chunks/config.js:8
import fsp, { constants } from "node:fs/promises";
^^^^^^^^^
SyntaxError: The requested module 'node:fs/promises' does not provide an export named 'constants'
at ModuleJob._instantiate (node:internal/modules/esm/module_job:128:21)
at async ModuleJob.run (node:internal/modules/esm/module_job:194:5)
at async Promise.all (index 0)
at async ESMLoader.import (node:internal/modules/esm/loader:385:24)
at async collect (file:///<<my-repo>>/node_modules/.pnpm/vitest@4.1.4_@opentelemetry+api@1.9.0_@types+node@24.10.12_happy-dom@20.7.0_jsdom@26.1._1f3cecf7a503aa385cd29990e9012582/node_modules/vitest/dist/chunks/cac.wyYWMVI-.js:2361:63)
11:44:48.429 ==> Discovered 0 test atoms in 309ms
11:44:48.429 ==> 'discover' [1] CircleCI received a non-zero exit code from the test runner: exit status 1
11:44:48.434 ==> The discover command exited with a non-zero exit code.
Try running it manually to see the error output.
vitest list --filesOnly
Error: failed to execute plugin 'testsuite': exit status 1
If I change the command to execute through pnpm instead (discover: pnpm test:list - in the package.json file this is just setup exactly the same) I receive the error:
ERROR: This version of pnpm requires at least Node.js v18.12
The current version of Node.js is v16.15.1
Running node -v returns v24.11.1 and AFAIK I don’t have any manually installed copy of v16 anywhere.
circleci version returns 0.1.34950+8ff907b (homebrew).
I’ve only attempted to test this locally and not within CircleCI at this point.
Sorry if this isn’t the right place for this but it seemed related to the last message.
The testsuite will execute the commands in a subprocess with the environment, there could be something not getting forwarded to that subprocess causing the wrong node version to be used.
How do you run tests without Smarter Testing?
Could you try the following please?
Remove pnpm from the test-suites.yml.
Prefix the command with pnpm:
pnpm circleci run testsuite “TypeScript” --verbose --doctor
Great feedback! What is the outcome you are looking to drive by being able to dynamically set the # of nodes depending on the # of tests to run? Is it to try to save cost, time, both or something else?
I think all of the above… we have an app that doesn’t work well with pytest-xdist level parallelism, meaning we have very high parallelism (lots of smalls, with tests running concurrently… I know).
So, if there are 3 tests to run vs 40,000, it seems silly to bring up over a hundred containers, but if there’s a change that triggers a full suite run, it would take forever.
But in addition to the time the tests take, there’s both the time and possible flakiness of all the setup items (installing dependencies, seeding the db, etc.), so just bringing up a whole bunch of containers and then having them shut down after setup runs still incurs both cost and the risk of something (e.g., a package install) failing. Plus there’s some shared startup time for some tests themselves.
In a sense, though, watching the way the tool allocates out tests, it feels like the logical next step would be just to set a min and max number of “executors” and let the test runner figure out how to allocate tests to them.
Also, what does it mean when Running N test atoms happens more than one time within a specific parallel run? is that intentional / by design in terms of another higher level of splitting?
I emailed you too, but I’m seeing another thing: one recent run ran a pretty large subset (1/3 maybe) of the tests when it had no changes to affected files, and didn’t touch any of the files that required a full run.
Here’s what I’m wondering: I’m running the analysis on a schedule 4-5 times a day (also with high parallelism). During that, I see
Updated test impact data in each of the parallel runs. If that run is sending data in over time, is it possible to get partial coverage results because it’s invalidated some of the old coverage / analysis data, but hasn’t yet fully received all of the new data?
