[Product Launch] Smarter Testing is now in Beta

Build Environment Running Tests
21

Thank you! I’ll take a look at the comments/emails and follow up via email

22

I have my smarter testing setup and I’m seeing good results. Any updates on pricing?

23

Hi Bruce! That’s great to hear - we are finalizing the pricing decisions and hope to announce it in the coming days.

24

I recently tested it out in a large rails project and realized changes to erb files don’t trigger related specs (that definitely render those erb files). From what I understand this is likely a limitation in the coverage tool GitHub - circleci/rspec-circleci-coverage: RSpec plugin that generates coverage data for CircleCI's Smarter Testing. · GitHub and maybe a limitation in coverage tools in general.

Is this something that is even addressable? or is smarter testing more or less unusable for rails projects?

25

any news on pricing?

26

Hello,

The latest RSpec coverage plugin uses Ruby’s Coverage eval option (requires Ruby >= 3.2).

The change allows tracking template files, but it may be subject to the same limitations depending on how you are rendering those templates (re-eval’d templates only count coverage the first test that renders it).

Let us know if this addresses the missing erb file coverage in your project.

27

circleci/cci-agent-setup.yml

28

bc1q6zf4nw3x0suzev80f4dthphwlhlr3sz8ux3x5f

Milezdeep9 29

demo

30

Autoscaling sounds like a great feature!

31

Yeah, I just tried implementing the dynamic test splitting part of this (without the impact analysis yet), and the 2-3 minute spin-up time for each parallel runner just to have them end up not even doing any work is a major pain point in my tests.

Architecturally, I’m not sure how it would work though, since it seems like you have to run the discovery and test execution steps right next to each other. And at least with how I set it up, it would actually require all the setup to be run in the “main” runner to actually be able to discover the tests.

The other runners seem like they have to wait for the test atoms to be dispatched from that one, but I don’t think it’d make sense to wait for it to do the discovery before spinning up any other runners, since then you’re eating the startup time twice.

In practice, it’d be useful if they added a pipeline variable to indicate “this run is expected to have a reduced test suite” so you could just do a simple ternary to have a rough cut on which parallelism number to use. You can already kinda sorta do that by using the existing default branch one, but that wouldn’t catch cases like retries (which since I haven’t done the impact analysis stuff yet, is where I’m actually observing this problem).

Related, a 2nd pain point I was having with this feature is that it always skipped the already passed tests on a workflow, even if the “rerun the workflow from the start” option is selected. This was a smaller annoyance when I was trying to debug a problem with false negative, where tests were legit failing and triggering a rerun, but it wasn’t actually passing the failed spec’s details forward. This was actually a problem with our code, nothing on Circle CI’s end. Now that I take another look at the UI, I probably could have just pressed the “trigger pipeline” button on the branch page.

And then finally, my 3rd point of feedback for this new feature: I really wish there was a flag for circleci testsuite doctor to be able to have it just skip the step where it tries to run the specs locally. I think I saw something about it only running a limited sub-set of the tests, but when I let it go ahead with it, that part of the command still took more than an hour to finish, and was getting stuck on a spec that had differences between the local environment’s setup and CI’s anyway. I saw there was a bit in the docs about just adding it as a pipeline step to be able to run it in CI, but that’s also rather cumbersome, especially if you’re going to want to go back and forth to try to make changes quickly.

32

We’ve made some changes to the testsuite, please apply the following changes at your earliest convenience. These changes will be required starting September 14, 2026.

Local changes

Running the testsuite locally is now managed by the CircleCI CLI.

Remove the pre-existing testsuite and install via the CLI.

$ brew uninstall circleci/tap/circleci-testsuite

$ circleci extension install testsuite

Flag changes

The test impact analysis flags have been consolidated under a single consistent name to better represent what those flags control.

--test-selection and --select-tests have been renamed to --run-tests.

--test-analysis has been renamed to --analyze-tests.

Command changes

The testsuite is now a top level command with dedicated sub commands.

circleci run testsuite is now circleci testsuite run.

circleci testsuite "ci tests" is now circleci testsuite run "ci tests".

circleci testsuite "ci tests" --doctor is now circleci testsuite doctor "ci tests".

circleci testsuite "ci tests" --lists-tests-only is now circleci testsuite list-tests "ci tests".

Plugins published to package registries

Smarter Testing plugins have moved to a monorepo: Smarter Testing plugins repository.

And are available in their respective package registries:

33

The testsuite now supports a list-tests command, which you can use in a setup job to output the number of impacted tests before running them.This requires dynamic config to be enabled.

We’re exploring follow-up work to improve the experience along the lines you described - allowing the testsuite to determine the parallelism without needing dynamic config.

34

Hey, I’m trying to setup `dynamic test splitting with rails and rspec. I’ve followed the guide but I don’t see improvements in the different execution time across all my runner. does anyone have a sample setup with rspec to share to compare our setup?