Test Splitting on Self-hosted Runners

We’re thrilled to announce a huge enhancement for our self-hosted Runner offering, official support for CircleCI’s Test Splitting feature.

Customers who use self-hosted runners can now take advantage of CircleCI’s parallelism and test splitting features. Split your tests intelligently and use parallelism to save time for your developers.

Please keep in mind that in order to use the Test Splitting feature on a job with self-hosted runners, your runner resource class must have at least two self-hosted runners associated with it.

Comment below with any questions.

1 Like

@sebastian-lerner I haven’t found any docs to explain this so far: how many jobs can a self-hosted runner run at one time? Currently for my workflows it looks like it only one job runs on the self-hosted runner at a time; another parallel job waits to run even though it should be running immediately.

Do we have to run one self-hosted runner for every job we want to execute in parallel? So, 50 self-hosted runners running to run 50 jobs in parallel?

If so, how do we autoscale these runners? We have an autoscaling policy on the host that’s running the runner, so if the CPU is stuck at >90% for 5 minutes, it will launch another instance, etc. But depending on the instance size and the job’s processing, a new one might never spawn. On the other hand, if we launched each instance with 5-10 runner processes, we would end up with one instance with 5 jobs running on it (slowly), and eventually a second instance, with more runners. But I can’t see how CircleCI Cloud would know which runner to schedule a job on next.

Any advice on this would be greatly appreciated. Thanks

You define a number of runners with the same resource class name, so as the workflow is run the jobs can be queued against the pool of available runners.

The number of runners you can define is restricted by the type of account you have with circleci, which starts at 5 runners for the free offering.

I’ve never tried to autoscale the availability of runners so can not comment on how the circleci system will react to runners becoming available once the workflow is being processed.

Thanks for your reply!

You define a number of runners with the same resource class name

Not exactly sure what this means, could you elaborate please? Do you mean you literally execute multiple runners? Or do something else in the configuration to ‘define a number of runners’ ?

Yes, you just start up a number of runners which are defined using the same resource class name. The result is a pool of runners that tasks will be distributed across.

Hey folks, we do call out some specific self-hosted runner details on our docs for test splitting: Test splitting and parallelism - CircleCI

You are correct though that one self-hosted Runner can run one job today. So as was mentioned earlier, you would need to make sure you have n runners installed and accepting work if you want to run a job with parallelism= n.

I will note that we are working on making significantly easier to accomplish with a more scalable and container-friendly self-hosted runner, now in Open Preview: Container agent (container runner) open preview - CircleCI You should be able to install this in a Kubernetes cluster and it’ll spin up n ephemeral pods for your jobs that use parallelism = n.