I just pushed a branch where I’m switching to a workflow to take better advantage of parallelism:
test_rb step is set with
parallelism: 3; all other jobs are the default of 1.
We are paying for 4 executors:
There are 2 unrelated jobs running at the moment, which means there should be 2 free. However, it appears that the next job to run (with parallelism 3) is waiting for 3 executors to all be available at the same time. Additionally, this is preventing any other jobs from processing. By attempting to use more parallelism, the workflow is actually forcing us into less.
Here’s the queue visualization from the last job:
How can I configure the job with parallelism 3 to not wait for 3 executors to be available, but instead run each job as soon as possible?