Hello,
We are using CircleCI as our CI tool for our monorepo. Since our repo is containing more and more libraries and applications, we are looking into ways of optimizing our CircleCI workflows.
So we split up the work in two workflows. First of all, we have a discover workflow, which just discovers what has changed and what needs to be re-tested and re-built. When it has found some work, it triggers a second workflow which does the actual work. If no changes are found, it stops here.
To trigger the second workflow we use pipeline parameters, which work.
We wanted to also optimize our second workflow to run as many jobs in parallel to speed things up (inspired by https://blog.nrwl.io/blazing-fast-distributed-ci-with-nx-a1f5974f7393).
So we have created a test-many job that can unit test up to 3 projects in parallel. If you pass 4 projects to this job, it will first start 3 in parallel and will start the 4th one if one of the other 3 is finished.
To speed things up more, we have 3 test-many jobs in our workflow. The discover workflow will split the changed projects into 3 buckets and send them as pipeline parameters (test1, test2, test3) to the second workflow.
To make my config smaller, I used the matrix feature to not repeat myself constantly. We have something like this:
- testing-many:
matrix:
parameters:
projects:
[
<< pipeline.parameters.test1 >>,
<< pipeline.parameters.test2 >>,
<< pipeline.parameters.test3 >>,
]
This works as well and passes the pipeline parameter as a job parameter to the testing-many job.
Now it happens from time to time that we only do small changes for one pull request and that only a small number of libraries/applications are affected and that we actually don’t need 3 jobs for running tests. That we can handle it with only 2 or 1.
In this case, I don’t want to waste time or resources, so I was looking for a way to conditionally skip those jobs and I hoped I could use exclude for this.
So I wanted to exclude the job if pipeline.parameters.test1 (or 2 or 3) was an empty string. In this case, the job should not run because we did not have any projects for this job. So I updated my workflow as follows:
- testing-many:
matrix:
parameters:
projects:
[
<< pipeline.parameters.lint2 >>,
<< pipeline.parameters.lint2 >>,
<< pipeline.parameters.lint3 >>,
]
exclude:
- projects: ''
So I would think that the job would be excluded if the pipeline parameter is an empty string. But this isn’t the case.
Does anyone have an idea why this would not work?
It works if I don’t use a pipeline parameter but a fixed value instead. For instance if I use the following configuration, only 2 jobs are created instead of 3:
- testing-many:
matrix:
parameters:
projects:
[
<< pipeline.parameters.lint2 >>,
<< pipeline.parameters.lint2 >>,
'',
]
exclude:
- projects: ''
I know I can use a when step in my job’s steps (which I’m doing now as a workaround) but this is not sufficient enough for me. One, it takes up resources and can get queued (slowing everything down, for something that isn’t doing anything) and secondly, I don’t want it to show up in my workflow UI. Because now I’ll see 3 jobs, but I’ll have to open them to see if they did something.
Another thing I thought about but will not work for my use case is creating multiple workflows according to the number of jobs I want to run. I could create a workflow to only run 1 test job, another to run 2, and the last one to run 3 test jobs. And then using pipeline parameters to run the correct workflow.
This would work if I had only one such a job. But my workflow contains 3 such jobs with all of them 3 pipeline parameters. If I want to create a workflow for each possible permutation I would already need 27 different workflows and this would rise exponentially if we add a parameter in a job.
Does anyone have some ideas, how I could move forward?
Thanks,
Joris.