Dynamic job parameter and number of jobs

Hello. I am looking for a way to implement following pattern:

  • a first job runs a script that analyze git history to identify which components are impacted by changes
  • according to result, one or more jobs have to be triggered with different parameters
  • aggregate jobs’ results to mark PR as valid or rejected

To be more concrete, here is the repository I would like to process: https://github.com/SynoCommunity/spksrc/
As a set of cross-compilation Makefiles, splitting everything means hundred of small (3-5 files each) repositories.

Thank you in advance for your help. Yves

Do you have a monorepo? If so, search for that word in this forum - it has been asked about a few times, but I don’t think there is official support for it.

Thanks for the hint. “monorepo” or “monolithic multi-project repository” are keywords I should search for.

I would expect to trigger all jobs in a single workflow to that to mark PR status in a single call.

If multiple workflows are triggered, I will face another issue - implement a “rendez-vous” to aggregate results.

I’ve not had to do this, but I wonder if I would set up Git mirrors for each sub-project in the monorepo, and then on each Git push, pull each mirror, copy the monorepo folder into each mirror, commit, and push. You could then attach a CircleCI project to each mirror.

The nice thing with this arrangement is that if there is no change in a mirror then the commit will fail, and pushing will have no effect. Thus, mirrors that are not changed during a push will not get a CI trigger.

Thanks for proposal but that is not so easy - even with Git submodules…
Content in repository is shared between components - there are buildable “terminal components” (which produces a package) and common “dependencies” (libraries) which are buildable on their own but not relevant alone.
Complexity is that a change in one dependency/library has to trigger builds on impacted “terminal components” (one or many) - but not necessarily all components in repository.
I will go on reading advices here in varia posts to see if I can achieve my goals with as less refactoring in build chain as possible.

You know your projects, so if you are certain it would not work, that’s fine. But there is nothing to say that you could not copy some shared folders in the monorepo to all downstream mirrors. Equally, the mirrors are not obligated to behave similarly, either - they would each have their own build config.

I wouldn’t touch submodules though, I still have nightmares about them :scream_cat:

Be sure to vote for https://ideas.circleci.com/ideas/CCI-I-208

… no I am not certain … but for my first use of cloud CI, I do not start with an easy project. I will report here my progress.

1 Like

Welcome to continuous integration :heart_eyes:

I’ve usually found using workflows to run all the jobs to be appropriate, fanning in as per the workflows guide, then adding a step in each that guards whether it should run or not a la:

      - run:
          name: skip the rest of this job if no changes since dev in this branch on ./path/of/interest
          command: |
            if git diff origin/dev.. --quiet -- ./path/of-interest; then
              circleci step halt
      - run: # further steps happen if there were changes
1 Like

Thanks. In my case, a PR may trigger multiple workflows (when a common library changes impact multiple packages)… so I wonder how to vote PR (accept/reject) in such a context.
Is there a way to “join” workflows result?

May not be the answer you want, but perhaps you could do one big workflow and others as (logical) subworkflows comprised of dependencies between each of the builds. Each build fans into a single “yay - finished” build, then a final fan-in brings the logical subworkflows together for a final success/failure.