Monorepo and executing jobs related to changed files

Build Environment
1

Hi! I’m still a newbie in ci/cd world and circleci in particular.
I have a repo with one service and I’m using the following configuration.

  • With all PRs the jobs build and test run.
  • When merging code in master, build, test and deploy.

Here is the configuration file.

version: 2.1

executors:
  custom-executor:
    docker:
      - image: circleci/node:lts

jobs:
  build:
    executor: custom-executor
    steps:
      - checkout
      - restore_cache:
          key: dependency-cache-{{ checksum "package-lock.json" }}
      - run: 
          name: Installing packages
          command: |
            npm install
      - save_cache:
          key: dependency-cache-{{ checksum "package-lock.json" }}
          paths:
            - node_modules
      - persist_to_workspace:
          root: /workspace
          paths: 
            - node_modules

  test:
    executor: custom-executor
    steps:
      - checkout
      - attach_workspace:
          at: /workspace
      - run:
          name: Unit Test
          command: npm run test

  deploy:
    executor: custom-executor
    steps:
      - checkout
      - attach_workspace:
          at: /workspace
      - run:
          name: Serverless
          command: | 
            ./node_modules/.bin/serverless deploy

workflows:
  version: 2
  build_test_deploy:
    jobs:
      - build
      - test:
          requires:
            - build
      - deploy:
          requires: 
            - build
            - test
          filters:
            branches:
              only: master

Now, I would like to migrate to a monorepo with the following structure:

|- main-folder
|-- application1
|-- application2

At a high level, changes can happen in main-folder and|or in any of the applications folder.
The problem: I just want to run the jobs for the folders where code changed.
Constraints:

  1. PRs should build and test just the service affected. If I change something in application2, jobs should not run for main-folder or application1, just for application2.
  2. If I change something in main-folder, it is opk if all jobs run, since we can presume it could impact application1 and application2.
  3. Some for the deploy job: it should execute just for the service that changed.

I don’t know if I am being clear. It is difficult to explain something that one does not know well enough.

Is there a way to identify through the files that changed which service or application should run particular jobs?

I will appreciate help and advice.

2 Likes
2

just move to github. actions support this Workflow syntax for GitHub Actions - GitHub Docs

1 Like
3

Hi

I’m doing the similar things. Did you find a solution without using Github Workflow actions?

4

Hi,

You can implement this logic by leveraging CircleCI’s Dynamic Config functionality.

More specifically, I recommend following this guide to execute specific workflows or steps based on which files are modified.

5

@yannCI I think it would be easier to take the proposition of staying on CircleCI if the compelling factor was more than just “Cheaper than Github”.

Currently there’s some fundamental shortcomings in CircleCI that will prevent me from ever choosing it in new projects.

What’s managements position on this for example:

6

Depending on your use case and the languages involved, one option is to look at tooling designed for working with monorepos (bazel, pants, etc.), though most of these solutions are, themselves, somewhat complicated and hard to learn. In other words, run the same workflows every time, but then use a separate tool that’s dependency-aware to figure out what needs to run (there are also some language specific tools, for example, for JS, tools like nx / turborepo / etc)

While the dynamic config option w/ path filtering orb more or less works, it feels to me like a hacky / bolted on solution. Also, if you are trying to use “required” checks, it can be a little complicated to deal with using path-filtering if the resulting workflows have different names.

Overall, as best I can see from the outside, CircleCI wasn’t really built with first class support for a monorepo type setup in mind. One of the few things I like about Google Cloud Build (even as someone who’s not a big fan of monorepos overall) is the ability to have file glob filters at the trigger level (natively) (on the other hand, there, you can’t do branch / tag filters aside from creating separate triggers and configs, so there’s always some pros and cons)