Does CircleCI 2.0 work with monorepos?


#1

‘monolithic multi-project repository’ was mentioned in the beta access form, but I didn’t see any details about it in the circle ci 2.0 topics.

Is it supported?

Thanks


Mono repo support
Does CircleCI 2.0 supports building a project separately in a Multi-Project repo
#2

Some people have reported issues with deploy keys, but pulling other modules and submodules will be supported if it’s not already.


#3

I’m thinking more about a github repo that contains lots of individual projects that I want to build independently.

Ideally, if I could have a circle.yml per project and only build that project if files within the project have changed within a commit.

Is something like this supported or planned to be supported in circle 2.0? It’s preventing us from moving all our build jobs to CircleCI and I would really like to do that.


#4

There’s no reason you can’t do that with 2.0. You have full control over what commands are executed. Though I don’t fully understand the use case of having multiple projects in a single repo, and I doubt there will be native support for this, you can achieve this with shell scripts.


#5

Im the same as mkellyclare, I read the bit about the monolithic repositories and was hoping to be able to build projects independently within a single repo. With the growing trend of microservices these days I can only see this becoming more and more necessary.

For example I have a repository structured as follows.

> ServiceA
> ServiceB
> ServiceC
> ServiceD
circle.yml

Currently I just setup a build commands to run in a shell script on the relevant branches. The problem with this that every time I commit to this repo unit tests are run for every service. I have setup certain branches to trigger deployments, so all services get deployed at once.

While this approach does work, it tests and releases unmodified code unnecessarily. Meaning my build times are far longer than they need be. Each of my services take roughly 2-3 minutes to test, build and deploy. I have six of these currently.

Ideally I would prefer to have a circle.yml per Service directory. This would only be run if modifications occurred in this directory only.


#6

I agree that your current situation isn’t ideal. I’ve created a feature request for this internally.

It’s possible–though a bit involved–to do this now. It would involve clipping the CIRCLE_COMPARE_URL to parse out the current and previous Git SHAs pushed to Github. From there, you can use something like git diff 1f7e1be0f11c...de2de1347fb4 --name-only to get a list of changed files. You could then use this list to check if your project folders are among those changed, and conditionally run tests based on that.


#7

Eric,

Do you have an example you could share? I’m trying to achieve the same thing as tomhughesnice and mkellyclare but am struggling to get this working.


#8

Okay, I put together a simple example.

CIRCLE_COMPARE_URL typically looks like this:

CIRCLE_COMPARE_URL=https://github.com/circleci/support-sandbox-picard/compare/1f7e1be0f11c...de2de1347fb4

For this demo, I’ve created commits that add files to foo/, bar/, or both.

$ git log -3 --oneline --decorate
565a9ee (HEAD -> monorepo-sandbox) Add foo/* and bar/*
0423741 Add foo/foo/*
517a81a Add a file in bar/bar/p

For these commits, the following bash and git logic will print foo when a commit range includes changes in foo/, and print bar for changes in bar/.

      - type: shell
        command: |
          COMMIT_RANGE=$(echo $CIRCLE_COMPARE_URL | sed 's:^.*/compare/::g')
          echo "Commit range: " $COMMIT_RANGE
          git diff $COMMIT_RANGE --name-status
          if [[ $(git diff $COMMIT_RANGE --name-status | grep "bar") != "" ]]; then
            echo "bar"
          fi
          if [[ $(git diff $COMMIT_RANGE --name-status | grep "foo") != "" ]]; then
            echo "foo"
          fi

This was the output I got for 3 different pushes:

Commit range: fe2dd5a2f0d5…79e2afc04032
A bar/a
A foo/d
bar
foo

Commit range: e1c076da7874…fe2dd5a2f0d5
A foo/c
foo

Commit range: 79e2afc04032…3c2d27509dd7
A bar/b
bar

This is a basic POC. You may want to make this more robust by switching from --name-status to --name-only, and passing a regex to grep that includes the beginning of the line. This will reduce the likelihood of unintentional matches.


#9

Eric,

Thanks for your help, I’ve managed to get it working using your approach:

version: 2
executorType: machine
stages:
  build:
    workDir: ~/infra
    steps:
      - type: checkout
      - type: shell
        name: "Install Dependencies"
        command: |
          curl -sL https://deb.nodesource.com/setup_4.x | sudo -E bash -
          sudo apt-get install nodejs=4.8.0-1nodesource1~trusty1
      - type: shell
        name: "Determine which Projects need to be built"
        command: |
          COMMIT_RANGE=$(echo $CIRCLE_COMPARE_URL | sed 's:^.*/compare/::g')
          echo "Commit range: " $COMMIT_RANGE
          git diff $COMMIT_RANGE --name-status
          if [[ $(git diff $COMMIT_RANGE --name-status | grep "lambda/circleci") != "" ]]; then
            echo "setting build circleci lambda"
            touch /tmp/build-circleci.lambda
          fi
          if [[ $(git diff $COMMIT_RANGE --name-status | grep "lambda/terminate_expired_servers") != "" ]]; then
            echo "setting build terminate-expired-servers lambda"
            touch /tmp/build-terminate-expired-servers.lambda
          fi
      - type: shell
        name: "Build Circle CI  Lambda"
        command: |
          if [ -f "/tmp/build-circleci.lambda" ]; then
           echo "[BUILDING] CircleCI lambda"
          else
            echo "[SKIPPING] CircleCI Lambda"
          fi
      - type: shell
        name: "Build Terminate Expired Servers Lambda"
        command: |
          if [ -f "/tmp/build-terminate-expired-servers.lambda" ]; then
            echo "[BUILDING] Terminate-expired-servers Lambda"
            cd /home/infra/aws/lambda/terminate_expired_servers
            npm install
          else
            echo "[SKIPPING] Terminate-expired-servers Lambda"
          fi
      - type: shell
        name: "Remove temporary files"
        command: |
          rm -f /tmp/build-*

I’m still quite new to CircleCI and couldn’t figure out if there was a way to set a variable/env var in the yml file which i could overwrite btw steps (it doesnt appear that I can from the docs), so in the end I just use files in the /tmp directory as flags. This is so that I can separate the check to see what projects need building from the steps to build each individual one.

Would appreciate any feedback you or others might have on improving this

Many thanks

Nadeem


CIRCLE_COMPARE_URL not set on first build
#10

Thank you @kiyanwang for sharing your config.

Using empty files is a good approach for tracking state. You can also set a global environment variable BASH_ENV to instruct bash to source a file before running commands. For instance:

jobs:
  build:
    docker:
      - image: ruby:2.4.0

    working_directory: ~/work
    environment:
      BASH_ENV: ~/.bashrc
    steps:
      - run: echo "echo 'hi'" >> ~/.bashrc
      - run: echo "test"

Each run step executes as a separate bash subshell. If other commands were grouped with the first run, they wouldn’t yet pick up the change to .bashrc.


#11

Something that is missing is to skip complete steps in a workflow based on whether a file has changed or not. Right now this hack helps to skip steps, but only if those steps are actually run steps guarded by an if. It does not help with e.g. setup_remote_docker nor does it allow to bail from executing the following steps of the current job without stopping the whole workflow.


#12

This doesn’t even have to be a big multi-project monorepo, though there are good reasons to use one of those. We have our front-end Javascript code and our back-end Java code in the same repo because we want to be able to atomically commit changes that touch both sides. It’s pointless to run unit tests on the Java code when someone makes a tweak to some display logic in the front-end code, or vice versa.

In CircleCI 1.0 we stick our entire build/test process in a shell script that does git show to figure out what changed and selectively runs tests accordingly, but it seems like that approach won’t let us take full advantage of some of 2.0’s (highly desirable!) new features since we can only put path-based conditional logic in run steps.


#13

This solution worked for us.

We have a mono-repo with subdirectories each containing a single Serverless project. After some digging, I came up with this chain of bash commands for config.yml:

version: 2
jobs:
  build:
    docker:
      ... Global image definitions for all services ...

    working_directory: ~/repo

    steps:
      - checkout
      - restore_cache:
          keys:
          - v1-dependencies-{{ checksum "package.json" }}
          - v1-dependencies-
      - run: echo "export STAGE=$(if [ $(git rev-parse --abbrev-ref HEAD) = 'master' ]; then echo prod; elif [ $(git rev-parse --abbrev-ref HEAD) = 'staging' ]; then echo staging; else echo dev; fi); sudo npm install -g serverless jest" >> $BASH_ENV
      - run: for dir in $(git log --name-only --oneline -1 | sed 1d | grep '/' | cut -d "/" -f1 | sed '/^\.circleci$/d' | sort -u); do cd $dir; npm run build; npm test; npm run deploy; cd ..; done

And in each subdirectory’s package.json:

"scripts": {
    "build": "npm install",
    "test": "jest",
    "deploy": "serverless deploy --stage $STAGE"
 }

The first run command is

echo "export STAGE=$(if [ $(git rev-parse --abbrev-ref HEAD) = 'master' ]; then echo prod; elif [ $(git rev-parse --abbrev-ref HEAD) = 'staging' ]; then echo staging; else echo dev; fi); sudo npm install -g serverless jest" >> $BASH_ENV

This does the following:

  • Configures automatically a $STAGE env var for serverless deployments, based on the Git branch. If it’s master, set STAGE to prod. If the branch is named staging, set it to staging. Else, set STAGE to dev.
  • Installs all necessary global modules for all services (serverless, jest, etc.)
  • Writes the above two steps to a special bash script with a randomized name that CircleCI specifies per-build, as $BASH_ENV, to run before any tests.

The second command is

for dir in $(git log --name-only --oneline -1 | sed 1d | grep '/' | cut -d "/" -f1 | sed '/^\.circleci$/d' | sort -u); do cd $dir; npm run build; npm test; npm run deploy; cd ..; done

This does the following:

  • Finds the list of files changed in the most recent commit
  • Grabs the subdirectory names for those files
  • Deduplicates the subdirectory names
  • Excludes the .circleci/ folder (which contains only config)

For each of the subdirectories changed:

  • Runs the npm scripts build, test, and deploy, in that order
  • Returns to the root directory.

This also accomplishes two important things:

  • It establishes a standard across all of our services to require build, test, and deploy, in package.json’s scripts section.
  • It relieves developers of the need to deploy to Serverless locally, as every commit will now deploy automatically to dev stage.

We run our tests using Jest, but you can substitute it for whatever you like.


#14

Based on the various answers (@Eric and @alfonsogoberjr) here I cobbled together a few lines that use the Circle provided variable, and trim that down to a list of folders to iterate over.

This has the benefit of not assuming that every commit is previously built, but rather using the commit range to collect all changes, but doesn’t depend on updates to this config to support additional paths.

          for folder in `git log --format="" --name-only ${CIRCLE_COMPARE_URL##http*/} | cut -d"/" -f1 | sort -u`; do
             echo "contents in $folder were modified"
             #this would be a call to specific testing, etc on changed directory
          done

#15

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.