Building Monorepo Projects With CircleCI

go
circle.yml

#1

I’ve been experimenting with setting up our code repo to build with CircleCI and I’m finding that the system isn’t particularly well set-up for our monorepo codebase. We have a few projects I’d like to build inside our single, larger repository, but I’m not sure how to handle this kind of project since you can only specify a single build directory. Right now I’m thinking that I’ll need to individually specify all of the test/build commands to run inside each individual application’s subfolders in the test:override: command of the circle.yml file, but that does come with two issues:

  • missing out on all of the nice automatic configuration magic
  • a slower build which executes every build for every commit

Admittedly the second point isn’t necessarily a problem, and there’s a strong argument to be made that a monorepo should be built in totality. But that aside, configuring a monorepo build isn’t the clearest of processes right now.

Ideally I’d like either a) a nice way to break up my single repository into multiple projects or b) better documentation of how I should approach setup for a multiapp repository. Any guidance would be appreciated!


Does CircleCI 2.0 supports building a project separately in a Multi-Project repo
#2

We’ve actually just switched over to a monorepo structure for our packages (coordinating between separate repos was becoming too annoying), and had the same experience as you in terms of configuration. FWIW, here’s what we did:

  • bit the bullet on point 1
  • wrote a little tool that only builds packages that have changed (or have dependencies that have changed)

Either way, agree: would be nice to have recommendations for how to approach using CircleCI for a monorepo


#3

Oh nice, we were actually looking into building some kind of change watcher to act like that; any chance you’ve open-sourced that tool?


#4

We based ours on https://github.com/kittens/lerna. There a good bit of Node-specific versioning stuff (dealing with changing versions in package.json and then publishing those to the npm registry).

We were first planning on contributing our changes back up to lerna, and ultimately got to the point where our tool needed to be pretty different/custom in purpose. Our plan is to wait and see where it ends up before we open source it.

But, I think lerna is a good starting point in terms of approach, and if you decide to go in that direction can definitely talk through what we decided to change and why


#5

Interesting…I actually remember looking into Lerna very lightly earlier this year, seems worth a second look. One thing that I think drove me away from it initially was the lack of support for multilanguage projects. Is your team a pure-JS stack or did you find another way around this particular issue?


#6

@troyastorino, if you were using Lerna, how would you go about making Circle only running the tests for the packages that changed? If I understand correctly, Lerna will build all of the packages that changed and publish them to NPM. Does the bootstrap command run the packages’ tests as well?


#7

+1 support for several go (golang) programs in 1 monorepo would be great!


#8

You can do it no problem actually, you just have to manually specify the commands to run for each program instead of letting Circle just infer it. We’ve got it running right now as individual entries under test.override that run a script which runs a build and executes our tests on each individual application.


#9

does anyone have this working for golang? what tools/script do you use to only rebuild/recompile those programs whose packages (or dependent packages) have changed? our problem right now is that we’re always rebuilding all our services and bumping their versions even when a code change only affected one of them.


#10

We trigger CircleCI builds via Jenkins using CircleCI API. What needs to be built is worked out by our modified Github and Github Pull Request Builder plugins (based on git diff between branch/PR and master) and then the path within the repo is passed as a build parameter to CircleCI. Our circle.yml just acts as a proxy, executing the same targets for every package that can be built. It works very well.


#11

Do you have a link to the tools you use to run this process? I’m really interested in how this process runs when all is said and done.


#12

We might publish a paper about this soon. You can have a look at the modification we made to GHPRB plugin for as start:


#13

The only problem with this statement, is that lerna support interdependencies. So technically, if only package A changed, it could still potentially impact packages B and C, which depend on A.
Additionally, there could be nested dependency logic. It seems to me that it’s safer to test everything, unless logic could be written to account for the dependency checking.


#14

@avindra, You are correct!

Like @moberemk mentioned in the parent comment,

there’s a strong argument to be made that a monorepo should be built in totality

I think that if you can handle dependencies well and make sure to test interdependent repositories where the dependencies changes then you’ll be fine.

If for example you have 10 packages and you update one that happens to touch every other one but the the other 9 packages still declare a dependency to the old version, then it would be ok to just test the 1.

On the other hand, if you change the one package and then also update al other 9 to use the newest version of the updated dependency then indeed all 10 packages changed and you would want to test them all.

It seems to me that looking at each package individually and checking for changes (even in the package.json's dependency) would be good enough to to know what did and didn’t change. If you’re using something like yarn or npm-shrinkwrap or strictly following semver you should be fine.

Back when I made this comment Lerna was super early. I’d love to hear how people have gone about managing monorepos in CircleCI since then!


#15

Same. Seems like most projects are using Makefile and build script hacks. Btw there’s talks among the leads of yarn and lerna about lerna being absorbed by yarn. Maaaaaybe something will come on their side for better CI support during the merge.


#16