I have a git monorepo, containing multiple projects, each in its own folder.
I would like to be able to:
have one Job by subfolder (the ideal would be 1 folder = 1 circle.yml file, but a single circle.yml containing every job, even if it is far from ideal, would do)
trigger the job only if the folder is affected by the commits (or the other way around, cancel / skip the build if the folder has not changed since the previous push)
I’m looking to move from Jenkins to CircleCI, and this is thg thing holding me back for the moment.
If I need to build all 10 of my projects every time I push to one of these, it will cost me a tremendous (and unnecessary) amount of time and money !
I’ve also considered moving to multiple repos with 1 project = 1 repo, but it would bring more problems than it would solve.
Add a script that accepts a project name and a string representing the command to run the tests for that project. The script uses git to check if there are changes in the given project. If it does, it runs the command given as arg, else it returns with exit 0.
Use this script instead of the normal command to run the tests
Also, do you have a way, in Circle CI, to completely skip the job ? I would like to just skip the entire job (build a Docker image, test, deploy to registry) if nothing has changed.
I could obviously run ‘run_tests.sh’ (or in my case ‘run_job.sh’) in every line but it doesn’t seem very optimal …
@tsauvajon I couldn’t find any way to skip the whole job, but I didn’t really look into it very deeply. You can skip jobs based on the branch name, or based on git tags, but that wasn’t useful to me.
Also, note that in order to run the run_tests.sh file you’ll need git if you use my version, so you’ll need to have the image built and the repo checked out. On my project this took around 1min to get to this point, per job, but this might change based on your project setup and dependencies, I suppose.
@mrcasals I got really inspired by your Circle CI config, and I’ve setup something quite similar (build a base image - checkout once for all modules - share the workspace - use yaml anchors not to copy and paste steps - run the tests or not thanks to run_tests.sh).
It doesn’t take too much time for me (new project, and only 3 modules so far). Each module takes about 5-10 seconds to setup, and 20 seconds for the base image. The total test time is about 3 minutes, and most of this time is retrieving the dependencies (golang + npm).
There are still some limitations due to how CircleCI works that are problematic for me though:
I am unable to skip a whole job, as pointed out before
I need 2 base images, circleci/node and circleci/golang. Unfortunately, golang source has to be located under /go/src/, and I can’t create the folder /go on the circleci/node. Obviously I could work this around, but don’t really have the time to do so; in the meantime, I have to build 2 environments and checkout the same code twice in order to share the workspace.
Nice, glad it helped! As you can see, we finally dropped support on conditional builds in our project, so we’re always running the whole build. We might separate the monorepo into multiple repos in the future, so this problem would be solved for us, but right now the internal APIs are not stable enough and making some changes would be a big burden.
