Introduction
Greetings! I’m a staff engineer working on the API/infrastructure team at a large enterprise. My fellow engineers and I like CircleCi sufficiently to use it for continuous integration, currently with five microservices. Additional services are on the horizon even now, with many more sure to follow.
The Problem
With five different microservices all written in node.js, we find ourselves sharing more than 90% of the CI logic between these services. To get CI for all five repos we can duplicate one CircleCI config into each repository, then make minor changes – mostly to the same areas of the config file – before committing those changes to each repository in turn.
This approach is repetitive and error-prone. I’ve crafted a relatively simple solution – combining two individual files to form a single YAML configuration file – to share most of the logic between CircleCi configs in different repositories, however my approach is sub-optimal to say the least and is hindered by limitations in the CircleCI platform.
The purpose of this post is to highlight CircleCI’s shortcomings regarding enterprise scale and to offer suggested solutions to those shortcomings.
Barriers to Enterprise Usage
- You can’t share CircleCI configuration logic between repositories. This is huge for us, and will become increasingly important as time goes on.
- You can’t easily share state between steps in a single job (it’s possible, but it involves hacks that don’t feel right and that are hard to maintain).
- No ACLs for approval jobs. Anyone can approve any manual step and there are no logs indicating who approved what unless you run CircleCI locally (we really don’t want to host this ourselves). When this limitation was discovered, the head of our operations department took a step back in disbelief. She isn’t at all interested in allowing just anyone to pull the trigger to deploy to production – which she is directly responsible for – with no accountability whatsoever. I am expecting a hard fight going forward to keep our CircleCI paid account purely as a consequence of this point.
- No built-in support for multiple AWS key pairs. There is a fairly easy workaround, but it shouldn’t be necessary and a single change ultimately requires maintenance to all microservice CircleCI config files.
My Stop-Gap Solution
I’ve whipped up a repository that concatenates a shared.yml file – containing all shared YAML anchors and job steps – with each individual service configuration file. The file structure for this simple repo looks like this:
root/
root/shared.yml
root/services/
root/services/microservice-n/
root/services/microservice-n/config.yml
root/services/...
root/update.sh
shared.yml contains all shared YAML anchors and CircleCI job steps (imported as aliases).
root/services/ houses a subdirectory for each microservice.
microservice-1 contains a config.yml file representing the CircleCi jobs and steps unique to this service.
update.sh is a shell script that regenerates each microservice CircleCI config.yml file in the specified repositories path, concatenating shared.yml with each individual config.yml file to produce a complete config for each microservice.
This approach has many limitations and smells of bad design. Firstly, all of my CircleCI source configurations must live in a separate repository divorced from the microservices that ultimately require them. This violates at least one core software engineering principle and requires extra “magical” configuration outside of the code itself, which can be lost or forgotten.
This hacked-together solution does work, for now, and it’s certainly more maintainable than duplicating all of the CircleCI steps in every repository. I can produce an example repository if anyone is interested that shows you precisely how to generate multiple CircleCI configs from shared + unique content. There are other, more specific technical limitations with this approach but I’ll avoid discussing them here since this is a sub-optimal solution for the core problem anyway.
Conclusion
I’m going to have a pretty hard time justifying us paying for CircleCI for very much longer if we don’t get access controls for approval jobs in a hurry. I’m also going to have a pretty rough time maintaining our CircleCI configs for multiple microservice repositories using my hacked-together solution, unless the wonderful folks working at CircleCI can come up with a clever way to let us share logic at an organization level. 
I hope this post was helpful for someone, somewhere! Please let me know if you would like clarification on any of the issues discussed.
Cheers!
