Circle.yml macros/includes?

circle.yml

#1

TL;DR - Are there any plans to add macro-like behaviour (or maybe “includes”) to circle.yml syntax?

We’ve started switching over a bunch of our stuff to Circle 2.0, and generally things are great. However, as we’re largely mono-repo focused (and using Gradle subprojects), there’s a fair amount of copy-and-paste duplication in our circle.yml, that would conceptually be greatly improved with metaprogramming.

Right now, we’re manually keeping circle.yml in sync with our project structure, but that doesn’t scale. We’re also considering writing a script that auto-generates our circle.yml, but it feels like it would be cleaner if we could do that at CI time. Is this a thing that might ever be possible? (Alternatively, are there other solutions to this problem?)

To illustrate, here’s one example of the duplication (where A, B, C are subprojects):

- restore_cache:
    keys:
      - platform-{{ .Branch }}-A-yarn-{{ checksum "A-frontend/yarn.lock" }}
      - platform-{{ .Branch }}-A-yarn
      - platform-develop-A-yarn

- restore_cache:
    keys:
      - platform-{{ .Branch }}-B-yarn-{{ checksum "B-frontend/yarn.lock" }}
      - platform-{{ .Branch }}-B-yarn
      - platform-develop-B-yarn

- restore_cache:
    keys:
      - platform-{{ .Branch }}-C-yarn-{{ checksum "C-frontend/yarn.lock" }}
      - platform-{{ .Branch }}-C-yarn
      - platform-develop-C-yarn

# ...

- save_cache:
    key: platform-{{ .Branch }}-A-yarn-{{ checksum "A-frontend/yarn.lock" }}
    paths:
      - A-frontend/node_modules

- save_cache:
    key: platform-{{ .Branch }}-B-yarn-{{ checksum "B-frontend/yarn.lock" }}
    paths:
      - B-frontend/node_modules

- save_cache:
    key: platform-{{ .Branch }}-C-yarn-{{ checksum "C-frontend/yarn.lock" }}
    paths:
      - C-frontend/node_modules

# ...

#2

We also need some kind of include, to allow us to keep individual user tokens, emails, and the like outside of version control while keeping the main CI info in.


#3

FWIW, we eventually sidestepped this issue by running Yarn via Gradle, so we now only need to cache the single Gradle cache directory (which in turn caches all of the node_modules dirs).


#4

@choliver Check out “aliases” as can be seen in the Facebook Jest config.yml. I can’t find them in documentation, so I assume they’re “beta”. I tried it in one of my repos, and it works beautifully.


#5

That’s not a CircleCI feature – it’s baked into YAML. It should work with any standard YAML, and is definitely a handy way to reduce duplication. There are some annoying limitations such as the inability to nest references. We are looking into a more robust approach for sharing configuration across jobs and across projects, but that’s in the R&D stage at the moment, so nothing firm to share. The issue of mono-repos is also something we have talked about, but that’s also not yet slated for first-class solutions.


#6