Emerging testing best practices for Orbs

Ecosystem Orbs
1

Wanted to quickly share some best practices that we have been developing internally around testing orbs, before I sign off for the weekend—

The biggest issue in testing orbs is that, because orbs are interpolated into an expanded config.yml at build inception, it is not straightforward to push a new commit to a repository containing orb source code (assuming the repo is configured with CI/CD) and have the newest changes to the orb that are contained in that commit, also be tested, as a new dev version of the orb, within the same commit.

Some previous approaches have included using inline orbs, or external repositories, and while the approach I’ll outline here is not perfect, I do think it is simpler and more functional than either of the former.

If you’d rather jump right into examples, here are three recent, fairly similar implementations:

Their common characteristics are:

  • Two workflows, one triggered on each commit, another triggered only by git tags
  • First workflow contains basic testing: linting, validation, etc.
  • First workflow concludes by publishing a dev:alpha version of the orb, then pushing a git tag back to the GitHub repository, which kicks off the second workflow
  • The second workflow, which builds on the same commit as the first, is technically a new build in our internal system, so it can use the same new dev:alpha version of the orb that was just published
  • The second workflow can simply run all the orb’s jobs/commands, to ensure they function as expected, or do additional testing using the orb
  • If those jobs succeed, the second workflow concludes by promoting the previously published new dev version of the orb to a semantic production release

Some caveats/specifics:

  • This setup requires creating a read/write deploy key, with the public portion stored in GitHub and the private portion stored in CircleCI
  • You can use the compare URL orb to control whether or not a patch, minor, or major release of the orb is eventually published, depending on what aspects of orb source are modified (for example, if there are new/altered jobs or commands, a major release; if there are new examples or descriptions, a minor release; if only the repo’s README or config.yml file was altered, a patch release—or none at all, depending on your preference)
  • This strategy assumes a 1:1 relationship between an orb and a GitHub repository—I can think of some ways it could work in an Orbs monorepo, like ours, but it would be more complex…
  • For this to work, there also needs to be a single initial dev:alpha release of the orb published manually, to bootstrap the process

As these practices firm up, we’ll look into abstracting some of this logic out into some kind of orbs testing orb (or adding it to the existing orb-tools orb), but for now, you should be able to achieve the same functionality by adapting any of the example config files shared above :slight_smile:

And also please feel free to share any other orbs testing practices you’ve been developing!

4 Likes
2

I jotted some notes of my experience with testing orbs @ Testing orbs

The gist:

I set up my CI pipeline to automatically publish my dev orb and then use it in a subsequent job (but same workflow) with a dynamically generated configuration file. I need to iterate on this script and publish a release for green master builds but I haven’t had an orb-related task in my daily work activities.

2 Likes
3

Yes, this is another smart solution! @juanca

4

@rose thanks for sharing this.

I think it would be a good idea to use a semantic version cli calculator to derive the decision of patch | minor | major from the commit messages in the given commit range.

But I’m off-hand not aware of such a tool, that can output the change type to stdout based on the commit messages of a given commit range.

1 Like
5

@jasonkuhrt That’s another good idea… Would certainly be possible! Just a bit of git & CircleCI scripting. An orb could make it simple, in fact!

6

Maybe I’m missing it, but I don’t see a way to reference an orb from a git repo. I’d like to contribute to CircleCI’s orbs, but it looks like I’m going to have to publish my own orb in order to test it out. It would be great if I could instead point at a Github repo and commit in order to test out an orb.