Verifying that changes to an Orb work within consumers

I’m working on an orb for Pantheon, a platform for Drupal and WordPress. Our initial release is pretty straightforward, it has a job for pushing code from Circle to Pantheon.

I personally have two real websites/repos and one demo site/repo consuming the orb. So when I make a PR on the orb itself I then switch my consuming sites over to using the dev release of the orb. It’s a manual and somewhat time-consuming process that I describe in greater detail over in my orb release process documentation.

I’m curious if anyone else is doing something similar? Are you also making pull requests on testing repos to switch those repos over to an orb version generated from a PR on the orb’s own repo?

@stevector Are you asking about essentially, how to automatically do integration testing on a just-published new dev release? That’s how we test our CircleCI orbs, I discuss our approach in these two blog posts:


Essentially, you make the orb’s GitHub repository double as a sample/demo project for testing purposes—so, once workflow would publish your dev orb, kicking off a second workflow that would attempt to deploy your sample/demo website to Pantheon using the dev orb that was just published.

You can see how it works in many of our orb repos, e.g.:


Hi @rose, thanks for the reply and the blog posts! I am asking about automatic integration testing of just-published dev releases.

I read those posts a few weeks ago and got hung up on thinking that for my use case it wouldn’t work to have the orb repo also be the integration test repo. Thinking it through again I’m only coming up with two blockers for why I’m reluctant to do it this way. And maybe they are surmountable.

  1. The Pantheon orb assumes a certain repository structure. Basically it assumes that the Drupal or WordPress site is either in the git root or a directory called web with a pantheon.yml file at the root. I could maybe work around this by making the orb more complex through one or more new parameter to handle alternate directory structures. I’ve already gotten a feature request for a different kind of alternative repo structure.

I’m also concerned that intermixing the integration testing Drupal and WordPress sites with the orb code makes them less approachable as example usages. But then again, the sample/demo repos don’t need to be where the integration testing happens.

  1. Naming collisions on the dev:alpha version name. I am using the orb-tools/publish-dev job but setting the alpha-version-ref param to 'dev:$CIRCLE_BRANCH' because I want to ensure that the integration tests pull in the correct release. I’m worried that if I push changes to two different branches at a similar time I’ll have a race condition where it is possible that the integration tests for one branch use the dev:alpha orb release that was created for the other branch.

Am I overthinking this?

No reason your orb repo can’t have that. It’s a fairly common situation for our orb repos—e.g., our ECR orb has some Dockerfile resources in it, etc. Won’t get in the way of the orb source code at all.

You mean for orb users? In that case, I think it’s alright, as you can always direct folks to the orb registry page, where the examples will appear uncluttered by whatever other code is in your source repository.

That’s a reasonable concern, and I certainly share it. We get around that by using @eddiewebb’s Queue orb:


https://circleci.com/orbs/registry/orb/eddiewebb/queue

Here’s an example from one of my personal orbs:

Thanks, hope this helps!

1 Like

Thanks again! I think I’m convinced. I may not get to automating this for a couple of weeks but I’ll try out putting the integration tests in the same repo with the queue orb to eliminate the race condition risk.

1 Like

Awesome, let us know how it goes @stevector! I know this flow can seem onerous/complicated, so we’re definitely always looking for ways we can improve it.

(Also, this should help a lot, when it’s ready: https://github.com/CircleCI-Public/api-preview-docs/blob/master/docs/api-changes.md#run-only-a-specific-workflow-in-a-triggered-pipeline)