Testing orbs

Ecosystem Orbs
4

After two days of fiddling, here are my thoughts – note some of these can be seen throughout my commit history in this PR:

  1. Schema validation

This seems to be the easiest way to automate some basic tests for validity. I would recommend checking out the circleci/circleci-cli and circleci/orb-tools orbs.

  1. Expansion testing

The concept is pretty straight-forward. Essentially, it validates the schema of a configuration that utilizes the orb under test. However, putting it in an automated CI pipeline requires a bit of work due to CLI limitations. I might be able to simplify the work by either (a) utilizing the circleci config pack functionality, (b) referencing a local orb file, or © circleci config processing a file the exact same way the “build processing” feature works.

I haven’t used this kind of test to assert on the structure of the resulting YAML file. As of this writing, I am favoring end-to-end tests because my git orb uses an external CLI and I do not see value in asserting string patterns.

  1. Runtime testing

This seems to be the cake for my use-case. I would like to make a build with a few different jobs (that test the commands, jobs, and parameters of my orb). Ideally, my assertions would test the output of some steps – essentially printing the state of the machine and asserting some values.

I think a few things should be noted for this section. This is copy pasta from my other thread:

  1. Job must be a machine executor – otherwise the docker daemon is not properly setup.
  2. You must install the latest circleci CLI – easiest way to do that is by using the circleci/circleci-cli orb.
  3. Make steps that executes the build with a processed configuration file:

There is probably a way to get this workin with orb-tools but I wasn’t able to grok it initially and went with hand rolling my own implementation for the time being. I will try to refactor back into orb-tools.

While this approach provides me with the most value, I realized the assertions and messaging patterns are not obvious. It would be nice to see some improvements on the syntax (perhaps a command?) which allows for flexible assertions and error messaging.

  steps:
  - git/checkout # which is my orb under test
  - verify:
      actual: git log --pretty=form:'' | wc -l
      expected: 1
      message: Expected git history to be truncated to 1 commit but got
  1. Integration testing

I have not a use case for this… yet.

2 Likes