Testing orbs


#1
  1. Is there a recommended approach for testing orbs?
  2. Is there a way for a local .circle/config.yml to reference an orb in the same repository?

Example repo:

.circle/config.yml
src/git/orb.yml

#2

Hi @juanca,

It is possible to test local orbs, but involves a bit of trickery. For initial development you can simply “inline” the orb in your config.yml - https://circleci.com/docs/2.0/creating-orbs/#creating-inline-orbs

For automated testing, we have some general guidance in our SDK repo, https://github.com/CircleCI-Public/config-preview-sdk/blob/987fd917522f5d5d9a2cae145878fc548e9ea79e/docs/orbs-testing.md

But the tooling is left open to authors. I’ve created an opinionated flow that works for my purposes, and that can be seen in this sample repo - https://github.com/eddiewebb/circleci-dmz-orb/blob/master/.circleci/config.yml

I chose BATS as the framework to drive my expansion testing, and use a number of helpful functions to enable the assembly of configuration, as well as assertions. https://github.com/eddiewebb/circleci-dmz-orb/blob/7ac0e063781d462878ae7e6c1477b3116ac4fd67/test/test_expansion.bats

The utilities convert the processed YAML to JSON so that I can run assertions on specific elements using JQ, https://github.com/eddiewebb/circleci-dmz-orb/blob/7ac0e063781d462878ae7e6c1477b3116ac4fd67/test/test_expansion.bats#L35,L40

  assert_jq_match '.jobs | length' 1  #only 1 job
  assert_jq_match '.jobs["build"].steps | length' 5  # it contains 5 steps
  assert_jq_match '.jobs["build"].steps[3].run.command' 'ssh -L 9001:104.154.89.105:80 -Nf ubuntu@ec2-18-191-19-150.us-east-2.compute.amazonaws.com || true'  #the 4th is my compiled command
  assert_jq_contains '.jobs["build"].steps[2].run.command' 'KEY_VALUE=`cat somefile`'
  assert_jq_contains '.jobs["build"].steps[2].run.command' 'KEY_VALUE=`echo "somefile"`'
  assert_jq_contains '.jobs["build"].steps[2].run.command' 'echo "ec2-18-191-19-150.us-east-2.compute.amazonaws.com ${KEY_VALUE}" >> ~/.ssh/known_hosts'

You can also run builds with the configuration locally using the CLI, or as part of your workflow using our orb tooling., https://github.com/CircleCI-Public/orb-tools-orb/blob/master/src/%40orb.yml#L107

That creates a job in the workflow that expects input configuration to be tested against the orb, and allows the output, status and container state to be inspected afterwards.

This is all still evolving, so we would love any feedback you have!


#3

I appreciate your response! Those are great resources and I’m going to go through each and experiment with all 4 types of testing strategies.


Update: resolved. See end of post for more information.

I’m trying out the validation tests. It seems I can locally validate my orb but cannot on the CI box.

Local:

~/workspace/orbs (tests) $ circleci version
0.1.3923+6fecf9d
~/workspace/orbs (tests) $ circleci orb validate src/git/orb.yml
Orb at `src/git/orb.yml` is valid.

CI:

root@9b77f54f4845:~# circleci version
Build Agent version: 0.1.1062-dce5e86e
built: 2018-11-10T01:11:11+0000

root@9b77f54f4845:~# circleci orb validate src/git/orb.yml
Error: unknown command "orb" for "circleci"
Run 'circleci --help' for usage.

Is there a way to update the CLI version on CI? Or perhaps I am using the wrong base image?

version: 2.1

executors:
  base:
    docker:
      - image: debian:stretch

workflows:
  schema-validations:
    jobs:
      - git-schema-validation

jobs:
  git-schema-validation:
    executor: base
    steps:
      - run: circleci orb validate src/git/orb.yml

Update: There exists two useful orbs (from CircleCI) that facilitate testing:

  1. orb-tools
  2. circleci-cli

You will probably want to use the circleci-cli to install the CLI or use its executor with the preinstalled CLI. You will also want to use the built-in commands / jobs in orb-tools for processing the orb yml files.


#4

After two days of fiddling, here are my thoughts – note some of these can be seen throughout my commit history in this PR:

  1. Schema validation

This seems to be the easiest way to automate some basic tests for validity. I would recommend checking out the circleci/circleci-cli and circleci/orb-tools orbs.

  1. Expansion testing

The concept is pretty straight-forward. Essentially, it validates the schema of a configuration that utilizes the orb under test. However, putting it in an automated CI pipeline requires a bit of work due to CLI limitations. I might be able to simplify the work by either (a) utilizing the circleci config pack functionality, (b) referencing a local orb file, or © circleci config processing a file the exact same way the “build processing” feature works.

I haven’t used this kind of test to assert on the structure of the resulting YAML file. As of this writing, I am favoring end-to-end tests because my git orb uses an external CLI and I do not see value in asserting string patterns.

  1. Runtime testing

This seems to be the cake for my use-case. I would like to make a build with a few different jobs (that test the commands, jobs, and parameters of my orb). Ideally, my assertions would test the output of some steps – essentially printing the state of the machine and asserting some values.

I think a few things should be noted for this section. This is copy pasta from my other thread:

  1. Job must be a machine executor – otherwise the docker daemon is not properly setup.
  2. You must install the latest circleci CLI – easiest way to do that is by using the circleci/circleci-cli orb.
  3. Make steps that executes the build with a processed configuration file:

There is probably a way to get this workin with orb-tools but I wasn’t able to grok it initially and went with hand rolling my own implementation for the time being. I will try to refactor back into orb-tools.

While this approach provides me with the most value, I realized the assertions and messaging patterns are not obvious. It would be nice to see some improvements on the syntax (perhaps a command?) which allows for flexible assertions and error messaging.

  steps:
  - git/checkout # which is my orb under test
  - verify:
      actual: git log --pretty=form:'' | wc -l
      expected: 1
      message: Expected git history to be truncated to 1 commit but got
  1. Integration testing

I have not a use case for this… yet.