Testing orbs

Ecosystem Orbs
1
  1. Is there a recommended approach for testing orbs?
  2. Is there a way for a local .circle/config.yml to reference an orb in the same repository?

Example repo:

.circle/config.yml
src/git/orb.yml
1 Like
2

Hi @juanca,

It is possible to test local orbs, but involves a bit of trickery. For initial development you can simply “inline” the orb in your config.yml - https://circleci.com/docs/2.0/creating-orbs/#creating-inline-orbs

For automated testing, we have some general guidance in our SDK repo, https://github.com/CircleCI-Public/config-preview-sdk/blob/987fd917522f5d5d9a2cae145878fc548e9ea79e/docs/orbs-testing.md

But the tooling is left open to authors. I’ve created an opinionated flow that works for my purposes, and that can be seen in this sample repo - https://github.com/eddiewebb/circleci-dmz-orb/blob/master/.circleci/config.yml

I chose BATS as the framework to drive my expansion testing, and use a number of helpful functions to enable the assembly of configuration, as well as assertions. https://github.com/eddiewebb/circleci-dmz-orb/blob/7ac0e063781d462878ae7e6c1477b3116ac4fd67/test/test_expansion.bats

The utilities convert the processed YAML to JSON so that I can run assertions on specific elements using JQ, https://github.com/eddiewebb/circleci-dmz-orb/blob/7ac0e063781d462878ae7e6c1477b3116ac4fd67/test/test_expansion.bats#L35,L40

  assert_jq_match '.jobs | length' 1  #only 1 job
  assert_jq_match '.jobs["build"].steps | length' 5  # it contains 5 steps
  assert_jq_match '.jobs["build"].steps[3].run.command' 'ssh -L 9001:104.154.89.105:80 -Nf ubuntu@ec2-18-191-19-150.us-east-2.compute.amazonaws.com || true'  #the 4th is my compiled command
  assert_jq_contains '.jobs["build"].steps[2].run.command' 'KEY_VALUE=`cat somefile`'
  assert_jq_contains '.jobs["build"].steps[2].run.command' 'KEY_VALUE=`echo "somefile"`'
  assert_jq_contains '.jobs["build"].steps[2].run.command' 'echo "ec2-18-191-19-150.us-east-2.compute.amazonaws.com ${KEY_VALUE}" >> ~/.ssh/known_hosts'

You can also run builds with the configuration locally using the CLI, or as part of your workflow using our orb tooling., https://github.com/CircleCI-Public/orb-tools-orb/blob/master/src/%40orb.yml#L107

That creates a job in the workflow that expects input configuration to be tested against the orb, and allows the output, status and container state to be inspected afterwards.

This is all still evolving, so we would love any feedback you have!

2 Likes
3

I appreciate your response! Those are great resources and I’m going to go through each and experiment with all 4 types of testing strategies.

Update: resolved. See end of post for more information.

I’m trying out the validation tests. It seems I can locally validate my orb but cannot on the CI box.

Local:

~/workspace/orbs (tests) $ circleci version
0.1.3923+6fecf9d
~/workspace/orbs (tests) $ circleci orb validate src/git/orb.yml
Orb at `src/git/orb.yml` is valid.

CI:

root@9b77f54f4845:~# circleci version
Build Agent version: 0.1.1062-dce5e86e
built: 2018-11-10T01:11:11+0000

root@9b77f54f4845:~# circleci orb validate src/git/orb.yml
Error: unknown command "orb" for "circleci"
Run 'circleci --help' for usage.

Is there a way to update the CLI version on CI? Or perhaps I am using the wrong base image?

version: 2.1

executors:
  base:
    docker:
      - image: debian:stretch

workflows:
  schema-validations:
    jobs:
      - git-schema-validation

jobs:
  git-schema-validation:
    executor: base
    steps:
      - run: circleci orb validate src/git/orb.yml

Update: There exists two useful orbs (from CircleCI) that facilitate testing:

  1. orb-tools
  2. circleci-cli

You will probably want to use the circleci-cli to install the CLI or use its executor with the preinstalled CLI. You will also want to use the built-in commands / jobs in orb-tools for processing the orb yml files.

2 Likes
4

After two days of fiddling, here are my thoughts – note some of these can be seen throughout my commit history in this PR:

  1. Schema validation

This seems to be the easiest way to automate some basic tests for validity. I would recommend checking out the circleci/circleci-cli and circleci/orb-tools orbs.

  1. Expansion testing

The concept is pretty straight-forward. Essentially, it validates the schema of a configuration that utilizes the orb under test. However, putting it in an automated CI pipeline requires a bit of work due to CLI limitations. I might be able to simplify the work by either (a) utilizing the circleci config pack functionality, (b) referencing a local orb file, or © circleci config processing a file the exact same way the “build processing” feature works.

I haven’t used this kind of test to assert on the structure of the resulting YAML file. As of this writing, I am favoring end-to-end tests because my git orb uses an external CLI and I do not see value in asserting string patterns.

  1. Runtime testing

This seems to be the cake for my use-case. I would like to make a build with a few different jobs (that test the commands, jobs, and parameters of my orb). Ideally, my assertions would test the output of some steps – essentially printing the state of the machine and asserting some values.

I think a few things should be noted for this section. This is copy pasta from my other thread:

  1. Job must be a machine executor – otherwise the docker daemon is not properly setup.
  2. You must install the latest circleci CLI – easiest way to do that is by using the circleci/circleci-cli orb.
  3. Make steps that executes the build with a processed configuration file:

There is probably a way to get this workin with orb-tools but I wasn’t able to grok it initially and went with hand rolling my own implementation for the time being. I will try to refactor back into orb-tools.

While this approach provides me with the most value, I realized the assertions and messaging patterns are not obvious. It would be nice to see some improvements on the syntax (perhaps a command?) which allows for flexible assertions and error messaging.

  steps:
  - git/checkout # which is my orb under test
  - verify:
      actual: git log --pretty=form:'' | wc -l
      expected: 1
      message: Expected git history to be truncated to 1 commit but got
  1. Integration testing

I have not a use case for this… yet.

2 Likes
5

If you create an orb, but only publish a dev version, will it show in the registry? Will it disappear from the registry if the dev orb expires without having been published?

For testing, it would be nice to be able to publish a dev version of someone else’s orb to your own namespace, even before there is full support for private orbs.

6

@wyardley great questions!

If you create an orb, but only publish a dev version, will it show in the registry?

Dev versions will not display in the registry UI, but they do get published to the registry. This is how we do our standard orb validation/publishing flow (see Emerging testing best practices for Orbs)

Will it disappear from the registry if the dev orb expires without having been published?

Yes, exactly. See Creating Orbs:

Dev versions are mutable and expire: their contents can change, and they are subject to deletion after 90 days.

For testing, it would be nice to be able to publish a dev version of someone else’s orb to your own namespace, even before there is full support for private orbs.

You can definitely do this! :slight_smile:

You’ll need to run circleci orb source $NAMESPACE/$ORB@$VERSION to get the orb source, and get it into a file somehow (e.g., circleci orb source $NAMESPACE/$ORB@$VERSION > orb.yml). From there, you can publish it to your own namespace (if an orb of that name doesn’t exist, you’ll have to create the orb in your namespace first, as well—or you could do something like create a $YOUR_NAMESPACE/sandbox orb, to be used specifically for publishing dev versions of other people’s orbs to your own namespace).

To be clear, @dev versions of orbs are not private—anyone can reference and use them. However, they will not be shown in the registry, and, as far as I know, there is no easy way to get a list of @dev versions of a particular orb via the CircleCI CLI. So, if you keep the details of your @dev:version orb relatively secret, it may serve that purpose for you in a pinch.

1 Like
7

Hi Rose!

How can I access published dev orb?

publishing:

$ circleci orb publish orb.yml proxyco/semantic-release@dev:first
Orb `proxyco/semantic-release@dev:first` was published.
Please note that this is an open orb and is world-readable.
Note that your dev label `dev:first` can be overwritten by anyone in your organization.
Your dev orb will expire in 90 days unless a new version is published on the label `dev:first`.

Usage:

version: 2.1
orbs:
  semantic-release: proxyco/semantic-release@dev:first

Build error:

#!/bin/sh -eo pipefail
# Error calling workflow: 'all'
# Cannot find orb 'proxyco' looking for job named 'proxyco/semantic-release'
# 
# -------
# Warning: This configuration was auto-generated to show you the message above.
# Don't rerun this job. Rerunning will have no effect.
false
Exited with code exit status 1

https://circleci.com/gh/proxyco/docker-lambda-deploy/628

What am I doing wrong?!

8

I’m having a similar issue.

I published my orb

circleci orb publish @orb.yml dalekurt/aws-ecr@dev:alpha

Which displayed the output

Orb `dalekurt/aws-ecr@dev:alpha` was published.
Please note that this is an open orb and is world-readable.
Note that your dev label `dev:alpha` can be overwritten by anyone in your organization.
Your dev orb will expire in 90 days unless a new version is published on the label `dev:alpha`.

Then updated my CircleCI config

version: 2.1

orbs:
  aws-ecr: dalekurt/aws-ecr@dev:alpha

Then I use circlceci local execute --job "build & push (development)

I get the error

Error: Error calling workflow: 'push'
Cannot find a definition for job named aws-ecr/build-and-push
9

Hi @dalekurt and @BeyondEvil I’m having a similar issue. I’m wondering if you could find a solution for it?

10

Welcome to the community @mf69! Do you have the same error?

11

Hi @thekatertot thanks :slight_smile: Yes when I run command: $ circleci config validate , I get this error: Cannot find a definition for job named … I have created a question here with more details if you like to check it out: https://stackoverflow.com/questions/63230920/circleci-validation-using-orb-cannot-find-a-definition-for-job

1 Like
12

Hello :wave:,

I have responded on StackOverflow but will copy my response here and expand:

Hello,

In the example you have provided, you are defining commands and naming them as if they were jobs.

The error is stating there is no jobs under that name, as there is not. There is however commands.

Commands must be used within jobs.

https://circleci.com/docs/2.0/orbs-faq/#difference-between-commands-and-jobs

How to author a job: https://circleci.com/docs/2.0/reusing-config/#authoring-parameterized-jobs

Live Examples:

Node “test” Job: https://github.com/CircleCI-Public/node-orb/blob/master/src/jobs/test.yml

Node “install-packages” Command: https://github.com/CircleCI-Public/node-orb/blob/master/src/commands/install-packages.yml

Jobs define what steps/commands to run within a given executor, like a docker image. A command only defines a list of steps. A command must be used within the context of a job.

The Orb Concepts - Configuration Elements page may help: https://circleci.com/docs/2.0/orb-concepts/#orb-configuration-elements

2 Likes