Reusable commands and jobs - a rant with questions

Build Environment
,
1

I feel like I am very confused by CircleCI. There appears to be a very strong impulse within CircleCI to explicitly define every single step, and to not do any form of reuse. I have two workflows that literally differ by a single line, but, based on everything I’ve read and searched for, it is preferable to have an extra hundred lines of repeated steps over a parameter. I am so confused as to what the difference between the commands, jobs, steps, workflows, pipelines and why none of it seems to be very well documented. It’s always, “here’s the ‘hello world’ of ‘commands’, good luck!” Can someone please explain this to me? Why are reusable steps and commands seemingly frowned on? Also, why is the circleci CLI so…frustrating?

1 Like
2

I am not going to disagree with anything you have stated as the docs do seem to leave someone with a steep learning curve and provide few complete examples.

I think in part this is because the CI system is designed for use by a DevOps team - so it all reads as here are the features we have, now map them to the way that you wish to operate. So depending on the languages and tools being used by the DevOps team they can choose how they write config.yml files.

As an example, I took one look at ‘commands’ and mapped them to functions/procedures and have parameters being passed around all the time. Someone else may instead just use the underlying shell and environment variables.

Within the docs the best starting place is

1 Like
3

One thing to note is that CircleCI has no concept of a return value, so while you can pass parameters to commands and jobs you can not simply return a result, instead you have to drop down to the shell and pass environment variables back using whatever the solution is for the shell being used.

4

You’re not wrong, but shell envs would be completely reasonable to use for me, but they don’t appear to be usable in my case. They can only be used within the steps of a singular job. So, my issue is that I have an orb that in order to use, needs to have three values. So, ideally, I’d implement something in a “commands” section and have parameters to use. Well, that didn’t work. Ok, let’s try setting the value as a shell env and just grab it when the call gets made. Nope. The orb can only be called from the “workflow” section (for some reason that I have yet to find any documentation for), I have to have nine different calls to the orb with only one line changing in each. If this was Jenkins, this would be dead simple. It is very clear to my mind that this should be possible, but the documentation of configuration options is just not adequate.

1 Like
5

One issue you may be having is that I think the terms used in Jenkins and CircleCI are often the same, but mean very different things (it’s many years since I last even looked at Jenkins).

One key thing is that in CricleCI each job is run in isolation as a separate system instance, so if you want to operate a singular instance where data can be easily shared you define a single job and place all the work you wish to perform into that single job. The result is that under workflows: you are basically doing condition checks to decide what parameters are past to the single defined job. CircleCI does provides workspace persist tools for moving info between jobs in a workflow run and a caching solution for moving data between workflow runs, but those are far more complex.

When using environment variables for storing values the issue is that for each ‘run: command:’ a new shell is created, so just setting an environment variable or even exporting the environment variable is not enough. Such information is lost when the shell is closed and the execution path returns to the calling shell. Instead, the following has to be done

 https://circleci.com/docs/set-environment-variable/

The one thing I can not easily comment on is the issue you are having with the ORB, as calling an ORB from within a statement is not normally an issue, which ORB are you trying to use?

As a working example, the best I can do is the following, which is the single defined job within one of my build config.yml files

jobs:
  build-runner-latest-from-branch:
#    executor: hosted_node
    executor: exec_node
    parameters:
      doppler_token:
        type: string
    steps:
      - clean_up_workspace
      - clean_up_docker_cache
      - show_system_details
      - checkout
      - generate_info_file
      - build_application_and_container:
          doppler_token: "<<parameters.doppler_token>>"
      - docker_login:
          doppler_token: "<<parameters.doppler_token>>"
      - docker_tag_image:
          doppler_token: "<<parameters.doppler_token>>"
      - docker_push_image:
          doppler_token: "<<parameters.doppler_token>>"
      - deploy_to_target:
          doppler_token: "<<parameters.doppler_token>>"
      - slack_general_status_report
      - clean_up_docker_cache
      - clean_up_workspace

So

  • At the workflow: level all I do is work out what branch needs to be built.
  • The job above is called with a doppler_token parameter set at the workflow level
  • All the steps I have created in the job are defined as commands with the doppler_token being passed on to some of the commands.
    – Side note - I normally use a self-hosted runner, so many of the steps above are there to correctly handle the fact that I have built-in persistence between runs as the same system instance is used for each run.

To be honest there is a chance that the way I write scripts for CircleCI is somewhat unique as all my secrets and variables are defined via a third-party service called Doppler. So the above example shows a single value being passed around, but that is the token that allows 100’s of values to be retrieved based on which token was selected at the workflow: level.

1 Like
6

You may find our JS/TS config SDK interesting: GitHub - CircleCI-Public/circleci-config-sdk-ts: Generate CircleCI Configuration YAML from JavaScript or TypeScript. Use Dynamic Configuration and the Config SDK together for live generative config.

This will let you abstract any part of CircleCI config into JS modules that you can import/export or even turn into other NPM modules.

Here’s an example of creating a “matrix” workflow, which defines “reusable” jobs as JS functions.

So this example doesn’t reuse workflows but it works exactly the same way, you could absolutely do it.

In this example I define the workflow “statically” as:

const myWorkflow = new CircleCI.Workflow('my-workflow');

You could instead do something like

const myWorkflow = (name: string) => new CircleCI.Workflow(name);
1 Like