Running additional containers for Orb job

Ecosystem Orbs
1

Using my rebar3 orb https://circleci.com/orbs/registry/orb/tsloughter/rebar3 is less helpful when a database or any other external service that is run in a container is needed by the tests that are run.

Instead of being able to use the job from the orb it requires redefining the job with the database added to images:

jobs:
  ct:
    docker:
      - image: circleci/erlang:22
        entrypoint: ["/bin/sh"]
      - image: circleci/postgres:11-alpine-ram
        environment:
          POSTGRES_USER: test
          POSTGRES_DB: test
    steps:
      - checkout
      - rebar3/with_deps_cache:
          steps:
            - rebar3/ct
      - rebar3/on_fail_store_crashdump:
          cmd: ct
      - store_test_results:
          path: ~/project/_build/test/logs/
      - store_artifacts:
          path: ~/project/_build/test/logs
          destination: common_test
      # for cover
      - persist_to_workspace:
          root: ~/project/
          paths:
            - _build/test/

At first I thought using pre-steps would be an option. But since the docker containers started directly in a step with the remote docker are not accessible from the host that the tests are run in this is not an option for starting the database before the orb job runs tests.

It would be useful if a job could be extended with additional images to run from in the workflow.

2

@tsloughter I believe you can use mustache templating syntax here… You’ll have to create an additional parameter for a possible database image, and give it an empty string as a default. The mustache syntax will act as a conditional operator and only include the requisite config.yml syntax for a second Docker image if the parameter value is non-empty.

docker:
  - image: first-image-always-loaded
  <<#parameters.second-image>>- image: <<parameters.second-image>><</parameters.second-image>>

Give that a shot and see if it works for you.

3

Right, in the orb? I can certainly extend the orb to make passing additional images a possibility. I didn’t know about the mustache templating, thanks for that, this should work fine, assuming it supports iterating over a list to have more than just 1 image and can be arbitrary yaml since the containers will need configuration.

But it still means every orb has to change to include support for inserting the images through templates. It would be useful if circle supported a way to just define images you want running during a job that don’t require modifying the job itself.

4

No, iterating over a list isn’t possible, this would just allow you to have one additional image. To have more, you’d have to repeat the same logic:

docker:
  - image: first-image-always-loaded
  <<#parameters.second-image>>- image: <<parameters.second-image>><</parameters.second-image>>
  <<#parameters.third-image>>- image: <<parameters.third-image>><</parameters.third-image>>

Etc.

That is what executors do:

https://circleci.com/docs/2.0/reusing-config/#executor

An executor can include any number of Docker images.

5

The job in the orb has an executor that is defined in the orb as well. Doesn’t it end up in the same place of having to add a parameter for each individual additional image to the executor if I wanted that to be able to be extended?

Hm, another option might be better than my current solution but still require a little redefining. I could make the executor a parameter (I forgot if I can just override the executor for a job or have to setup a parameter in each job for this) and the user of the orb could define their own executor and pass its name to the job as a parameter. Assuming an executor defined in the yaml using the orb can be referenced by a name passed as a parameter to an orb.

It still requires the user duplicating the content of the orb executor in their own executor so it works as expected but is definitely better than what I’ve been doing so far :). So still think being able to extend would be nice but I’ll be giving this a try sometime soon and see if it works.

6

Yup, no way around that.

Exactly.

7

Thanks, this worked fine. Every job in my orb now takes an executor as a parameter https://circleci.com/orbs/registry/orb/tsloughter/rebar3

8

It would be great to have a native way to extend an executor from additional docker images.
And maybe a native way to extend a job doing before/after steps.

These two features could allow more composability of orbs.

It was great to find this topic. Thanks!

1 Like
9

Hi @bcardiff—you can use mustache templating to optionally extend an executor with an additional Docker image only if a certain parameter value is set, e.g.:

parameters:
  extra-image:
    type: string
    default: ""
    description: >
      If set, additional Docker image (including tag) to use in this executor.

docker:
  - image: primary/image:tag<<#parameters.extra-image>>
  - image: <<parameters.extra-image>><</parameters.extra-image>>

That said, because there is no list-type parameter, and no iteration, you will have to pick a set number of optional images and image parameters to use.

Does this not suffice?

https://circleci.com/docs/2.0/reusing-config/#using-pre-and-post-steps

1 Like
10

How did I missed pre / post - steps? :see_no_evil: Yes! those suffice. Thanks!

I understand the workaround for the extra-image and is useful. I wanted to vouch for a native support that would not require every author of orbs to decide/design how to pass the extra images.

With apps consuming many services like db, redis, elasticsearch, etc it is easy to exceed the 1 or 2 extra images an author might provide.

1 Like
11

Agreed on still desiring a built-in way to add additional images to have running during a job.

1 Like
12

Also worth pointing this out:

https://circleci.com/docs/2.0/reusing-config/#overriding-keys-when-invoking-an-executor

When invoking an executor in a job any keys in the job itself will override those of the executor invoked. For example, if your job declares a docker stanza, it will be used, in its entirety, instead of the one in your executor.

13

After doing some testing I have feedback for both stories: extending an executor and extending a job.

The workaround for extending an executor using handlebars has some limitations.

You are not able to pass a whole executor structure. Only a name. So it’s impossible to customize the environment of each image. IIRC there was some environment settings that are shared with all images, but even if that is possible the end result is not great. Some images could need different values for the same parameter.

I think I would like to have something like the following. If extend exist, the parameters and docker keys could combined with clear semantics.

# disclaimer: design idea. not real config.

executors:
  custom:
    extend: some_orb/base_executor
    docker:
      - environment: # environment of main image (or all images) not sure
          DATABASE_URL: '...'
      - image: mysql:5.7
        environment:
          MYSQL_DATABASE: '...'
          MYSQL_ALLOW_EMPTY_PASSWORD: 'yes'

This will not allow to combine two executors into one. But I am happy with the scope.

The proposed features for pre/post steps for extending jobs have also some limitations.

The pre/post steps are available when invoking a job in a workflow, but not when definining a job.

When defining a job you can use parameters to allow reusability. I would like to be able to do:

# disclaimer: design idea. not real config.

jobs:
  awesome_foo:
    extend: some_orb/foo    
    pre-steps:
      - run: 
          command: echo "make it awesome!"

workflows:
  version: 2
  test:
    jobs:
      - awesome_foo:
          name: awesome_foo_42
          param_from_original_orb_foo: 42
      - awesome_foo:
          name: awesome_foo_42
          param_from_original_orb_foo: 15

Without using yaml anchors (as one need to do before orbs), I think it’s impossible to define a pre/post step once and reuse it many times as in the example above.

The workaround is expanding the original job steps some_orb/foo (and params), defeating the purpose of extensibility.

Thanks @rose for all the workarounds and pointing to possible directions. I wanted to sum up somewhere my last experience in case it helps some future reader.

1 Like
14

@bcardiff there shouldn’t be any reason you can’t pass environment and its additional keys/values, along with your image name/tag, using the method I mentioned previously. Orbs/YAML accept multi-line parameter values. That said, you may have to play around with the right multiline YAML syntax to get it to interpolate correctly.

How is this different than simply running some_orb/foo with your desired pre-steps?

15

So if the environment variable is passed as String and using the VAR=value syntax it seems I can do something like the following.

workflows:
  version: 2

  test:
    jobs:
      - my_orb/test:
          executor:
            name: my_orb/executor
            environment: >
              DATABASE_URL=mysql://root@localhost/sample_app
            extra-image: mysql:5.7
            extra-image-environment: >
              MYSQL_DATABASE='sample_app'
              MYSQL_ALLOW_EMPTY_PASSWORD='yes'

But it is creating a ad-hoc syntax hard to follow.

In my current case the base executor is only defining the docker image so it is not big deal to duplicate that. Since the docker stanza is dropped if another image is mentioned when using the orb executor, then there is no gain to use it.

Regarding the extending jobs story, the difference is that the pre/post step can only be used inside the workflows jobs. If I want to use the orb job + a pre-step multiple times with different orb job parameters I would prefer to define a custom inline job extending the orb job with the pre-step and invoke multiple time the custom inline job. That can’t be done and I am forced to include the pre-step in even invocation of the orb job.

The orb job is bigger than the executor so it is not an option to copy its content to the inline job.

So I am failing to take some advantage of the orbs when facing the need to extending the need with additional services.

1 Like
16

Most orb jobs are made up of orb commands, so it shouldn’t be to hard to define your custom in-line job using whatever commands make up the orb job that you are looking to extend.

17

Yes I agree. We’ve exposed many commands, an executor, and a job that use the above with the configurable executor as proposed. That seems the best it can get without complicating things and is great, really.

The Job is a bit more complex than using commands since we aim for nice cache and test reporting out of the box, hence the value of extending it without duplicating its definition upon usage in a non simple scenario.

Thanks again for all the attention @rose!

1 Like
18

Hi!

I don’t know if I should create a new thread or necro this one. But I have a question on how to set environment variables for extra images.

What I am trying to do is this:

docker:
  - image: <<parameters.jdk-image>>
  <<#parameters.postgres-image>>- image: <<parameters.postgres-image>>
    environment:
      POSTGRES_PASSWORD: password
      POSTGRES_DB: <<parameters.service-name>>
  <</parameters.postgres-image>>

Where I have the parameters:
jdk image: for the main image to use
service-name: the name of the service
postgres-image: for the postgres image to use.

What I want is that if you supply a postgres image it should be started and the environment variables should be set.

But I am getting the error syntax error: expected <block end>, but found '?' for the yaml linter and the pack command also throws an error Error: An unexpected error occurred: yaml: line 4: did not find expected '-' indicator