Parameterized docker images

Build Environment
1

My mono-repo currently has a pretty verbose CircleCI configuration. One of the pain points is our repeated blocks of docker images, docker commands, and docker environments.

For example, all our jobs use the same base_image but then different jobs will augment their environment with a database (or two), elastic search, browsers, etc.

It looks like parameters is the solution to my problem but I can’t seem to get it to parse correctly – as far as I can tell, I don’t really understand the conceptual model of parameters at compile time.

This is basically what I have working:

executors:
  base_executor:
    docker:
      - image: &base_image some_image
  rails_executor:
    docker:
      - image: *base_image
      - image: &mysql_image mysql_image
        command: &mysql_command
          - mysqld ...
        environment: &mysql_environment
          FOO: bar
  rspec_executor:
    docker:
      - image: *base_image
      - image: *mysql_image
        command: *mysql_command
        environment: *mysql_environment
      - image: &redis_image redis_image
  end_to_end_executor:
    docker:
      - image: *base_image
      - image: *mysql_image
        command: *mysql_command
        environment: *mysql_environment
      - image: *redis_image
      - image: firefox_image

It is pretty obvious that I want an additive approach to our docker images but executors replace the node value – unless it is the environment key.

I was then hoping to see if something with parameters would work:

executors:
  org_executor:
    parameters:
      use_db:
        type: boolean
        default: false
      use_inmemory_db:
        type: boolean
        default: false
      use_browser:
        type: boolean
        default: false
    docker:
      - image: some_image
      <<# use_db >>
      - image: mysql_image
        command:
          - mysqld ...
        environment:
          FOO: bar
      <</ use_db >>
      <<# use_inmemory_db >>
      - image: &redis_image redis_image
      <</ use_inmemory_db >>
      <<# use_browser >>
      - image: firefox_image
      <</ use_browser >>

Help would be appreciated.

2

Assuming you are just in the process of migrating to 2.1, do you have a copy of your 2.0 config file? I wonder if YAML references would be a better solution to this - it just leverages the fact that YAML allows you to define blocks that can be repeatedly used. You can even re-use a block and then replace single keys, if you have something that is nearly the same.

See the Wikipedia article on YAML if you want to see how this works - it is surprisingly detailed.

3

I did the migration to 2.1 and am now exploring how to leverage the new features (executors, commands, etc) over YAML node aliases.

In the “pre-executor” state, it was something along the lines:

job1:  
  docker:
    - image: &base_image some_image
job1-1:  
  docker:
    - image: *base_image
job1-2:  
  docker:
    - image: *base_image
job2:
  docker:
    - image: *base_image
    - image: &mysql_image mysql_image
      command: &mysql_command
        - mysqld ...
      environment: &mysql_environment
        FOO: bar
job2-1:
  docker:
    - image: *base_image
    - image: *mysql_image
      command: *mysql_command
      environment: *mysql_environment
job3:
  docker:
    - image: *base_image
    - image: *mysql_image
      command: *mysql_command
      environment: *mysql_environment
    - image: &redis_image redis_image
job4:
  docker:
    - image: *base_image
    - image: *mysql_image
      command: *mysql_command
      environment: *mysql_environment
    - image: *redis_image
    - image: firefox_image

You can see that there can be some redundancy even when using aliased nodes – our workflow has 14 jobs!

I’ll see if perhaps more aliasing is the right answer – I also do not understand the extent of overriding values with aliasing.

4

After a little bit of investigation: it looks like YAML anchors and references will not save the day for a “really DRY” configuration.

I was able to improve my current configuration a bit by realizing that:

I can anchor the mysql mapping (image, command, environment).
I cannot merge two lists.

The best I could do was something along the lines of:

executors:
  base_executor:
    docker:
      - &base_docker
        image: some_image
  rails_executor:
    docker:
      - *base_docker
      - &mysql_docker
        image:  mysql_image
        command:
          - mysqld ...
        environment:
          FOO: bar
  rspec_executor:
    docker:
      - *base_docker
      - *mysql_docker
      - &redis_docker
        image: redis_image
  end_to_end_executor:
    docker:
      - *base_docker
      - *mysql_docker
      - *redis_docker
      - image: firefox_image

And a bunch of sharing environment variables between the executors as well (via anchors and references).

5

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.