Allow bundling several steps, e.g. by supporting nested steps

circle.yml

#1

I just learned that you can use yaml references in your config.yml (which is btw not mentioned anywhere in the docs that I have seen) which I’ve now started using extensively to remove a lot of repetition. All my steps now look something like

 - *cache_restore_node_download
 - *cmd_install_node
 - *cache_save_node_download

with those references defined as:

  - &cache_restore_node_download
    restore_cache:
       name: Restoring cache - node installation package
       key: cache-downloads-node-{{ checksum ".tmp/node_version" }}

  - &cmd_install_node
    run:
      name: Installing node
      command: |
        # [snip] ...commands to download the correct node version if another version was restored above...
        # [snip] ...install node from downloaded archive...

  - &cache_save_node_download
    save_cache:
      name: Saving cache - node installation package
      key: cache-downloads-node-{{ checksum ".tmp/node_version" }}
      paths:
        - ~/.downloads/node/

However, having to do this in every job in a workflow is still a lot of repetition, especially since this is not my only cache and I have to do this several times.

It would make the config much more concise if it was possible to combine all those three steps into one, e.g. with a reference such as:

- &install_node
  - restore_cache: ...
  - run: ...
  - save_cache: ...

so that one of my steps could be:

 - *install_node

This however translates to a nested array, which is not a structure that Circle supports. This could be allowed by Circle if nested steps were flattened to a 1-dimensional array, e.g. if

 - - restore_cache: ...
   - run: ...
   - save_cache: ...

was rewritten as simply

 - restore_cache: ...
 - run: ...
 - save_cache: ...

This is to say, nested steps are just one way of supporting bundling multiple steps into one yaml reference. I’m open to hear about other possibilities of allowing that if somebody has other ideas or already does this in some other way!


#2

I agree this would be useful. However in your case couldn’t you have a job that restores npm cache and installs them if out of date then saves the cache? The cache would only get written if it was a new cache. Then you’d have a single reusable anchor for restoring your nose cache (which would also be used in the restore npm cache job).

Subsequent jobs would simply require this one job, and all of your cache writing could happen in a single job.

Regardless, using this I still have to do:

steps:
  - *restore_npm
  - *restore_bower
  - *restore_composer

At least it’s only one anchor per cache rather than 3, but would be nice if I could just use a single anchor to bring in all three steps.


#3

That’s exactly what I’m doing! :slight_smile:

The very first step of my workflow updates all the caches so that I can rely on an up-to-date cache in later steps.

But, like you, I have a bunch of caches that I’m restoring, so I’d like to combine them into just one anchor.


#4

There’s a bug report I came across that is related to this, but doesn’t look like it has much traction as an issue since it’s technically the way that yaml is supposed to work (or not work however you want to put it)

https://discuss.circleci.com/t/yaml-anchors-implementation-incomplete?source_topic_id=15339

#5

Any solution here CircleCI Team?

This is something that would really improve the configuration file.

Using references you guys would just need to flatten the steps. To be a 1D array.

More info here https://stackoverflow.com/a/30770740/710693


#6

This is much needed. Yes, this is how YAML is supposed to work - so imo Circle should handle this at the application layer. Sounds not like a complicated feature to me?
For reference, popular SO question: https://stackoverflow.com/questions/4948933/is-there-a-way-to-alias-anchor-an-array-in-yaml

With Circle 2.0 workflows there are a lot of step “bundles” that I would like to share between jobs, but I have to repeat the references to every step :frowning:


#7

Yep, I have a similar issue and would really like to flatten my array of arrays to pass the CircleCI config validator. Now I get

{
  "steps" => [
    [
        {
            "type" => "cache-restore",
            "name" => "Restore Repository Cache",
            "key" => "v2-spc-repo-{{ .Environment.CIRCLE_SHA1 }}"
        }, 
        "checkout", 
        {
            "type" => "cache-save",
            "name" => "Save Repository Cache",
            "key" => "v2-spc-repo-{{ .Environment.CIRCLE_SHA1 }}",
            "paths" => ["~/spc"]
        }
    ],
    [
        {
            "run" => {
                "name" => "Which bundler?",
                "command" => "bundle -v"
            }
        }, 
        {
            "type" => "cache-restore",
            "name" => "Restore Bundle Cache",
            "keys" => ["v2-spc-bundle-{{ checksum \"Gemfile.lock\" }}"]
        }, 
        {
            "run" => {
                "name" => "Bundle Install",
                "command" => "bundle check || bundle install --path vendor/bundle"
            }
        }, 
        {
            "type" => "cache-save",
            "name" => "Save Bundle Cache",
            "key" => "v2-spc-bundle-{{ checksum \"Gemfile.lock\" }}",
            "paths" => ["./vendor/bundle"]
        }
    ]
  ]
}

But ideally I’d like to have one array.


#8

Any movement on this?

This seems like a previous easy and obvious solve that would move a significant amount of duplication.


#9