Dynamic config - Passing data from setup workflow to generated workflow

Build Environment
1

TLDR: We’re using dynamic config to generate a workflow. What’s the best way to pass a large amount of data from the setup workflow to the generated workflow?

In our setup workflow, we want to work out how many parallel test jobs we need, and allocate tests to the jobs. The tests consist of dynamically generated lists of shell commands which can get pretty long.

Do the 2 workflows share a workspace? If so, that’s the obvious way.

Otherwise, I can think of 3 ways, but none are ideal…

  1. just write the commands directly into the config file. But we can’t write it in the parallel test job, since the job only has one definition and each parallel instance needs different commands. So it gets a bit messy. It also makes the config file big and hard to read; there’s escaping to think about, etc.

  2. pass the text in workflow parameters, say using base64 encoding, but is there a size limit for parameters?

  3. create a cache in the setup workflow, then re-read it in the generated workflow. It sounds hacky.

Thoughts?

1 Like
2

I recently gave a try at the workspace persistence to no avail. The setup and subsequent workflows have distinct ID’s so I believe that precludes them from being able to share workspace layers.

If you go with option 1 (this is how I am currently approaching it). You might consider using the FYAML merging provided by the circleci config pack command in order to make building large complex yaml files easier. But be warned that the circleci cli tool on the build agents is not the same as the one we install locally so you need to install that on your agent first in order to use the FYAML command. The rough layout of it looks something like this: cci dynamic config example · GitHub

Option 2. “is there a size limit for parameters?” Same key concern I had, I’ll likely avoid that as it will complicate the overall structure of the workflow and job definitions. I suspect the opaque serialization complexity being at such a high level will be more than I want to stomach in an ongoing fashion.

Option 3. I believe the best bet will be to zip your params into a single artifact then lean on the CCI api to pull the artifact you will need to invoke your job commands. CircleCI API - getJobArtifacts It wouldn’t be terrible but puts a decent chunk of complexity into every generated workflow/jobs runtime. Having to make multiple round trips to the CCI API in order to finally get what you need for the artifacts api call. Pushing all the complexity into the config generation should result in a lot less headache in the long run I think.

1 Like