[Workaround] Using scheduled workflows with Dynamic config

Helpful Resources Support Solutions
1

[Workaround] Using scheduled workflows with Dynamic config

We recently released a feature called Dynamic config that unlocks a lot of options in using CircleCI. However, at the moment this feature is unable to be used in conjunction with scheduled workflows. The team is actively working on addressing this to allow for them to be used together. Since there isn’t a specific date of that release this guide is to help ensure both features can be used at the same time while that is being worked on.

This guide will walk through setting up a secondary repository that will not have dynamic config enabled. It will simply contain a config.yml file that has scheduled workflows. Those workflows will execute a job that sends an API call to trigger a build at the repository that has dynamic config enabled, allowing for builds to be triggered on a specific cron schedule.

Requirements

  1. This guide utilizes Pipeline parameters, so you will need to be on at least version 2.1
  2. This guide requires the creation of a new repository, you will need the proper permissions at your organization to do that

Create new ‘schedule’ repository

First step will create a new repository, ideally under the same organization where you have your main repository you want to have scheduled workflows on.

Within that new repository you will create a directory called .circleci at the root of the project. Within that directory create a file called config.yml.

The contents of config.yml will be similar to the following:

version: 2.1

jobs:
  run-scheduled-workflow:
    docker:
      - image: cimg/base:stable
    steps:
      - run:
          name: Run scheduled workflow
          command: |
            curl --location --request POST 'https://circleci.com/api/v2/project/(vcs)/(org)/(project)/pipeline' \
            --header 'Content-Type: application/json' \
            -u "${CIRCLE_TOKEN}:" \
            --data-raw '{
              "branch": "main",
                "parameters": {
                  "scheduled-workflow": true
              }
            }' 
      
workflows:
  scheduled-workflow:
    triggers:
       - schedule:
           cron: "30 9 * * *"
           filters:
             branches:
               only:
                - main
    jobs:
      - run-scheduled-workflow

You will want to modify the following items:

  1. The URL of the curl you need to change (vcs) to either github or bitbucket, (org) to the organization where your dynamic config repo lives, (project) to the repository of your dynamic config project.
  2. In the --data-raw section adjust "branch": "main" to the branch where you want to run the scheduled workflow.
  3. In the --data-raw section add any additional parameters you may need/want to pass along.
  4. Adjust the cron: "30 9 * * *" to be the schedule in which you want to run on.

Once the above changes are made and committed to the repository, set this project up in CircleCI.

Configure Environment Variables

The curl in the job requires a personal API key to successfully trigger the workflow. This needs to be stored as a variable and can be done at the project level, or, if you plan to use it across projects – set these up in a context.

Variable Name Description
CIRCLE_TOKEN This will be a personal API token.

Note: The personal API token used for the curl needs to have write access to the dynamic config repository, which means as long as the token is under your user and you have write access, it will work.

Add ‘scheduled-workflow’ parameter to dynamic config

Now that we have the 2nd repository setup and have it running on a schedule, we need to ensure our dynamic config knows how to act when it receives the API call.

You will need to add scheduled-workflow as a pipeline parameter, like:

version: 2.1

setup: true

parameters:
  scheduled-workflow:
    type: boolean
    default: false

Next specify a new workflow that will only execute when scheduled-workflow is true:

workflows:
  version: 2
  scheduled-workflow:
    when: << pipeline.parameters.scheduled-workflow >>
    jobs:
      - schedule-job

Set the jobs to be the jobs you want to run on a schedule. If you don’t want your other workflows to trigger when this API call happens you will also need to add the following condition to the rest of your workflows:

when:
  equal: [false, << pipeline.parameters.scheduled-workflow >>]

Resources

  1. Here is a config.yml example for the schedule repository.
  2. Here is a config.yml example for the dynamic config repository.
2

This example does not actually work, this is because any configuration with the setup stanza (ie setup: true) can only have a single workflow. Please see this support ticket:

if you want to do this you will need to use a continuation orb to call a scheduled job configuration file with a pipeline parameter set. The Dynamic Config will then look like

version: 2.1

setup: true

parameters:
  scheduled-workflow:
    type: boolean
    default: false

orbs:
  continuation: circleci/continuation@0.1.2

jobs:
  execute_workflow:
    docker:
      - image: cimg/base:stable
    steps:
      - when: << pipeline.parameters.scheduled-workflow >>
        steps:
          - continuation/continue:
              configuration_path: .circleci/scheduled_workflow.yml
    - when: 
        not: << pipeline.parameters.scheduled-workflow >>
        steps:
          - continuation/continue:
              configuration_path: .circleci/regular_workflow.yml

workflows:
  setup:
    jobs:
      - execute_workflow

secondly you cannot use the default ${CIRCLE_TOKEN}, this will not let you execute the pipeline of a private pipeline. You will need to create a new API token and use that in the curl request. I suggest editing the curl request to be

OUTPUT=$(curl -X POST 'https://circleci.com/api/v2/project/{project_slug}/pipeline' \
 -H "Content-Type: application/json" \
 -H "Circle-Token: ${CIRCLECI_API_TOKEN}" \
 -d '{"branch": "main", "parameters": {"scheduled-workflow":  true} }')

echo "${OUTPUT}"

as this will print the ID of the pipeline and you can then curl
https://circleci.com/api/v2/pipeline/{pipeline-id}
and see any errors that occur

1 Like
3

Hello, I am interested in dynamic configs since they will solve a specific problem in my project but I also need a few scheduled workflows which the project is dependent on.

Has any of the approaches mentioned here been tested ? I would hate to invest time in it and find out it’s not working in the end.

Thanks in advance !

4

Hi @dragosMC91,
yes we are using this exact method, and so this is a tested solution at the time of writing.

Regards
W

5

In the meantime I have implemented the same idea and it worked fine so far. Thank you for the valuable information @wenisman . Good luck!

6

Any news about the support of schedule workflows with dynamic config? It was released 4 months ago, and the workaround for us is to use Github Action to trigger Circle Ci workflows…
Is there any ticket where we can vote?

Thanks

7

Hi @ahocquard !

You can indeed vote here:

The above is tracking the full completion of this feature, which entails implementation of a way to schedule workflows/pipelines with dynamic configs. If you vote on the above you’ll receive automatic updates on the progress of the feature. I will note that it’s something actively being worked on and we should have further updates shortly.

Thanks!

8

As an update on this workaround, we just launched ‘Scheduled Pipelines’ into Open Preview:

This feature works with Dynamic Configs/Setup workflows and should allow you to continue to run pipelines on a specific schedule.

Hope the above helps!
-Nick

2 Likes
9

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.