Scheduled Pipelines Are Here

Scheduled Pipelines are a new functionality we are introducing to help customers run scheduled work. They are intended to replace scheduled workflows, which will eventually be deprecated.

What are scheduled pipelines?

A scheduled pipeline is a way to trigger a pipeline based upon a schedule. The schedule is defined by the following:

  • One or more days of the week
  • Hours in which the schedule runs
  • A number of times per hour that it runs (i.e. 1 time per hour, 2 times per hour/every 30 minutes, etc.)

Outside of the scheduled aspect, these pipelines are triggered similarly to the API, so they will run all workflows in a pipeline and you can set pipeline parameters as part of your schedule as well.

How do I use a scheduled pipeline?

Right now schedules can be created via an API. You can get more information about the process in our preview docs.

FAQs

Don’t you already have a way to schedule work?

Yes, we have a feature called scheduled workflows. However, this solution has a number of limitations including incompatibility with dynamic config and restricted contexts. We are planning to deprecate scheduled workflows in the future and scheduled pipelines will be its replacement.

What if I already use scheduled workflows?

You will need to migrate to scheduled pipelines before the deprecation date. This will involve creating a scheduled pipeline and removing your scheduled workflow. The deprecation date will occur when scheduled pipelines is GA and will be about 6 months from that date. Any scheduled workflows will stop running after the deprecation date.

How do I run a particular workflow in a schedule?

By default, a scheduled pipeline will run every workflow in the same way a webhook or API triggered pipeline would. Since scheduled pipelines let you set pipeline parameters, if you define a parameter in your config you can use it to filter specific workflows with a “when” or “unless” clause.

Do I have to use an API to manage this?

For now, this is API only. We are working on a UI for managing this, it will be available by GA. Both the UI and the API are using the same backend, so you will be able to use either or both. We are removing the option to manage these changes via code since storing in code has caused a number of reliability issues. However, this new API-focused approach will enable management via Infra as Code and/or Config Management tools (Terraform, Ansible, Chef, Puppet, etc.)

What is the “actor” thing?

By default, a schedule will be run as a “scheduled user”. However, by selecting an alternate actor, these schedules can be run as that user and have access to the same restricted contexts that user has access to. That actor will need to be the one creating/modifying the schedule.

Where did the cron syntax go?

We’ve found that the overwhelming majority of schedules that are being run now generally fit a set of simple patterns. Keeping schedules to this simpler syntax makes scheduled pipelines more stable while also covering the most common use cases.

6 Likes

Hi @thekatertot,

We got several scheduled workflows which are running once a month. The new scheduled pipelines we are forced to run these pipelines once a week.
Are you planning to add monthly schedule as well?

Thanks,
Andor

7 Likes

Hey @andormarkus, we are definitely planning on expanding schedules, so this is really good feedback. We don’t have a timeline for any new additions to schedules yet, but it always helps us to know how people are using the features, so thank you!

Hi all, the UI for scheduled pipelines is now in open preview. You can manage scheduled pipelines under the “Triggers” section in the project settings. Please let us know if you have any feedback.

So basically, if I have 20 workflows and want only 1 scheduled workflow, I will to update the 19 others to tell them to not be triggered ? Why not instead, reverse the logic, ignore the scheduled trigger by default and force the user to add the condition to their scheduled workflow ?

1 Like

Hi all, scheduled pipelines are now GA, you can find more information at the docs here and a tutorial here

1 Like

We chose this approach since it seemed the simplest while still resolving the known issues. We are continuing to work on ways to make this functionality better, can you can post on our ideas portal?

1 Like

@salil
We have a scheduled workflow that runs on the 1st of every month. How can it be achieved with scheduled pipelines?

Right now Scheduled Pipelines only covers schedules that run at least once a week since that covered the majority of schedules currently used. We are investigating additional types of schedules, especially those that run less frequently.

@salil are there any known deadlines yet for scheduled workflows? Earlier it was mentioned that scheduled workflows will get deprecated in the future and I would like to learn how much time we have for migrating things to scheduled pipelines. Thank you.

The deprecation for Scheduled Workflows is June 3rd 2022.

1 Like

Yeah, I would love a way to schedule a workflow to run every two weeks to match our release cadence.

Hi @salil @thekatertot

Will you add additional types of schedules till the deprecation of scheduled workflows?
If not we need to build custom logic which ignores the weekly trigger except the first week on month.

Thanks,
Andor

2 Likes

Hi @salil @thekatertot ,

We have scheduled workflows that run monthly so this hard deadline for deprecation doesn’t really make sense as there is no matching way to do that with scheduled pipelines. I can also see a case for other cadences such as every X amount of weeks/months. Adding custom logic to cancel pipelines seems like a headache and a workaround to a simple problem. Will you be adding support for higher cadences than once a week before this June 3rd deadline?

Thanks,
Josiah

Really disappointed to be forced to support this. For my simple nightly workflow this feels a lot more difficult to configure and I guess I have to cross my fingers I got it right this evening? The documentation was not at all clear. Very unimpressed.

Seems like a giant leap backwards.

Triggers should have feature parity with scheduled jobs before any deprecation date is set.

3 Likes

Hello @thekatertot

I see as per the documentation on CircleCI for schedule workflows, it says | Scheduled workflows will be phased out starting June 3, 2022|

I see there is also a migration guide for migration to Scheduled Pipelines. But currently, the pipeline can only be set up at max for weekly jobs. We have some jobs that run monthly and they are currently working fine with schedule workflow. But I do not see any update for the scheduled pipeline to run on a monthly basis. Is there any discussion/ticket open for the same?

1 Like