Scheduled Pipelines Are Here

Scheduled Pipelines are a new functionality we are introducing to help customers run scheduled work. They are intended to replace scheduled workflows, which will eventually be deprecated.

What are scheduled pipelines?

A scheduled pipeline is a way to trigger a pipeline based upon a schedule. The schedule is defined by the following:

  • One or more days of the week
  • Hours in which the schedule runs
  • A number of times per hour that it runs (i.e. 1 time per hour, 2 times per hour/every 30 minutes, etc.)

Outside of the scheduled aspect, these pipelines are triggered similarly to the API, so they will run all workflows in a pipeline and you can set pipeline parameters as part of your schedule as well.

How do I use a scheduled pipeline?

Right now schedules can be created via an API. You can get more information about the process in our preview docs.

FAQs

Don’t you already have a way to schedule work?

Yes, we have a feature called scheduled workflows. However, this solution has a number of limitations including incompatibility with dynamic config and restricted contexts. We are planning to deprecate scheduled workflows in the future and scheduled pipelines will be its replacement.

What if I already use scheduled workflows?

You will need to migrate to scheduled pipelines before the deprecation date. This will involve creating a scheduled pipeline and removing your scheduled workflow. The deprecation date will occur when scheduled pipelines is GA and will be about 6 months from that date. Any scheduled workflows will stop running after the deprecation date.

How do I run a particular workflow in a schedule?

By default, a scheduled pipeline will run every workflow in the same way a webhook or API triggered pipeline would. Since scheduled pipelines let you set pipeline parameters, if you define a parameter in your config you can use it to filter specific workflows with a “when” or “unless” clause.

Do I have to use an API to manage this?

For now, this is API only. We are working on a UI for managing this, it will be available by GA. Both the UI and the API are using the same backend, so you will be able to use either or both. We are removing the option to manage these changes via code since storing in code has caused a number of reliability issues. However, this new API-focused approach will enable management via Infra as Code and/or Config Management tools (Terraform, Ansible, Chef, Puppet, etc.)

What is the “actor” thing?

By default, a schedule will be run as a “scheduled user”. However, by selecting an alternate actor, these schedules can be run as that user and have access to the same restricted contexts that user has access to. That actor will need to be the one creating/modifying the schedule.

Where did the cron syntax go?

We’ve found that the overwhelming majority of schedules that are being run now generally fit a set of simple patterns. Keeping schedules to this simpler syntax makes scheduled pipelines more stable while also covering the most common use cases.

6 Likes

Hi @thekatertot,

We got several scheduled workflows which are running once a month. The new scheduled pipelines we are forced to run these pipelines once a week.
Are you planning to add monthly schedule as well?

Thanks,
Andor

2 Likes

Hey @andormarkus, we are definitely planning on expanding schedules, so this is really good feedback. We don’t have a timeline for any new additions to schedules yet, but it always helps us to know how people are using the features, so thank you!

Hi all, the UI for scheduled pipelines is now in open preview. You can manage scheduled pipelines under the “Triggers” section in the project settings. Please let us know if you have any feedback.