Feedback wanted: moving from v1.1 job triggering to v2 pipeline triggering

With API v2 emerging into production use we are planning to ask everyone who uses the v1.1 job triggering endpoint to move over the new method of triggering in the v2 API. We will not be immediately disabling the v1.1 endpoint for triggering a job, but for a variety of reasons we would prefer to turn it off in a few months (current target is March 2020, but that is subject to feedback from those reading this).

We think the new pipeline triggering is superior for a variety of reasons, some for our internal operation and some for API developers. The benefits to you are:

  1. The new triggering endpoint offers pipeline parameters, which are more explicit and expressive.
  2. The new triggering allows you to run workflows
  3. The new triggering works with v2.1 config, allowing use of reusable config elements, including orbs
  4. The new triggering works best with our new pipelines UI

For those using build parameters in the v1.1 job triggering endpoint you will need to make some slight configuration adjustments to either explicitly inject the environment variables that represent the build parameters used in your jobs or swap out the environment variables with the pipeline variable syntax (eg: << pipeline.parameters.mything >>). I am eager to hear from anyone who finds this onerous, as there are things we might consider if it turns out this is problematic due to issues of scale (for instance, if you have lots of projects to change I’d like to hear from you to talk about how we might be able to help).

I am starting this topic as a place to talk about how to make the transition and to encourage discussion from you if you run into problems or find the new mechanism insufficient to make the switch.

I also hope to solicit input on whether anyone reading this feels like they need more time than March 2020 to stop using the job triggering in v1.1 and switch to pipeline triggering in v2.

Docs on pipeline triggering: https://circleci.com/docs/api/v2/#trigger-a-new-pipeline

Docs on pipeline parameters: https://circleci.com/docs/2.0/pipeline-variables/

2 Likes

Hello,

Thanks for the explanation and letting us know how things are moving, appreciate that.

I was previously triggering a specific job through the v1.1 API and was confused as why jobs triggered that way didn’t have a name in the UI (as can be seen from this Twitter exchange). I guess I didn’t really understand the difference between jobs, workflows and pipelines until then. Things are much clearer now.

I migrated my code to use the new v2 API and the parameter syntax. It took a bit of trial and error to understand how to specify which workflow I want to run… and it turns out it is not possible to specify it directly from the call. AFAICT all workflows are run if I manually trigger my pipeline from the API.

The solution I pieced together from your documentation and various comments on this forum is to add a when clause to each workflow and use parameters to flag which workflow I actually want to run. Below is an abbreviated config file, could you let me know if I’m on the right track or if you see any way to improve it?

version: 2.1

jobs:
  # omitted for brievity

parameters:
  workflow_commit:
    type: boolean
    default: true
  workflow_automatedRelease:
    type: boolean
    default: false

workflows:
  version: 2
  # Build/Test/Lint on each commit
  commit:
    when: << pipeline.parameters.workflow_commit >>
    jobs:
      - ci
  # Run every week
  weeklyUpdate:
    triggers:
      - schedule:
          cron: '0 9 * * 2'
          filters:
            branches:
              only:
                - master
    jobs:
      - weeklyUpdate
  # Run only when manually triggered by the API
  automatedRelease:
    when: << pipeline.parameters.workflow_automatedRelease >>
    jobs:
      - automatedRelease

And then, the curl call that explicitly turns off the commit workflows and turns on the automatedRelease one. The weeklyUpdate workflow being on a trigger, it seems that I don’t need to explicitly turn it off, is that correct?

curl -u ${CIRCLECI_TOKEN}: -X POST --header "Content-Type: application/json" -d '{
  "parameters": {
    "workflow_commit": false,
    "workflow_automatedRelease": true
  }
}' https://circleci.com/api/v2/project/github/pixelastic/pathfinder-society/pipeline

This seems to correctly trigger only the automatedRelease workflow when used. Could you confirm this is the recommended way to trigger a specific workflow using the v2 API?

We started with CircleCI back in the 1.0 days. While we migrated all our yml to 2.0 we still have a number of apps that are job-based and use this API to trigger. Could there be any relief on the timeline as it will take us a while to convert them all over? Any help would be appreciated.

Do you have account manager? If so, best to go through them. If not, do you know how many projects you need to change?

Yes we do and I’m going to email her as well.

We have 6 apps (not necessarily projects/repos in the CircleCI sense) that are built around the https://circleci.com/api/v1.1/project/github/{org}/{repository}/tree/{branch} endpoint being able to pass a CIRCLE_JOB parameter to trigger a specific CircleCI job. Basically we have build pipeline automation where something happens on a schedule or external trigger (not in Github) that triggers a job in CircleCI. Since we’ve been with CircleCI for 3 years we’d invested a lot of our pipeline automation tooling in the v1 API.

We also use parameters/environment variables quite a bit as well and the changing parameter handling is also pretty impactful.

We completely understand the benefits and the need to move forward but we’ll have to change a fair amount of code to move to API version 2. Ideally we’d love to update everything to use 2 but that will be impossible to do by March 1. It’s not just changing yml for us but redesigning some of the pipeline automation we’ve built to work with the new API version and new parameter handling.

Thanks Brian, this is super-helpful detail. We have been experimenting with a stop-gap approach where all API v1.1 job triggers are re-routed through our pipelines machinery. If that works we will likely create some grace period and can work with you directly on a transition plan.

Have you already done a direct assessment of the migration cost? Our goal is that when changes are necessary to move from the old approach to the new approach for triggering work that the cost will be “only” in config tweaks. In most cases, configs are fairly “small”, and the changes are minimal. But, if you’re finding that the changes you’d need to make will be fairly onerous I’d love to understand and see if there’s things we could do on our end to ease that pain.

Thanks Nathan. I appreciate the understanding. What’s got us is that we’ve built other business apps that trigger Circle jobs via the 1.1 API from other places (AWS, Slack, Salesforce). After thinking about it more I think the real questions is how we would reach in and trigger a specific job.

We have a few projects that look like this:

jobs:
  build:
    <build steps here>

  deploy:
    <deploy steps here>

We then use the endpoint and parameters I mentioned above to trigger one of the jobs (like deploy) from an external service. Is there going to be a replacement to that? What would be the recommended approach in the new API? Would we convert to workflows? I didn’t see a way to trigger a workflow in the API documentation.

We would prefer that move to workflows, yes. We recently added a when parameter under each workflow declaration that can be used in conjunction with pipeline variables to dynamically run certain workflows based on boolean variables (including parameters you can define for the pipeline). We are also considering adding similar functionality to individual jobs within a workflow, but for now it’s not clear we need to add that complication - we do invite feedback, of course.

When we built the when clause we wanted to provide a more general-purpose semantics around how a person writing config could manage what to do based on certain inputs (parameters explicitly declared for the pipeline) or based on the metadata associated with the pipeline (today we only expose the pipeline ID and number this way, but we hope to add trigger information, so you can have dynamism based on, for instance, whether it’s a “Draft PR”). We decided that the names of workflows is an internal concern of the config, and that a config author should not need to worry about whether or not an API client triggering pipelines knows about the internal structure of config beyond the declare parameters. We felt this was a cleaner contract and a more deterministic interface with less chance of unforeseeable breakage (ie: changing the shape of explicit inputs is different in kind from changing the arbitrary name given to a workflow that is run). Thus, rather than make job name or workflow name the input, we let the config author declare specific inputs (along with type and default values) and use that to drive the dynamism based on what is sent with the API POST request to trigger work.

You can see examples of using when under a workflow here: https://circleci.com/docs/2.0/pipeline-variables/#conditional-workflows

Assuming you’ve been running your API-triggered jobs regularly this week, it’s likely that you ran through our code that now automatically runs them through the pipelines machinery, which will automatically wrap workflows around them anyway (you can check this by seeing if those API-triggered runs show up on the workflows list (or on the Pipelines list in our new UI now in preview).