[Product Update] Trigger pipelines from anywhere: custom webhooks

Custom webhooks (formerly known as “inbound webhooks”) are now also available to orgs that integrate with CircleCI’s GitHub OAuth App

Trigger a pipeline from any location that can emit a webhook or run a curl request. Validate changes from external tools such as model/dataset registries

Creating a webhook trigger

  1. In the CircleCI web app, navigate to “Project Settings” >> “Pipelines”. If there is already a pipeline set up, proceed to step 3 below. Else: click “Set Up Pipeline” >> “Connect”. Install CircleCI’s GitHub App into your GitHub organization and choose whether to grant access to all repositories or a subset of repositories (CircleCI organizations that use the GitHub OAuth App can also install the GitHub App, read more here).

  2. Name the pipeline and select the repository where the CircleCI YML configuration file is stored. Enter the file path (include the .circleci directory). The file path can be something other than .circleci/config.yml, for
    example, .circleci/webhook.yml. Save the pipeline.

  1. Go to “Project Settings” >> “Triggers” >> “Add trigger”

  2. Select “Custom Webhook” >> “Next”

  1. Give the trigger a name & description and select the pipeline that you created in step 1 above. Save the trigger.

  1. Store the webhook URL & secret.

Now every time the custom webhook’s URL is pinged, a pipeline will be triggered using the CircleCI configuration file that you specified in the file path.

Example of triggering from a curl command:

curl -X POST -H "content-type: application/json" 'https://internal.circleci.com/private/soc/e/6ccfca1c-5ed6-4dcf-96ca-374969d6edcb?secret=insertSecret'

You must use a POST command with content-type: application/json. Always include the secret in the URL when calling it.

Example of using an custom webhook to make a pipeline trigger a pipeline from another project:

In this example, when project A finishes running a pipeline, we will enable project B to automatically run a pipeline as well (for example, if you’d like to run a set of integration tests on a repo after you’ve made a change to a separate repo).

In project B, set up an custom webhook in Project Settings > Triggers. Copy the “Secret” and add it as an environment variable in Project A’s Project Settings > Environment Variables (ie.WEBHOOK_SECRET_2 ).

In project A’s configuration file, call the custom webhook’s “Webhook URL” via a curl command like the below example with the secret as an environment variable.

version: 2.1

jobs:
  say-hello:
    docker:
      - image: cimg/base:stable
    steps:
      - checkout
      - run: 
          name: example
          command: echo "one step"

      - run:
          name: Kick off new pipeline
          command: |
              curl -X POST -H "content-type: application/json" "https://internal.circleci.com/private/soc/e/6ccfca1c-5ed6-4dcf-96ca-374969d6edcb?secret=${WEBHOOK_SECRET_2}"

workflows:
  say-hello-workflow:
    jobs:
      - say-hello

At this point, every time project A runs a pipeline, project B will subsequently run a pipeline as well.

This functionality is also available to orgs that integrate with CircleCI’s GitHub OAuth App, read more about that here.

Known Limitations

Contexts that are restricted to a GitHub security group will not work if you are using custom webhooks.

The configuration file that was used for pipelines triggered by a custom webhook will only be visible in the CircleCI web app if the configuration file path is .circleci/config.yml.

2 Likes

I don’t want to use Hugging Face.

I’m looking for a replacement for the “trigger a pipeline” API.

The web app advertises that I can trigger a pipeline from anywhere that can run a curl command.

What curl command do I need to use?

I set up the inbound webhook follwing your instructions.

I use a curl command like this to test the webhook:

curl -G "https://internal.circleci.com/private/soc/e/bb29a2a6-5673-403d-baef-5b46f59f7332" -d @/tmp/webhooksecret

I get this error.

{"Status":"Endpoint not found"}

Is something missing?

curl -X POST -H "content-type: application/json" 'https://internal.circleci.com/private/soc/e/6ccfca1c-5ed6-4dcf-96ca-374969d6edcb?secret=insertSecret'

Worked for me. I can get this post updated with that example.

You may want to take the time to change the DNS name used for this service. ‘internal’ is not the best name to assign to a public service and so will cause a lot of misunderstandings going forwards.

1 Like

Hey there

I’m curious on the restriction to “Github App” only right now…

Looking at the list of missing features for the “Github App” integration (no manual trigger and no schedules!!!), we’re not in a position to transition right now. But having the ability to trigger pipelines via a Webhook would be extremely useful.

Is this feature likely to be available to Github OAuth users? Or when will the Github App integration be brought up to feature parity?

Thanks

@fatmcgav The feature will eventually be available to all CircleCI users, I unfortunately don’t have a timeline to share though for when that will happen.

Manual triggers we expect to be possible within the next ~3 months for users on the GitHub App integration. The timeline for the ability to schedule builds is still TBD.

I’m curious, what is your use case for using inbound webhooks? Any chance you’d be open to talking to our User Research team for 45 minutes about this and a new prototype to simplify setting up inbound webhooks? We offer compensation in the form of a Tremendous gift card, email me at sebastian@circleci.com if you’re interested, we’d love to hear from you.

@fatmcgav we have a “beta” version of this feature working for GH OAuth users. Any interest in giving it a shot?

An FYI to anyone reading this post that this functionality is now available to orgs that integrate with CircleCI’s GH OAuth App integration.

1 Like

@sebastian-lerner If possible I would like to give the webhook triggers for GH OAuth users a shot.

Is the feature tied to one personal or can you enable it for our entire org?

@martinth this should be available to all orgs right now, we removed the need to enable it for specific orgs yesterday Now in preview - custom webhooks for orgs that integrate via GitHub OAuth App - CircleCI

Are you seeing “Pipelines” & “Triggers” tabs in Project Settings of the CircleCI UI? If yes, then you’re good and can follow the instructions above.

Let me know if you run into any issues.

Thanks for coming back on this - I checked our setup.

  • After checking with the “GitHub OAuth app integration overview” page we are using the GitHub OAuth app as our URL pattern is pipelines/github/<organization name>.
  • I do see the Pipelines and also the Triggers menu items in the project settings
  • But when I navigate to Triggers → Add Trigger in the project settings and select “Custom Webhook” I just get an error message saying

Custom webhooks are currently only available to projects that have at least one GitHub App pipeline. Please set up a GitHub App pipeline in order to continue.

That seems to contradict that it should work for the OAuth app setup. I did try a logout/login (in case this is tied to just my user session) but that did not make a difference.

Ah, so the first step is to create a “pipeline” which will define the yaml file that you want to run when the custom webhook is triggered.

You do this in the “pipelines” tab. It will involve an installation of the GitHub App into your GitHub organization. The GitHub App & the oAauth App can live side-by-side just fine.

So you go to “Pipelines” tab first, create a “GitHub App pipeline” which involves installing the GitHub App into the org and telling us which filepath you want to use for your yml file (ie. .circleci/webhook.yml). Then once you’ve saved that “pipeline”, you go to “Triggers” and add the “custom webhook” trigger. When adding the “custom webhook” trigger, you’ll select the pipeline that you created in the first step to tell us to run that pipeline when the webhook is fired.

@martinth Any luck?

It is good to know it is tied to installing the GitHub App into the org. That is something I am currently not comfortable to do without clarifying with our security team. I will try it once I got a clarification if we are okay with that.

Sounds good. The nice thing about the GitHub App is that it requests significantly less permissions than the OAuth App that is already installed in your org. It also uses short-lived tokens by default and you can choose which repos CircleCI has access to instead of having to give access to all repos (which is the behavior of the OAuth App).

Let me know if I can help answer any questions for your security team.