Here is some follow up from CircleCI support, these steps worked for us to fix the issue:
This will usually resolve itself, given a little time, but you can speed this up by resetting the webhooks. To do this:
- Remove all CircleCI webhooks and services from your GitHub or Bitbucket repo settings. (Check under Settings > Webhooks on the relevant service.)
- Tell the project to stop building on CircleCI through “Project settings”. (Click the gear icon in the upper right corner of the page listing all recent builds for that project.)
- Add the project back on CircleCI.