[CircleCI Security Alert] Rotate any secrets stored in CircleCI

For GitHub, this means going to https://github.com/settings/applications, selecting “Authorized OAuth Apps”, then revoking the CircleCI entry. Once that’s done, log out and back into CircleCI to trigger reauthorization.

UPDATE: We have created a tool for discovering all your secrets on CircleCI.

The difference between this and the BASH script originally posted is that this will support both repositories that are under an individual user’s GitHub account, and repositories that are under a GitHub organization. As mentioned earlier, the BASH script version only works for repositories listed under a given GitHub org.


Original response

We have created the following gist as a guide to output a list of projects and contexts that currently contain env vars, and a link to the “Environment Variables” section of each of these projects, and a link to each context retrieved.

This was created by one of my colleagues on the support team - we are also in the process of creating another script that will be a bit more native, but we are offering this as a preliminary option for your teams to use.

Please be aware this method only works for GitHub users as this time

1 Like

OK thanks, sounds like we’ll need every user in our Org to perform this step manually.

We have put together a gist (linked in this thread) as a guide to output a list of projects and contexts that currently contain env vars.

We will provide more updates about the incident and share additional tool/scripts (if applicable) as they become available.

1 Like

We recommend rotating any and all secrets and env vars stored in CircleCI, out of an abundance of caution.

Thank you for your patience as we try to answer all of your questions. For easier access, we will be compiling answered questions on this thread as well. Please continue to ask questions here, and we will reply as quickly as possible.

1 Like

We will provide you updates about this incident, and our response, as they become available available to us.

Additionally, we’ve put together a compilation of Q&A from this thread, here, and will update our blog with new info regarding the incident.

1 Like

Thank you jerdog.
My question is, how do I even know, as a circleci admin, which users have api tokens that need rotating. Some people may not even know that they have api tokens, and some may belong to robot users. So is there a way to audit which users have api tokens?

Hello! Thanks for all this information and good luck in the nearest days.

Could you please clarify why CircleCI suggests to rotate all secrets and not to invalidate them for the first?

Ahh, ok. We are currently working on surfacing this to Admins, and hope to have some info around this.

We recommend rotating all secrets as this process will invalidate the existing secrets. The FAQ compilation and our support article provides guidance for rotating secrets.

We’ll provide you further updates about this incident as they become available available to us.

After revoking CircleCI access from BitBucket and re-authorising via signing back in to CircleCI using my BitBucket ID, I am no longer able to see my organisation - the ‘Connect’ button doesn’t even appear on the User Settings / Account Authorisations page. Id there something else that needs to be done to re-connect? (Yes, I have refreshed permissions, cleared cookies and logged in via a virgin browser…)

1 Like

Hi @scottatwinr, just to confirm are you on the account integrations section of the User Settings? https://app.circleci.com/settings/user - when I’ve looked to reproduce this by revoking on BB side and navigating back to the user settings page I am still seeing the Bitbucket section with the connect section. What do you see for BitBucket if you navigate to https://app.circleci.com/projects/connect-vcs/?create-new-organization? If you’d like to troubleshoot further, please write into support@circleci.com or submit a ticket through https://support.circleci.com/hc/en-us. Thanks!

FYI the blog post has been updated with additional information:

So the process I have to follow for this is:

  1. Get a list of users who have granted CircleCI Github access: (No idea how to do this)
  2. Tell all of those users to Revoke and Reauthorize
  3. Request audit logs from CircleCI
  4. Manually review audit logs to ensure all users from 1 revoked access…

There has to be a better way to do this @CircleCI

Thank you Circle team for the fast work today! We’re appreciating the new tool for discovering secrets.

It looks like that tool (and similar community scripts) can only access project and context environment variables, based on the capabilities of the underlying Circle API. However, one of the other primary recommendations is to rotate Project API tokens. This only appears to only be possible through the per-project UI. For organizations with hundreds of projects, this could be error-prone and hard to audit.

Do you have any recommendations for programmatically discovering Project-level API tokens?

Thanks!

1 Like

Can CircleCI automatically revoke all OAuth tokens forcing a new login without us having to ask hundreds or perhaps even thousands of users to do so manually?

3 Likes

Are secrets from projects or organizations hard- or soft-deleted? If the latter, could the threat actor have accessed them? How do we get a list of secrets that were in the system? Thank you.

A coworker and I are in the same boat as @scottatwinr: after revoking the CircleCI app authorization in Bitbucket, then signing back in, we’re unable to see the Bitbucket organization, and there’s no option to Connect in the user settings. The page you asked him to confirm is the one we’re trying. (I also previously had a GitHub connection, and that did work. I tried removing both, then signing in with Bitbucket alone to no avail, and did the same refresh/clear/incognito steps Scott mentioned above. My Bitbucket-only coworker has no ability to see any organizations or projects at all.)

I tried connecting through https://app.circleci.com/projects/connect-vcs/?create-new-organization as suggested, but that asks me to verify my email and results in a modal that says “Verification email failed to send.” Based on what appears in the console upon submission, that possibly appears to be due to a CORS error:

(Additionally, mine tried to send that email to the personal Gmail account tied to my connected GitHub account, rather than the work email that’s tied to my Bitbucket account. My coworker with only Bitbucket had the correct email address, but got the same error message.)

I’ve already logged a support ticket (#124615), but wanted to post here as well in case it helps more of us get it resolved more efficiently. Thank you!

EDIT: Just wanted to follow up here—a support engineer replied to my ticket and manually disconnected the link between CircleCI and my Bitbucket & GitHub accounts. From there, I was able to reauthenticate to Bitbucket, connect my account back to GitHub, and follow projects again.