[CircleCI Security Alert] Rotate any secrets stored in CircleCI

For CircleCI server customers (full self-hosted installation), this alert does not apply

We appreciate the impact that this has. We currently do not have a specific script to go through and do this, but there are some community-generated ones above.

For more information, you can view the Support Article, and we will be sharing more information as it becomes available. In the meantime, you are safe to rotate going forward.

Does it apply to self hosted runners? What is the expected remediation?
We are using the saas version with self hoster runners running in K8s?
Should I rotate the resource classes tokens?

Reading through the Rotating Secrets doc (https://support.circleci.com/hc/en-us/articles/11816211460891) for integrating with Github I’m a little confused. Does each individual user need to revoke their oAuth tokens in GitHub individual, or can we remove then re-approve CircleCI under the " Third-party application access policy" menu in GitHub?

These are users who do not have a CircleCI account, but interact with your repositories.

For example if someone, who has appropriate permission in the VCS repository, pushes a commit or open a pull-request that triggers a build in your CircleCI project, but this VCS user has not signed up with CircleCI, then the actor for that build will appear as unregistered.

1 Like

Yes, this applies to all tokens associated

Could we start a thread here of any IOC’s that people have discovered in their investigations? We saw this one already: 54.145.167.181https://infosec.exchange/@sanitybit/109634301629418182
Can CircleCI start giving out some intel for the rest of us to keep our own orgs safe?

2 Likes

At this point, we are confident that there are no unauthorized actors active in our systems; however, out of an abundance of caution, we want to ensure that all customers take certain preventative measures to protect your data as well. You are safe to rotate any and all secrets stored in CircleCI.

We will provide you updates about this incident, and our response, as they become available.

Out of an abundance of caution, it is important that you rotate any and all secrets stored in CircleCI. Please refer to the support article posted above

For GitHub, this means going to https://github.com/settings/applications, selecting “Authorized OAuth Apps”, then revoking the CircleCI entry. Once that’s done, log out and back into CircleCI to trigger reauthorization.

UPDATE: We have created a tool for discovering all your secrets on CircleCI.

The difference between this and the BASH script originally posted is that this will support both repositories that are under an individual user’s GitHub account, and repositories that are under a GitHub organization. As mentioned earlier, the BASH script version only works for repositories listed under a given GitHub org.


Original response

We have created the following gist as a guide to output a list of projects and contexts that currently contain env vars, and a link to the “Environment Variables” section of each of these projects, and a link to each context retrieved.

This was created by one of my colleagues on the support team - we are also in the process of creating another script that will be a bit more native, but we are offering this as a preliminary option for your teams to use.

Please be aware this method only works for GitHub users as this time

1 Like

OK thanks, sounds like we’ll need every user in our Org to perform this step manually.

We have put together a gist (linked in this thread) as a guide to output a list of projects and contexts that currently contain env vars.

We will provide more updates about the incident and share additional tool/scripts (if applicable) as they become available.

1 Like

We recommend rotating any and all secrets and env vars stored in CircleCI, out of an abundance of caution.

Thank you for your patience as we try to answer all of your questions. For easier access, we will be compiling answered questions on this thread as well. Please continue to ask questions here, and we will reply as quickly as possible.

1 Like

We will provide you updates about this incident, and our response, as they become available available to us.

Additionally, we’ve put together a compilation of Q&A from this thread, here, and will update our blog with new info regarding the incident.

1 Like

Thank you jerdog.
My question is, how do I even know, as a circleci admin, which users have api tokens that need rotating. Some people may not even know that they have api tokens, and some may belong to robot users. So is there a way to audit which users have api tokens?

Hello! Thanks for all this information and good luck in the nearest days.

Could you please clarify why CircleCI suggests to rotate all secrets and not to invalidate them for the first?

Ahh, ok. We are currently working on surfacing this to Admins, and hope to have some info around this.