[CircleCI Security Alert] Rotate any secrets stored in CircleCI

[jerdog]

@u-santos

Hi jerdog. With this failure of CircleCI, the organization I work for has more than 270 projects, and each one with its own secrets -aws, ssh, deployment, auth tokens… Can you give me some idea of what we can do? Is there any solution other than going from one to one changing all these secrets?
Is there any chance that this breach occur again or are we safe to rotate everything?

We appreciate the impact that this has. We currently do not have a specific script to go through and do this, but there are some community-generated ones above.

For more information, you can view the Support Article, and we will be sharing more information as it becomes available. In the meantime, you are safe to rotate going forward.

[AlonGluz]

Does it apply to self hosted runners? What is the expected remediation?
We are using the saas version with self hoster runners running in K8s?
Should I rotate the resource classes tokens?

[ring-pete]

Reading through the Rotating Secrets doc (https://support.circleci.com/hc/en-us/articles/11816211460891) for integrating with Github I’m a little confused. Does each individual user need to revoke their oAuth tokens in GitHub individual, or can we remove then re-approve CircleCI under the " Third-party application access policy" menu in GitHub?

[yannCI]

@tthayer

Is there a particular thing we should be looking for in the circle org audit log? I see a fair number of ‘unregistered’ values in the ACTOR_TYPE column but don’t know what that indicates.

These are users who do not have a CircleCI account, but interact with your repositories.

For example if someone, who has appropriate permission in the VCS repository, pushes a commit or open a pull-request that triggers a build in your CircleCI project, but this VCS user has not signed up with CircleCI, then the actor for that build will appear as unregistered.

[jerdog]

@AlonGluz

Does it apply to self hosted runners? What is the expected remediation?
We are using the saas version with self hoster runners running in K8s?
Should I rotate the resource classes tokens?

Yes, this applies to all tokens associated

[Stopher]

Could we start a thread here of any IOC’s that people have discovered in their investigations? We saw this one already: 54.145.167.181https://infosec.exchange/@sanitybit/109634301629418182
Can CircleCI start giving out some intel for the rest of us to keep our own orgs safe?

[jerdog]

@Vespinosa

So, are we safe rotating all the environment variables? How do we know that this is not still open/vulnerable? To be honest, the information provided about the issue is really poor.

At this point, we are confident that there are no unauthorized actors active in our systems; however, out of an abundance of caution, we want to ensure that all customers take certain preventative measures to protect your data as well. You are safe to rotate any and all secrets stored in CircleCI.

We will provide you updates about this incident, and our response, as they become available.

[jerdog]

@marguerite

Is there a way to audit which users have access tokens that should be rotated?

Out of an abundance of caution, it is important that you rotate any and all secrets stored in CircleCI. Please refer to the support article posted above

Rotating Secrets for January 4th Incident

[jerdog]

@ring-pete

Reading through the Rotating Secrets doc (https://support.circleci.com/hc/en-us/articles/11816211460891) for integrating with Github I’m a little confused. Does each individual user need to revoke their oAuth tokens in GitHub individual, or can we remove then re-approve CircleCI under the " Third-party application access policy" menu in GitHub?

For GitHub, this means going to https://github.com/settings/applications, selecting “Authorized OAuth Apps”, then revoking the CircleCI entry. Once that’s done, log out and back into CircleCI to trigger reauthorization.

[zhudson]

UPDATE: We have created a tool for discovering all your secrets on CircleCI.

The difference between this and the BASH script originally posted is that this will support both repositories that are under an individual user’s GitHub account, and repositories that are under a GitHub organization. As mentioned earlier, the BASH script version only works for repositories listed under a given GitHub org.

---

Original response

We have created the following gist as a guide to output a list of projects and contexts that currently contain env vars, and a link to the “Environment Variables” section of each of these projects, and a link to each context retrieved.

https://gist.github.com/yaningo/f3c62fda479467a50c43789ea736cff5

This was created by one of my colleagues on the support team - we are also in the process of creating another script that will be a bit more native, but we are offering this as a preliminary option for your teams to use.

Please be aware this method only works for GitHub users as this time

[ring-pete]

OK thanks, sounds like we’ll need every user in our Org to perform this step manually.

[nickokeefe]

@cyberplant

Do you have any script already done, or a docker image, that lists all the secrets that could have been compromised?

We have lots of repositories, so we need a tool to dump everything we have configured, rotate credentials and then run again to be sure everything was rotated.

I see some scripts but not sure if they are listing everything that could have been compromised.

We have put together a gist (linked in this thread) as a guide to output a list of projects and contexts that currently contain env vars.

We will provide more updates about the incident and share additional tool/scripts (if applicable) as they become available.

[nickokeefe]

@mrtc0

Hi, thank you for your incident response.

I noticed that projects that don’t already use CircleCI (deleted .circleci/config.yml) still have environment variables. (This can be viewed by specifying the repository name directly in the URL https://app.circleci.com/settings/project/github/ //environment-variables).

Do I need to rotate credentials for such projects as well?

We recommend rotating any and all secrets and env vars stored in CircleCI, out of an abundance of caution.

[emilycook]

Thank you for your patience as we try to answer all of your questions. For easier access, we will be compiling answered questions on this thread as well. Please continue to ask questions here, and we will reply as quickly as possible.

[FAQ Compilation] Security Alert - Rotate any secrets stored in CircleCI

On January 4, 2023, we identified a security incident that requires users to rotate any secrets stored in CircleCI. You can find the original forum post here: [CircleCI Security Alert] Rotate any secrets stored in CircleCI This post is to serve as a compiled list of questions and answers from the original post for easier access. Please continue to ask questions there, and we will update this post regularly as questions are answered. In addition, you can view the Support Article for more infor…

[nickokeefe]

We will provide you updates about this incident, and our response, as they become available available to us.

Additionally, we’ve put together a compilation of Q&A from this thread, here, and will update our blog with new info regarding the incident.

[marguerite]

Thank you jerdog.
My question is, how do I even know, as a circleci admin, which users have api tokens that need rotating. Some people may not even know that they have api tokens, and some may belong to robot users. So is there a way to audit which users have api tokens?

[circleci-user1]

Hello! Thanks for all this information and good luck in the nearest days.

Could you please clarify why CircleCI suggests to rotate all secrets and not to invalidate them for the first?

[jerdog]

@marguerite

My question is, how do I even know, as a circleci admin, which users have api tokens that need rotating. Some people may not even know that they have api tokens, and some may belong to robot users. So is there a way to audit which users have api tokens?

Ahh, ok. We are currently working on surfacing this to Admins, and hope to have some info around this.

[nickokeefe]

We recommend rotating all secrets as this process will invalidate the existing secrets. The FAQ compilation and our support article provides guidance for rotating secrets.

We’ll provide you further updates about this incident as they become available available to us.