[FAQ Compilation] Security Alert - Rotate any secrets stored in CircleCI

[emilycook]

On January 4, 2023, we identified a security incident that requires users to rotate any secrets stored in CircleCI.

You can find the original forum post here: [CircleCI Security Alert] Rotate any secrets stored in CircleCI

This post is to serve as a compiled list of questions and answers from the original post for easier access. Please continue to ask questions there, and we will update this post regularly as questions are answered.

In addition, you can view the Support Article for more information.

---

Update 01/06/2023

Since our last update, our team has addressed the following areas on behalf of customers. More information can be found on our original blog post, which will be updated regularly.

• Personal and Project API Tokens: We have removed all Personal and Project API Tokens created before 00:00 UTC on January 5, 2023.
• Bitbucket OAuth: As of 10:00 UTC on January 6, 2023 our partners at Atlassian expired all OAuth tokens for Bitbucket users. Bitbucket tokens will refresh for users upon login, and no additional action is needed here. Bitbucket users will still need to replace SSH tokens.
• GitHub OAuth: We are currently rotating all GitHub OAuth tokens on behalf of customers. We expect this process to be complete by 00:00 UTC on Jan 7, 2023. We will update here when this process is done. Customers who wish to rotate their own GitHub OAuth tokens may follow the directions below.

---

FAQ

What keys and tokens are affected?

A list of tokens and how to rotate them can be found here: https://support.circleci.com/hc/en-us/articles/11816211460891

What about the source code itself?

We recommend viewing the security audit logs of your VCS for any unauthorized access.

How can I access my audit logs for CircleCI?

We have expanded access to self-serve audit logs to all customers, including free customers. Customers can access self-serve audit logs via our UI. Customers can query up to 30 days of data and have 30 days to download the resulting logs. While we understand the requests for access to CircleCI audit logs, our recommendation to all customers is to focus your audits and investigations on the logs of any systems which had secrets stored in CircleCI.

Is it safe to add new secrets in CircleCI?

Yes, it is safe to add new secrets to CircleCI at this time.

Can I build?

Yes, it is safe to build. We’re confident that we have eliminated the risk that led to this incident.

What resources can help make this faster?

Please refer to this post below.

Does this incident impact self-hosted CircleCI environments in addition to the SaaS version?

For CircleCI server customers (full self-hosted installation), this alert does not apply

Are self-hosted runners compromised too ? And so all others secrets injection outside of CircleCI context/variables?

Yes, this would apply as well.

Other than leaked credentials & secrets, is there any chance for attackers to have injected code or tampered with our builds?

We advise that you check the logs of any systems which had secrets stored in CircleCI.

Will CircleCI be updating the official blog post with additional guidance from questions/answers here? Or is this the place to look for the most accurate and updated information?

We will be sending updates as we’re able via email and our blog, as well as answering questions on the forum post.

How does CircleCI encrypt secrets in it’s backend? Was it a single encryption key the was “leaked”? What actions were taken to ensure environment integrity was restored?

You can read more about how we encrypt secrets and sensitive data in our security policy. We cannot share details about what was leaked and any remediation actions at this time outside of what has already been publicly disclosed. We are committed to sharing more details with customers in the coming days.

Should we rotate the deploy keys in each project?

Yes, we advise rotating the keys for each project.

If we pull in secrets from a secrets manager during a job, should we also be concerned about the compromise of those secrets, and thus rotate them?

There is no indication that secrets in jobs were compromised. Out of an abundance of caution, users should rotate these secrets. But, they can be prioritized lower than ones in contexts and environment variables.

How does this incident affect OIDC authentication against AWS services and do we need to take any further action in that direction?

At this point, we do not anticipate any further action needed concerning OIDC/AWS authentication.

Does each individual user need to revoke their oAuth tokens in GitHub?

We are rotating all GitHub OAuth tokens on behalf of customers and will update when this process is complete. Customers who wish to do so may rotate their own OAuth tokens by logging out of the CircleCI application, going to https://github.com/settings/applications, selecting “Authorized OAuth Apps”, and then revoking the CircleCI entry. Once that’s done, log back into CircleCI to trigger reauthorization.

Update: We expect this process to be complete by 00:00 UTC on Jan 7, 2023

Is there a particular thing we should be looking for in the circle org audit log? I see a fair number of ‘unregistered’ values in the ACTOR_TYPE column but don’t know what that indicates.

These are users who do not have a CircleCI account, but interact with your repositories.

For if someone, who has appropriate permission in the VCS repository, pushes a commit or open a pull-request that triggers a build in your CircleCI project, but this VCS user has not signed up with CircleCI, then the actor for that build will appear as unregistered.

If CCI can provide more insight into attack signature and what prompted the recommendation to rotate these creds, that’d be much appreciated to help us further assess if we were directly impacted.

Unfortunately, at this time we do not have any additional insight to provide. We will be sending updates as we’re able via email and our blog.

For GitHub OAuth, this will not affect the projects that are configured already in CircleCi, correct? I.e. removing the GH app will not trigger some cleanup job on the circle side?

Correct, re-authorizing does not delete projects/orgs/builds/etc.

Do we have documentation on how to access these logs and review them?

If you are looking for audit logs from CircleCI, you can request them from settings. Documentation on this feature is here: Audit logs - CircleCI 171

Could you please clarify why CircleCI suggests to rotate all secrets and not to invalidate them for the first?

Rotating secrets will invalidate the existing ones, so it’s not necessary to invalidate them first.

Are variables piped in from .env files in Github affected and need to be rotated?

No, they are not affected.

I am unable to reconnect my Bitbucket account to CircleCI after revoking access and/or I would like Bitbucket removed as an integration. How can I get this resolved?

Please submit a support request. For this particular incident, we are working to address free accounts as quickly as possible. If you have opened a support ticket and still have not heard back, post on the original forum topic with your ticket ID and/or subject line and we will get back to you as quickly as possible.

[emilycook]

What resources can help make this process faster?

Below is a tool for discovering all of your secrets on CircleCI. Note that this only works for GitHub users.

[CircleCI Security Alert] Rotate any secrets stored in CircleCI

UPDATE: We have created a tool for discovering all your secrets on CircleCI. The difference between this and the BASH script originally posted is that this will support both repositories that are under an individual user’s GitHub account, and repositories that are under a GitHub organization. As mentioned earlier, the BASH script version only works for repositories listed under a given GitHub org. Original response We have created the following gist as a guide to output a list of projects an…

---

Here are some additional resources that may be helpful:

https://circleci.com/docs/api/v2/index.html#tag/Context

^ this API will be helpful for quickly cycling keys

For project env vars:

https://circleci.com/docs/api/v2/index.html#operation/listEnvVars

Customers can use the create endpoints with the same env var name to replace them:

https://circleci.com/docs/api/v2/index.html#operation/addEnvironmentVariableToContext

https://circleci.com/docs/api/v2/index.html#operation/createEnvVar

---

Listed below are some community-generated scripts you can use. We are still reviewing them as endorsed solutions.

[CircleCI Security Alert] Rotate any secrets stored in CircleCI

I used the github CLI to get a list of all github repos gh api -H "Accept: application/vnd.github+json" '/orgs/ORG/repos?per_page=100&page=1' | jq 'map(.full_name) | .[]' -r > ./repos.txt gh api -H "Accept: application/vnd.github+json" '/orgs/ORG/repos?per_page=100&page=2' | jq 'map(.full_name) | .[]' -r >> ./repos.txt gh api -H "Accept: application/vnd.github+json" '/orgs/ORG/repos?per_page=100&page=3' | jq 'map(.full_name) | .[]' -r >> ./repos.txt gh api -H "Accept: application/vnd.github+jso…

[CircleCI Security Alert] Rotate any secrets stored in CircleCI

I wrote a script. If it’s helpful then I’m happy.

[CircleCI Security Alert] Rotate any secrets stored in CircleCI

For those working with circleCI and GitHub, here is a script we did today QuickStart python main.py list-ctx python main.py list-env python main.py list-keys python main.py replace-keys with the following main.py file import requests import argparse from config import API_TOKEN, GH_TOKEN, ORG # CircleCi api Token, Github api token, Github organisation name ORG_SLUG = f"github/{ORG}" auth_header = { 'Circle-Token': API_TOKEN } github_headers = { "X-GitHub-Api-Version": "2022-11-28", …

[CircleCI Security Alert] Rotate any secrets stored in CircleCI

In case it helps anybody else, I’ve produced this tool for listing my CircleCi secrets: GitHub - rupert-madden-abbott/circleci-audit Looks like CircleCI have their own tool now as well but it’s not clear to me what it covers. My tool handles: Project Env Vars, Keys and Jira Integration Context Env Vars