You can find the original forum post here: [CircleCI Security Alert] Rotate any secrets stored in CircleCI
This post is to serve as a compiled list of questions and answers from the original post for easier access. Please continue to ask questions there, and we will update this post regularly as questions are answered.
In addition, you can view the Support Article for more information.
Since our last update, our team has addressed the following areas on behalf of customers. More information can be found on our original blog post, which will be updated regularly.
- Personal and Project API Tokens: We have removed all Personal and Project API Tokens created before 00:00 UTC on January 5, 2023.
- Bitbucket OAuth: As of 10:00 UTC on January 6, 2023 our partners at Atlassian expired all OAuth tokens for Bitbucket users. Bitbucket tokens will refresh for users upon login, and no additional action is needed here. Bitbucket users will still need to replace SSH tokens.
- GitHub OAuth: We are currently rotating all GitHub OAuth tokens on behalf of customers. We expect this process to be complete by 00:00 UTC on Jan 7, 2023. We will update here when this process is done. Customers who wish to rotate their own GitHub OAuth tokens may follow the directions below.
We recommend viewing the security audit logs of your VCS for any unauthorized access.
We have expanded access to self-serve audit logs to all customers, including free customers. Customers can access self-serve audit logs via our UI. Customers can query up to 30 days of data and have 30 days to download the resulting logs. While we understand the requests for access to CircleCI audit logs, our recommendation to all customers is to focus your audits and investigations on the logs of any systems which had secrets stored in CircleCI.
Yes, it is safe to add new secrets to CircleCI at this time.
Yes, it is safe to build. We’re confident that we have eliminated the risk that led to this incident.
For CircleCI server customers (full self-hosted installation), this alert does not apply
Are self-hosted runners compromised too ? And so all others secrets injection outside of CircleCI context/variables?
Yes, this would apply as well.
Other than leaked credentials & secrets, is there any chance for attackers to have injected code or tampered with our builds?
We advise that you check the logs of any systems which had secrets stored in CircleCI.
Will CircleCI be updating the official blog post with additional guidance from questions/answers here? Or is this the place to look for the most accurate and updated information?
We will be sending updates as we’re able via email and our blog, as well as answering questions on the forum post.
How does CircleCI encrypt secrets in it’s backend? Was it a single encryption key the was “leaked”? What actions were taken to ensure environment integrity was restored?
You can read more about how we encrypt secrets and sensitive data in our security policy. We cannot share details about what was leaked and any remediation actions at this time outside of what has already been publicly disclosed. We are committed to sharing more details with customers in the coming days.
Yes, we advise rotating the keys for each project.
If we pull in secrets from a secrets manager during a job, should we also be concerned about the compromise of those secrets, and thus rotate them?
There is no indication that secrets in jobs were compromised. Out of an abundance of caution, users should rotate these secrets. But, they can be prioritized lower than ones in contexts and environment variables.
How does this incident affect OIDC authentication against AWS services and do we need to take any further action in that direction?
At this point, we do not anticipate any further action needed concerning OIDC/AWS authentication.
We are rotating all GitHub OAuth tokens on behalf of customers and will update when this process is complete. Customers who wish to do so may rotate their own OAuth tokens by logging out of the CircleCI application, going to Sign in to GitHub · GitHub, selecting “Authorized OAuth Apps”, and then revoking the CircleCI entry. Once that’s done, log back into CircleCI to trigger reauthorization.
Update: We expect this process to be complete by 00:00 UTC on Jan 7, 2023
Is there a particular thing we should be looking for in the circle org audit log? I see a fair number of ‘unregistered’ values in the ACTOR_TYPE column but don’t know what that indicates.
These are users who do not have a CircleCI account, but interact with your repositories.
For if someone, who has appropriate permission in the VCS repository, pushes a commit or open a pull-request that triggers a build in your CircleCI project, but this VCS user has not signed up with CircleCI, then the actor for that build will appear as
If CCI can provide more insight into attack signature and what prompted the recommendation to rotate these creds, that’d be much appreciated to help us further assess if we were directly impacted.
Unfortunately, at this time we do not have any additional insight to provide. We will be sending updates as we’re able via email and our blog.
For GitHub OAuth, this will not affect the projects that are configured already in CircleCi, correct? I.e. removing the GH app will not trigger some cleanup job on the circle side?
Correct, re-authorizing does not delete projects/orgs/builds/etc.
If you are looking for audit logs from CircleCI, you can request them from settings. Documentation on this feature is here: Audit logs - CircleCI 171
Could you please clarify why CircleCI suggests to rotate all secrets and not to invalidate them for the first?
Rotating secrets will invalidate the existing ones, so it’s not necessary to invalidate them first.
No, they are not affected.
I am unable to reconnect my Bitbucket account to CircleCI after revoking access and/or I would like Bitbucket removed as an integration. How can I get this resolved?
Please submit a support request. For this particular incident, we are working to address free accounts as quickly as possible. If you have opened a support ticket and still have not heard back, post on the original forum topic with your ticket ID and/or subject line and we will get back to you as quickly as possible.