[CircleCI Security Alert] Rotate any secrets stored in CircleCI

[emilycook]

A new update has been pushed to the blog post:

Security update 01/06/2023

Our team is working to take every action available to assist customers in the mitigation of this incident.

Since our last update, our team has addressed the following areas on behalf of customers:

• Personal and Project API Tokens: We have removed all Personal and Project API Tokens created before 00:00 UTC on January 5, 2023.
• Bitbucket OAuth: As of 10:00 UTC on January 6, 2023 our partners at Atlassian expired all OAuth tokens for Bitbucket users. Bitbucket tokens will refresh for users upon login, and no additional action is needed here. Bitbucket users will still need to replace SSH tokens.
• GitHub OAuth: We are currently rotating all GitHub OAuth tokens on behalf of customers. We expect this process to be complete by 00:00 UTC on Jan 7, 2023. We will update here when this process is done. Customers who wish to rotate their own GitHub OAuth tokens may follow the directions below.

CircleCI security alert: Rotate any secrets stored in CircleCI (Updated Jan 6)

Sorry if I miss tagging anyone, trying to get everyone who posted something related to this:
@duffn @traviscrist @ring-pete @david-davidson @atnak @microbit-matth @bwalding @TheMetalCode @agius

Thank you all for your patience as we try to get answers for other outstanding questions.

[jscheel]

Just for clarity, are you rotating all github tokens, even ones that have been manually rotated since the breach was announced?

[emilycook]

@jscheel

Just for clarity, are you rotating all github tokens, even ones that have been manually rotated since the breach was announced?

Yes, we’ve kicked off a queue of all users to rotate in addition to the notice sent. So it’s possible a manually rotated token will be rotated again, but there is no detriment or issue with rotating tokens multiple times. Sorry for any confusion this may cause!

[paulmiller3000]

Be careful doing this. As others have mentioned, if you have Bitbucket, be prepared to wait for a few days for free support to help you, or pony up for premium support.

That’s because CircleCI will NOT remove the Bitbucket integration on their end and your projects will error. According to everything we’ve learned, only CircleCI can remove what are essentially “ghost tokens.” So, we have to wait…

[DrTorte]

@paulmiller3000

Be careful doing this. As others have mentioned, if you have Bitbucket, be prepared to wait for a few days for free support to help you, or pony up for premium support.

That’s because CircleCI will NOT remove the Bitbucket integration on their end and your projects will error. According to everything we’ve learned, only CircleCI can remove what are essentially “ghost tokens.” So, we have to wait…

Hey there,

For this particular matter, we’re making sure free accounts are being addressed as quickly as possible. If you haven’t heard back yet, please feel free to shoot me a message here with the ticket ID and/or the subject line.

[paulmiller3000]

@DrTorte thank you for the quick reply. My ticket number is 124712. You can also see additional information (and more people affected) here: Cannot reconnect BitBucket account

If we can just get the error on https://app.circleci.com/projects/project-dashboard/github/{my-project}/ cleared, I’ll be happy. We took this opportunity to ditch Bitbucket, but CircleCI still errors out.

[nickokeefe]

@andyshinn

The wording in the article for API tokens says rotated which to me sounded like a new token was generated in its place. But do you really mean tokens were removed?

Hey Andy, we just updated the language in the support article to better reflect the latest updates in the blog post and the FAQ Compilation. To confirm though, CircleCI has revoked all personal and project API tokens created before 00:00 UTC on January 5, 2023.

@andyshinn

Also, this broke a lot of my automation suddenly. I am surprised there was no warning on this. We were using API token auth to rotate our secrets…

Apologies for my late reply and the disruption this has caused; this could be communicated more clearly going forward.

[DrTorte]

@paulmiller3000

@DrTorte thank you for the quick reply. My ticket number is 124712. You can also see additional information (and more people affected) here: Cannot reconnect BitBucket account

If we can just get the error on https://app.circleci.com/projects/project-dashboard/github/{my-project}/ cleared, I’ll be happy. We took this opportunity to ditch Bitbucket, but CircleCI still errors out.

Thanks! I just responded in the ticket, and we’ll continue the conversation there.

All the best,

Kris

[dtcsight]

“Delete the deploy key and add it again.” Nope, you still have to manually delete the associated SSH key after that.

[DrTorte]

Hey there,

I would definitely suggest rotating these as well. I’m afraid that you are correct in regards to how to get them out. If it is a private project, one of our colleagues created an (unofficial) config that might be helpful:

GitHub - kelvintaywl-cci/get-project-env-vars: Extracting project environment...

Absolutely do NOT run this on a public repo. And as with anything else, please make sure you look through it before you use it so that you can validate it.

[dtcsight]

@nickokeefe

Hey Andy, we just updated the language in the support article to better reflect the latest updates in the blog post and the FAQ Compilation. To confirm though, CircleCI has revoked all personal and project API tokens created before 00:00 UTC on January 5, 2023.

is this why my might’ve build broke this morning even though i already reset my keys and removed the old one?

[DrTorte]

@dtcsight

“Delete the deploy key and add it again.” Nope, you still have to manually delete the associated SSH key after that.

Hey there,

In regards to your SSH key, that seems unusual and not as expected. Would you mind trying again? If it doesn’t delete properly, could you let us know where you’re seeing them?

@dtcsight

is this why my might’ve build broke this morning even though i already reset my keys and removed the old one?

In regards to builds breaking this morning: If the Personal or Project API Tokens used were created before January 5th, 00:00 UTC, then they would have been removed.

If the tokens were from after that point, though, they should not have been revoked. Is there a more specific error message?

[jchen123]

please help me understand - how is it possible that unencrypted environment variables fell into the hands of an attacker?

[bwalding]

@bwalding

Is there a reason CircleCI hasn’t clicked on “Revoke all user tokens” in their GitHub OAuth app to do this for everyone?

It’s actually very difficult to do this properly on the customer / client side because we have to somehow convince every user of our org who might have created an OAuth token for CircleCI to go and do this - 100s of users potentially - some of whom might be on leave etc.

CircleCI is now doing this on their side - see their blog updates.

GitHub OAuth: We are currently rotating all GitHub OAuth tokens on behalf of customers. We expect this process to be complete by 00:00 UTC on Jan 7, 2023. We will update here when this process is done. Customers who wish to rotate their own GitHub OAuth tokens may follow the directions below.

[voidecker]

Thanks to the CircleCI team for keeping us updated in this thread.

One thing I’d like to understand the implications of, following-up on Step 6 of the updated instructions:

CircleCI security alert: Rotate any secrets stored in CircleCI (Updated Jan 6)

Regarding Project SSH keys:

6. Project SSH keys:
  1. Go to Project Settings > SSH Keys.
  2. Delete the Deploy Key and add it again.
  3. If you were using any additional keys, then those need to be deleted and recreated.
  4. Note: We recommend this for all projects (“go to Project Settings”), orgs (”go to Organization 
  Settings”), and users (“go to User Settings”).

Would the actor(s) who maliciously gained access to the Deploy SSH Keys also know the GitHub (or any other applicable VCS) org and project name/alias that the key is associated with?

Currently trying to understand the risks of someone accessing private source code, and I imagine knowing the github.com/$ORG_NAME/$REPO_NAME would be key to leveraging the Deploy SSH Keys.

[yannCI]

@squaringthecircle

Is there any way of extracting the legacy AWS_ACCESS_KEY_ID from CircleCI? The UI gives the option to delete the legacy AWS credentials, but you can’t actually see what the ID is. This is important to be able to revoke/rotate the affected AWS access key. Is there an API/UI for finding this information?

I know there is the option of making a config.yml which spits out the value, but that doesn’t scale well for many repos.

@squaringthecircle,
Whether you retrieved them via the UI or the CircleCI API, the value of environment variables is always masked.

Also, by default the secrets-masking feature is enabled for all projects, so the values can’t be printed out in a build job; unless we specifically disable said feature; we can do so on a per-project basis, and upon your explicit request.

[yannCI]

@jscheel

Just for clarity, are you rotating all github tokens, even ones that have been manually rotated since the breach was announced?

@jscheel,
This is likely how you understood it, but I want to make sure this topic really clear.

The GitHub tokens we’ve rotated are the OAuth tokens; the one that are initially automatically created by the CircleCI-GitHub integration when a user signed up to CircleCI via GitHub.

These are not the GitHub personal access tokens you might have created yourself under your GitHub account; we have no access to these.

Unless you stored any GitHub personal access tokens as an environment variable in a CircleCI project or context, you don’t specifically need to rotate them; although it is good practice to do so regularly.

[yannCI]

@rupert-madden-abbott

With Jira integration, each project has a token generated by a CircleCI plugin in Jira and then added to each project’s settings page via the Jira section.

 

Do we need to rotate this secret?

Yes.
 

How do we rotate this secret in the CircleCI plugin in Jira? The secret appears to be hardcoded and I can’t see how to regenerate it.

You need to uninstall the “CircleCI for Jira” app in your Jira instance; this will automatically revoke the authentication in your project’s “Jira Integrations” section.

Next, you need to re-install the “CircleCI for Jira” app; doing so will generate a new token (in Jira) that you can then add to your project’s “Jira Integrations” settings.

[makotoshimazu]

Hi,

Step 6-4 in “Updated instructions as of 17:52 UTC on January 6” says “Note: We recommend this for all projects (“go to Project Settings”), orgs (”go to Organization Settings”), and users (“go to User Settings”).”, but as far as I checked app.circleci.com/settings/organization/github/${MY_ORG}/overview and app.circleci.com/settings/user , no SSH key related settings seems to exist.

Could anyone help me to understand what are the exact steps?

[arvid-m]

Hi, regarding this quote:

OAuth tokens : For GitHub: As of 07:30 UTC on January 7, all GitHub OAuth tokens have been rotated on behalf of CircleCI customers. Customers who wish to do so may rotate their own OAuth tokens by logging out of the CircleCI application, going to Sign in to GitHub · GitHub, selecting “Authorized OAuth Apps”, and then revoking the CircleCI entry. Once that’s done, log back into CircleCI to trigger reauthorization.

I don’t understand the phrasing. The first sentence claims that the token has already been rotated, but the second sentence claims that we should rotate it. Does that mean that we need to revoke and reauthorize the access of circleci in Github or not?