[CircleCI Security Alert] Rotate any secrets stored in CircleCI

[agius]

@aaronstillwell I think y’all have to hit “Revoke all user tokens” on your Github app, as per @bwalding 's comment. As Github Enterprise customers, if we deauthorize the CircleCI app for our org, or remove engineers from our org, the CircleCI tokens stop having access to company resources. Cool! That’s what we want.

But if we reinstate those engineers or re-enable the app for our org, the potentially-compromised tokens start working again. Users do not have to re-authorize CircleCI with Github to continue using CircleCI and see company projects and resources. The compromised tokens do not get fully invalidated when the users or app gets removed.

So in short, it looks like there is nothing we can do, even as Github Business / Enterprise owners to nuke the Github access tokens provisioned by CircleCI. But if y’all hit that button, users should have to re-authorize and the potentially-compromised tokens should be nuked. Please hit it ASAP. Thanks!

[jerdog]

Sorry for the delay in the response. Answering in-line:

@aaroncarson_2

• Do we know whether CircleCI’s platform was used for any access to customers’ systems or whether secrets were just grabbed and run off with?

More information will be shared on this as it becomes available. We will provide updates at that time on the original blog post.

@aaroncarson_2

• How regularly do the IPs on all.knownips.circleci.com change? And if they’ve changed recently, please can we have a list of all the previous IPs?

We have not changed our IP since April 2022.

[jerdog]

@dkinzer

Were repositories also exposed. We deploy using an ansible playbook repository. Secretes are stored encrypted in the playbook but we un-encrypt at deploy time. Do we need to assume that all secrets that are stored in the playbook have also been exposed?

We recommend checking your audit logs on Github to see if repo were accessed. If the key used for decryption was stored as a secret you will want to rotate.

[jerdog]

@korrupt

will this discussion serve as a way to share any indicators of compromise that we can use to search through our logs or will some other form of communication be shared?

No update at this time, but once there is we will post here as well as update the original blog post as done yesterday.

[squaringthecircle]

Is there any way of extracting the legacy AWS_ACCESS_KEY_ID from CircleCI? The UI gives the option to delete the legacy AWS credentials, but you can’t actually see what the ID is. This is important to be able to revoke/rotate the affected AWS access key. Is there an API/UI for finding this information?

I know there is the option of making a config.yml which spits out the value, but that doesn’t scale well for many repos.

[nickokeefe]

Following the steps in our support article, we recommend rotating all secrets listed in that page out of an abundance of caution.

Additionally, we’ve updated the page to share that we’re rotating all GitHub OAuth tokens for all users, and will provide an update when the process is complete.

[squaringthecircle]

That page doesn’t mention legacy AWS credentials which you are holding for certain projects as identified by - https://support.circleci.com/hc/en-us/articles/360021415793-Wrong-AWS-credentials-being-used

Does that mean those are safe and do not need to be rotated? How can we rotate them if we don’t know what they are?

[andyshinn]

The wording in the article for API tokens says rotated which to me sounded like a new token was generated in its place. But do you really mean tokens were removed?

Also, this broke a lot of my automation suddenly. I am surprised there was no warning on this. We were using API token auth to rotate our secrets…

[emilycook]

A new update has been pushed to the blog post:

Security update 01/06/2023

Our team is working to take every action available to assist customers in the mitigation of this incident.

Since our last update, our team has addressed the following areas on behalf of customers:

• Personal and Project API Tokens: We have removed all Personal and Project API Tokens created before 00:00 UTC on January 5, 2023.
• Bitbucket OAuth: As of 10:00 UTC on January 6, 2023 our partners at Atlassian expired all OAuth tokens for Bitbucket users. Bitbucket tokens will refresh for users upon login, and no additional action is needed here. Bitbucket users will still need to replace SSH tokens.
• GitHub OAuth: We are currently rotating all GitHub OAuth tokens on behalf of customers. We expect this process to be complete by 00:00 UTC on Jan 7, 2023. We will update here when this process is done. Customers who wish to rotate their own GitHub OAuth tokens may follow the directions below.

CircleCI security alert: Rotate any secrets stored in CircleCI (Updated Jan 6)

Sorry if I miss tagging anyone, trying to get everyone who posted something related to this:
@duffn @traviscrist @ring-pete @david-davidson @atnak @microbit-matth @bwalding @TheMetalCode @agius

Thank you all for your patience as we try to get answers for other outstanding questions.

[jscheel]

Just for clarity, are you rotating all github tokens, even ones that have been manually rotated since the breach was announced?

[emilycook]

@jscheel

Just for clarity, are you rotating all github tokens, even ones that have been manually rotated since the breach was announced?

Yes, we’ve kicked off a queue of all users to rotate in addition to the notice sent. So it’s possible a manually rotated token will be rotated again, but there is no detriment or issue with rotating tokens multiple times. Sorry for any confusion this may cause!

[paulmiller3000]

Be careful doing this. As others have mentioned, if you have Bitbucket, be prepared to wait for a few days for free support to help you, or pony up for premium support.

That’s because CircleCI will NOT remove the Bitbucket integration on their end and your projects will error. According to everything we’ve learned, only CircleCI can remove what are essentially “ghost tokens.” So, we have to wait…

[DrTorte]

@paulmiller3000

Be careful doing this. As others have mentioned, if you have Bitbucket, be prepared to wait for a few days for free support to help you, or pony up for premium support.

That’s because CircleCI will NOT remove the Bitbucket integration on their end and your projects will error. According to everything we’ve learned, only CircleCI can remove what are essentially “ghost tokens.” So, we have to wait…

Hey there,

For this particular matter, we’re making sure free accounts are being addressed as quickly as possible. If you haven’t heard back yet, please feel free to shoot me a message here with the ticket ID and/or the subject line.

[paulmiller3000]

@DrTorte thank you for the quick reply. My ticket number is 124712. You can also see additional information (and more people affected) here: Cannot reconnect BitBucket account

If we can just get the error on https://app.circleci.com/projects/project-dashboard/github/{my-project}/ cleared, I’ll be happy. We took this opportunity to ditch Bitbucket, but CircleCI still errors out.

[nickokeefe]

@andyshinn

The wording in the article for API tokens says rotated which to me sounded like a new token was generated in its place. But do you really mean tokens were removed?

Hey Andy, we just updated the language in the support article to better reflect the latest updates in the blog post and the FAQ Compilation. To confirm though, CircleCI has revoked all personal and project API tokens created before 00:00 UTC on January 5, 2023.

@andyshinn

Also, this broke a lot of my automation suddenly. I am surprised there was no warning on this. We were using API token auth to rotate our secrets…

Apologies for my late reply and the disruption this has caused; this could be communicated more clearly going forward.

[DrTorte]

@paulmiller3000

@DrTorte thank you for the quick reply. My ticket number is 124712. You can also see additional information (and more people affected) here: Cannot reconnect BitBucket account

If we can just get the error on https://app.circleci.com/projects/project-dashboard/github/{my-project}/ cleared, I’ll be happy. We took this opportunity to ditch Bitbucket, but CircleCI still errors out.

Thanks! I just responded in the ticket, and we’ll continue the conversation there.

All the best,

Kris

[dtcsight]

“Delete the deploy key and add it again.” Nope, you still have to manually delete the associated SSH key after that.

[DrTorte]

Hey there,

I would definitely suggest rotating these as well. I’m afraid that you are correct in regards to how to get them out. If it is a private project, one of our colleagues created an (unofficial) config that might be helpful:

GitHub - kelvintaywl-cci/get-project-env-vars: Extracting project environment...

Absolutely do NOT run this on a public repo. And as with anything else, please make sure you look through it before you use it so that you can validate it.

[dtcsight]

@nickokeefe

Hey Andy, we just updated the language in the support article to better reflect the latest updates in the blog post and the FAQ Compilation. To confirm though, CircleCI has revoked all personal and project API tokens created before 00:00 UTC on January 5, 2023.

is this why my might’ve build broke this morning even though i already reset my keys and removed the old one?

[DrTorte]

@dtcsight

“Delete the deploy key and add it again.” Nope, you still have to manually delete the associated SSH key after that.

Hey there,

In regards to your SSH key, that seems unusual and not as expected. Would you mind trying again? If it doesn’t delete properly, could you let us know where you’re seeing them?

@dtcsight

is this why my might’ve build broke this morning even though i already reset my keys and removed the old one?

In regards to builds breaking this morning: If the Personal or Project API Tokens used were created before January 5th, 00:00 UTC, then they would have been removed.

If the tokens were from after that point, though, they should not have been revoked. Is there a more specific error message?