[CircleCI Security Alert] Rotate any secrets stored in CircleCI

[jchen123]

please help me understand - how is it possible that unencrypted environment variables fell into the hands of an attacker?

[bwalding]

@bwalding

Is there a reason CircleCI hasn’t clicked on “Revoke all user tokens” in their GitHub OAuth app to do this for everyone?

It’s actually very difficult to do this properly on the customer / client side because we have to somehow convince every user of our org who might have created an OAuth token for CircleCI to go and do this - 100s of users potentially - some of whom might be on leave etc.

CircleCI is now doing this on their side - see their blog updates.

GitHub OAuth: We are currently rotating all GitHub OAuth tokens on behalf of customers. We expect this process to be complete by 00:00 UTC on Jan 7, 2023. We will update here when this process is done. Customers who wish to rotate their own GitHub OAuth tokens may follow the directions below.

[voidecker]

Thanks to the CircleCI team for keeping us updated in this thread.

One thing I’d like to understand the implications of, following-up on Step 6 of the updated instructions:

CircleCI security alert: Rotate any secrets stored in CircleCI (Updated Jan 6)

Regarding Project SSH keys:

6. Project SSH keys:
  1. Go to Project Settings > SSH Keys.
  2. Delete the Deploy Key and add it again.
  3. If you were using any additional keys, then those need to be deleted and recreated.
  4. Note: We recommend this for all projects (“go to Project Settings”), orgs (”go to Organization 
  Settings”), and users (“go to User Settings”).

Would the actor(s) who maliciously gained access to the Deploy SSH Keys also know the GitHub (or any other applicable VCS) org and project name/alias that the key is associated with?

Currently trying to understand the risks of someone accessing private source code, and I imagine knowing the github.com/$ORG_NAME/$REPO_NAME would be key to leveraging the Deploy SSH Keys.

[yannCI]

@squaringthecircle

Is there any way of extracting the legacy AWS_ACCESS_KEY_ID from CircleCI? The UI gives the option to delete the legacy AWS credentials, but you can’t actually see what the ID is. This is important to be able to revoke/rotate the affected AWS access key. Is there an API/UI for finding this information?

I know there is the option of making a config.yml which spits out the value, but that doesn’t scale well for many repos.

@squaringthecircle,
Whether you retrieved them via the UI or the CircleCI API, the value of environment variables is always masked.

Also, by default the secrets-masking feature is enabled for all projects, so the values can’t be printed out in a build job; unless we specifically disable said feature; we can do so on a per-project basis, and upon your explicit request.

[yannCI]

@jscheel

Just for clarity, are you rotating all github tokens, even ones that have been manually rotated since the breach was announced?

@jscheel,
This is likely how you understood it, but I want to make sure this topic really clear.

The GitHub tokens we’ve rotated are the OAuth tokens; the one that are initially automatically created by the CircleCI-GitHub integration when a user signed up to CircleCI via GitHub.

These are not the GitHub personal access tokens you might have created yourself under your GitHub account; we have no access to these.

Unless you stored any GitHub personal access tokens as an environment variable in a CircleCI project or context, you don’t specifically need to rotate them; although it is good practice to do so regularly.

[yannCI]

@rupert-madden-abbott

With Jira integration, each project has a token generated by a CircleCI plugin in Jira and then added to each project’s settings page via the Jira section.

 

Do we need to rotate this secret?

Yes.
 

How do we rotate this secret in the CircleCI plugin in Jira? The secret appears to be hardcoded and I can’t see how to regenerate it.

You need to uninstall the “CircleCI for Jira” app in your Jira instance; this will automatically revoke the authentication in your project’s “Jira Integrations” section.

Next, you need to re-install the “CircleCI for Jira” app; doing so will generate a new token (in Jira) that you can then add to your project’s “Jira Integrations” settings.

[makotoshimazu]

Hi,

Step 6-4 in “Updated instructions as of 17:52 UTC on January 6” says “Note: We recommend this for all projects (“go to Project Settings”), orgs (”go to Organization Settings”), and users (“go to User Settings”).”, but as far as I checked app.circleci.com/settings/organization/github/${MY_ORG}/overview and app.circleci.com/settings/user , no SSH key related settings seems to exist.

Could anyone help me to understand what are the exact steps?

[arvid-m]

Hi, regarding this quote:

OAuth tokens : For GitHub: As of 07:30 UTC on January 7, all GitHub OAuth tokens have been rotated on behalf of CircleCI customers. Customers who wish to do so may rotate their own OAuth tokens by logging out of the CircleCI application, going to https://github.com/settings/applications, selecting “Authorized OAuth Apps”, and then revoking the CircleCI entry. Once that’s done, log back into CircleCI to trigger reauthorization.

I don’t understand the phrasing. The first sentence claims that the token has already been rotated, but the second sentence claims that we should rotate it. Does that mean that we need to revoke and reauthorize the access of circleci in Github or not?

[mathew]

Hi A few clarifications please:

For Bitbucket: As of 10:00 UTC on January 6, 2023 our partners at Atlassian expired all OAuth tokens for Bitbucket users. Bitbucket tokens will refresh for users upon login, and no additional action is needed here. Bitbucket users will still need to replace SSH tokens.

Can you clarify what that last part means please as it is not clear to me? i.e. what ssh tokens and where?

Can you confirm that only projects connected to CCI i.e. via Bitbucket need to be audited? And that other non connected projects on that GitHub or BB account are not at risk?

Can you confirm if there is an audit log available for a bitbucket repository or organisation? Other than going through commits on individual repos, I see no access logs available in Bitbucket, but the suggestion in your FAQ is that there is a way of doing this.

Thanks

[rit1010]

@jchen123

please help me understand - how is it possible that unencrypted environment variables fell into the hands of an attacker?

All posts so far seem to indicate that CircleCi is taking the view that all data is at risk and so should be regenerated. This makes a lot of sense for a system-wide issue as anything else would need the reviewing of a lot of fine detail logs which may or may not exist.

[microbit-matth]

I am also confused about this!

I know User SSH keys exist as an option – see this process here. Less common than deploy keys in my experience. If you used it maybe you’d see something on the user page? Clarification from CircleCI would be great.

User SSH keys are an area where it would be great to be able to see in CircleCI UI or API which users had used the feature. Users need to take individual action to revoke, which is hard for organisations to manage. For many organisations this could confirm no-one had used the feature.

I don’t understand what would be on the organisation page. There is no notion of an organisation SSH key in GitHub so far as I’m aware. Clarification from CircleCI here would be really useful.

[emilycook]

So as of Friday, we completed this action so the customer no longer needs to do it. What we were trying to convey is if they already had rotated user OAuth tokens or wish to do so again, there is no detriment. Sorry for any confusion!

[yannCI]

@makotoshimazu

Hi,

Step 6-4 in “Updated instructions as of 17:52 UTC on January 6” says “Note: We recommend this for all projects (“go to Project Settings”), orgs (”go to Organization Settings”), and users (“go to User Settings”).”, but as far as I checked app.circleci.com/settings/organization/github/${MY_ORG}/overview and app.circleci.com/settings/user , no SSH key related settings seems to exist.

Could anyone help me to understand what are the exact steps?

HI @makotoshimazu,

Thank you for pointing this out, and please accept our apologies for the confusion.

There are indeed no SSH key related information under “User Settings” or “Organization Settings”.

We are in the process of updating the “Project SSH keys” section of the blog post to bring more clarity about the required actions.

[yannCI]

@microbit-matth

I am also confused about this!

 
Hello @microbit-matth,

I’m sincerely sorry this paragraph caused some confusion.

I know User SSH keys exist as an option – see this process here . Less common than deploy keys in my experience. If you used it maybe you’d see something on the user page? Clarification from CircleCI would be great.

I confirm that SSH keys are only present in “Project Settings”.

There are 3 types of SSH keys you can find in the “SSH Keys” section of a given “Project Settings”:

• Deploy Key. There can be either one or none per project.
• User Key. There can be either one or none per project.
• Additional SSH Key(s). There can be none, only one, or several in a given project.

There are no SSH key related settings/information in the “User Settings”.

 

User SSH keys are an area where it would be great to be able to see in CircleCI UI or API which users had used the feature. Users need to take individual action to revoke, which is hard for organisations to manage. For many organisations this could confirm no-one had used the feature.

User SSH keys added to “Project Settings > SSH Keys” can easily be linked to the VCS user they belong to. Those keys are always named according to this format: <vcs_username>/<project_name> user key.

The above is actually incorrect. The format for the SSH keys names is:

• <org_name>/<project_name> deploy key for deploy keys
• <org_name>/<project_name> user key for user keys

(Thank you @microbit-matth for pointing this out )

 

I don’t understand what would be on the organisation page. There is no notion of an organisation SSH key in GitHub so far as I’m aware. Clarification from CircleCI here would be really useful.

This is correct. There are no SSH key settings of any sort in the “Organization Settings”.

 
We are currently updating the “Project SSH keys ” section of the blog post to bring more clarity about the required actions.

[jerdog]

@mathew

A few clarifications please:

For Bitbucket: As of 10:00 UTC on January 6, 2023 our partners at Atlassian expired all OAuth tokens for Bitbucket users. Bitbucket tokens will refresh for users upon login, and no additional action is needed here. Bitbucket users will still need to replace SSH tokens.

Can you clarify what that last part means please as it is not clear to me? i.e. what ssh tokens and where?

Can you confirm that only projects connected to CCI i.e. via Bitbucket need to be audited? And that other non connected projects on that GitHub or BB account are not at risk?

Can you confirm if there is an audit log available for a bitbucket repository or organisation? Other than going through commits on individual repos, I see no access logs available in Bitbucket, but the suggestion in your FAQ is that there is a way of doing this.

Any repository stored with BitBucket for a project on CircleCI will need to have their SSH keys replaced, and those repositories audited. In BitBucket, you can find the user-level audit log under >https://bitbucket.org/ account/settings/auditlog/, and for the workspace/org level, you can find the audit log under > https://bitbucket.org/ <org_name>/workspace/settings/auditlog.

(ty @yannCI for the info!)

[rit1010]

As I have a free plan-based service can I ask for ticket #125038 to be upgraded from the general 3-5 business day service level as I have manually deleted the connection at the Bitbucket level.

While you have been working with Atlassian to make the process far simpler I’m starting from a blank slate as our environment has been a mix of R&D and production configurations for far too long.

[microbit-matth]

Thanks for the response, really useful to know it’s all about the project keys and user keys are shown there. I was able to use the project API to audit these by checking for type-=github-user-key and check in the UI for the key names.

I see the key names in the format “{my org}/{my project} user key” so doesn’t seem to directly link to the user for org projects.

In my case I can identify the user based on the project history but for users with more widespread use this might be more of a challenge.

[s22chan]

How are we supposed to remove the secrets in an archived repo? In the old circleci, I can goto the project page of a repo that is “not setup”, but now it pops up a wizard that requires a valid config.yml reference.

[yannCI]

@microbit-matth

Thanks for the response, really useful to know it’s all about the project keys and user keys are shown there. I was able to use the project API to audit these by checking for type-=github-user-key and check in the UI for the key names.

I see the key names in the format “{my org}/{my project} user key” so doesn’t seem to directly link to the user for org projects.

In my case I can identify the user based on the project history but for users with more widespread use this might be more of a challenge.

You’re absolutely right, @microbit-matth!

The format of the SSH keys names is: <org_name>/<project_name> user key.

It just happens to be equivalent to <vcs_username>/<project_name> user key when the organization in CircleCI corresponds to a:

• GitHub personal account

or

• Bitbucket personal/default workspace (if said workspace’s name and the Bitbucket username match, which by the way, they should for the related Bitbucket organization to be supported in CircleCI)

 
But the rule is indeed: <org_name>/<project_name> user key.

Thank you for pointing this out.

[makotoshimazu]

I see, thanks for checking that!