[CircleCI Security Alert] Rotate any secrets stored in CircleCI

We need to rotate the personal tokens too?

Thank you, this was very helpful.

I used it to extend to Legacy AWS Configuration too, since those keys are still dangerous but not exposed in the UI or new API (which is a very silly thing, in my opinion):

const getProjectSettings = async (
  projectName: string,
  pageToken?: string
): Promise<any[]> => {
  const headers = {
    authorization: `Basic ${Buffer.from(API_TOKEN).toString("base64")}`,
  };
  const response = await fetch(
    `https://circleci.com/api/v1.1/project/${CVS}/${OWNER_NAME}/${projectName}/settings${pageToken ? `?page-token=${pageToken}` : ""
    }`,
    { headers }
  );
  const data = await response.json();
  return data.aws;
};

Update: submitted this as a PR against xhiroga’s repo, in case anyone wants it working. Thank you again.

1 Like

With Jira integration, each project has a token generated by a CircleCI plugin in Jira and then added to each project’s settings page via the Jira section.

  1. Do we need to rotate this secret?
  2. How do we rotate this secret in the CircleCI plugin in Jira? The secret appears to be hardcoded and I can’t see how to regenerate it.

@z00b ,
Would it be possible to confirm if “Deploy Keys” are on the list of things that need to be rotated
I know a few others have asked but it would be great to get a clear list of elements that need to be rotated as part of this advisory (With where to find them on projects and if possible scripts to enumerate where they all exist on an org wide basis. Also if so update to the original advisory to make it completely clear as to what needs rotating. Given I don’t think it was made clear that “secrets” included SSH keys as part of the original advisory. I feel adding that adding this to the messaging would be a great help to people just picking up on this / over the next few days :raised_hands:)
Thanks for any help on this front!

The lack of engagement by CircleCI staff on this thread is concerning, to say the least

5 Likes

For those working with circleCI and GitHub, here is a script we did today

QuickStart

python main.py list-ctx
python main.py list-env
python main.py list-keys
python main.py replace-keys

with the following main.py file

import requests
import argparse
from config import API_TOKEN, GH_TOKEN, ORG # CircleCi api Token, Github api token, Github organisation name

ORG_SLUG = f"github/{ORG}"

auth_header = { 'Circle-Token': API_TOKEN }

github_headers = {
    "X-GitHub-Api-Version": "2022-11-28",
    "Accept": "application/vnd.github+json",
    "Authorization": f"Bearer {GH_TOKEN}"
}

def list_contexts():
    url = "https://circleci.com/api/v2/context"
    req_params = {
        "owner-slug": ORG_SLUG,
        "owner-type": "organization",
    }

    page_token = ""

    contexts = []

    while True:
        params = {**req_params, "page-token": page_token}
        response = requests.get(url, params=params, headers=auth_header).json()
        contexts += response.get("items", [])
        page_token = response.get("next_page_token", None)

        if page_token is None:
            return contexts

def list_context_variables(context_id):
    url = f"https://circleci.com/api/v2/context/{context_id}/environment-variable"
    
    req_params = {}

    page_token = ""

    env_vars = []

    while True:
        params = {**req_params, "page-token": page_token}
        response = requests.get(url, params=params, headers=auth_header).json()
        env_vars += response.get("items", [])
        page_token = response.get("next_page_token", None)

        if page_token is None:
            return env_vars

def list_github_repositories():
    url = f"https://api.github.com/orgs/{ORG}/repos"

    req_params = {
        "per_page": 100
    }

    page_idx = 1

    repos = []

    while True:
        params = {**req_params, "page": page_idx}
        response = requests.get(url, params=params, headers=github_headers).json()
        repos += response
        if len(response) == 0:
            return repos
        page_idx += 1

def list_project_env_variables(project):
    url = f"https://circleci.com/api/v2/project/gh/{ORG}/{project}/envvar"
    
    req_params = {}

    page_token = ""

    env_vars = []

    while True:
        params = {**req_params, "page-token": page_token}
        response = requests.get(url, params=params, headers=auth_header).json()
        env_vars += response.get("items", [])
        page_token = response.get("next_page_token", None)

        if page_token is None:
            return env_vars

def list_project_keys(project):
    url = f"https://circleci.com/api/v2/project/gh/{ORG}/{project}/checkout-key"
    
    req_params = {}

    page_token = ""

    keys = []

    while True:
        params = {**req_params, "page-token": page_token}
        response = requests.get(url, params=params, headers=auth_header).json()
        keys += response.get("items", [])
        page_token = response.get("next_page_token", None)

        if page_token is None:
            return keys 

def list_github_deploy_keys(project):
    url = f"https://api.github.com/repos/{ORG}/{project}/keys"

    req_params = {
        "per_page": 100
    }

    page_idx = 1

    keys = []

    while True:
        params = {**req_params, "page": page_idx}
        response = requests.get(url, params=params, headers=github_headers).json()
        keys += response
        if len(response) == 0:
            return keys
        page_idx += 1

def delete_github_deploy_key(project, key):
    url = f"https://api.github.com/repos/{ORG}/{project}/keys/{key['id']}"

    response = requests.delete(url, headers=github_headers)

    if response.status_code != 204:
        print("Could not delete github key", key['title'], response.json())


def get_project_checkout_keys(project, fingerprint):
    url = f"https://circleci.com/api/v2/project/gh/{ORG}/{project}/checkout-key/{fingerprint}"

    response = requests.get(url, headers=auth_header).json()

    return response

def delete_project_checkout_keys(project, fingerprint):
    url = f"https://circleci.com/api/v2/project/gh/{ORG}/{project}/checkout-key/{fingerprint}"

    response = requests.delete(url, headers=auth_header)

    if response.status_code != 200:
        print("Could not delete circleci key for project", project) 

def create_project_checkout_keys(project, key_type):
    url = f"https://circleci.com/api/v2/project/gh/{ORG}/{project}/checkout-key"

    response = requests.post(url, headers=auth_header, json={"type": key_type})

    if response.status_code != 201:
        print("Could not create circleci key for project", project, response.json()) 

def main_list_project_env_var():
    repos = list_github_repositories()
    print("project;var")
    for repo in repos:
        env_vars = list_project_env_variables(repo["name"])
        for var in env_vars:
            print(f'{repo["name"]};{var["name"]}')

def main_list_project_checkout_keys():
    repos = list_github_repositories()
    print("project;type;fingerprint;created_at")
    for repo in repos:
        keys = list_project_keys(repo["name"])
        for key in keys:
            print(f'{repo["name"]};{key["type"]};{key["fingerprint"]};{key["created_at"]}')


def main_list_contexts():
    contexts = list_contexts()

    print("context;var;last_modified")
    for ctx in contexts:
        env_vars = list_context_variables(ctx["id"])
        for var in env_vars:
            print(f'{ctx["name"]};{var["variable"]};{var["created_at"]}')

def main_replace_project_checkout_keys():
    repos = list_github_repositories()
    for repo in repos:
        project = repo["name"]
        gh_keys = list_github_deploy_keys(project)

        for key in gh_keys:
            delete_github_deploy_key(project, key)

        ci_keys = list_project_keys(project)
        for key in ci_keys:
            delete_project_checkout_keys(project, key["fingerprint"])
            create_project_checkout_keys(project, key_type=key['type'])
            

command_dict = {
    "list-ctx": main_list_contexts,
    "list-env": main_list_project_env_var,
    "list-keys": main_list_project_checkout_keys,
    "replace-keys": main_replace_project_checkout_keys
}

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("cmd", choices=list(command_dict.keys()))
    args = parser.parse_args()
    
    command_dict[args.cmd]()
2 Likes

Hello CircleCI!

Do you have any script already done, or a docker image, that lists all the secrets that could have been compromised?

We have lots of repositories, so we need a tool to dump everything we have configured, rotate credentials and then run again to be sure everything was rotated.

I see some scripts but not sure if they are listing everything that could have been compromised.

Thanks!

For the various teams at CircleCI, working day & night on this - Thank you!
It was not an easy decision to go public without having all the details, even though this now means additional pressure to work the case faster # YouDidTheRightThing.

3 Likes

When does CircleCI plan to notify customers by email?

1 Like

Will CircleCI be updating the official blog post with additional guidance from questions/answers here? Or is this the place to look for the most accurate and updated information?

3 Likes

Is there a particular thing we should be looking for in the circle org audit log? I see a fair number of ‘unregistered’ values in the ACTOR_TYPE column but don’t know what that indicates.

They did at around 3am GMT last night!

Hey folks, thanks for sticking with us here. The team is understandably busy fielding a higher than usual level of questions across multiple channels.

We’ve just posted the following support article that aims to provide a comprehensive list of what we advise to rotate out of an abundance of caution.

2 Likes

First off. I appreciate all the efforts CircleCI’s full team is taking to keep us as up to date as possible. I empathize with all of you during these times.

Question:

Can we expect guidance on IOC’s / things to monitor related to the events that led to the recommendation for customers to rotate their keys?

I am looking to empower our SOC w/ as much information as possible. Appreciate a prompt response.

2 Likes

Yes, please rotate deploy and user keys.

Deploy keys should also be rotated. This support doc should help with understanding what has been impacted: https://support.circleci.com/hc/en-us/articles/11816211460891

We will be sending updates as we’re able via email and our blog.

So deploy keys were possibly compromised? So we would have to assume that our repositories could have been accessed?

Yes, please also rotate deploy keys. Here is a support article listing what was affected and how to rotate them: https://support.circleci.com/hc/en-us/articles/11816211460891

Does this incident impact self-hosted CircleCI environments in addition to the SaaS version?

1 Like