[CircleCI Security Alert] Rotate any secrets stored in CircleCI

will this discussion serve as a way to share any indicators of compromise that we can use to search through our logs or will some other form of communication be shared?

2 Likes

I am concerned if “Deploy keys” shall also be rotated
@z00b can you help us with that? Will you share more IOCs?

2 Likes

Two quick questions:

  • Do we know whether CircleCI’s platform was used for any access to customers’ systems or whether secrets were just grabbed and run off with?
  • How regularly do the IPs on all.knownips.circleci.com change? And if they’ve changed recently, please can we have a list of all the previous IPs?
2 Likes

Thanks. Is there a script that could be used to identify all secrets within a circleci org? We have hundreds of repos and a variety of platform teams working in CircleCI, as a security team it’s difficult to ensure we have rotated everything.

2 Likes

I used the github CLI to get a list of all github repos

gh api -H "Accept: application/vnd.github+json" '/orgs/ORG/repos?per_page=100&page=1' | jq 'map(.full_name) | .[]' -r > ./repos.txt
gh api -H "Accept: application/vnd.github+json" '/orgs/ORG/repos?per_page=100&page=2' | jq 'map(.full_name) | .[]' -r >> ./repos.txt
gh api -H "Accept: application/vnd.github+json" '/orgs/ORG/repos?per_page=100&page=3' | jq 'map(.full_name) | .[]' -r >> ./repos.txt
gh api -H "Accept: application/vnd.github+json" '/orgs/ORG/repos?per_page=100&page=4' | jq 'map(.full_name) | .[]' -r >> ./repos.txt
gh api -H "Accept: application/vnd.github+json" '/orgs/ORG/repos?per_page=100&page=5' | jq 'map(.full_name) | .[]' -r >> ./repos.txt

And then this script to list variables attached to a repo

curl -su $(sed -nr 's/token: (.*)/\1/p' ~/.circleci/cli.yml): \
  https://circleci.com/api/v2/project/gh/${1}/envvar | \
  jq '.items | map(.name) | .[]' -r | xargs -n1 echo ${1}

And then xargs to combine the two

cat ./repos.txt | xargs -n1 sh ./get-repo-vars.sh

For contexts I just used the CircleCI CLI.

To be able to see everything it works best when you’re a github admin.

2 Likes

I wrote a script. If it’s helpful then I’m happy.

1 Like

Is CircleCI able to disable the once in 30 day limit for audit logs? Our org did one 3 weeks ago and are now unable to verify anything. I imagine I’m not the only one in this situation. Thanks

1 Like

Were repositories also exposed. We deploy using an ansible playbook repository. Secretes are stored encrypted in the playbook but we un-encrypt at deploy time. Do we need to assume that all secrets that are stored in the playbook have also been exposed?

Thank you, this looks promising!

1 Like

So, are we safe rotating all the environment variables? How do we know that this is not still open/vulnerable? To be honest, the information provided about the issue is really poor.

We need to rotate the personal tokens too?

Thank you, this was very helpful.

I used it to extend to Legacy AWS Configuration too, since those keys are still dangerous but not exposed in the UI or new API (which is a very silly thing, in my opinion):

const getProjectSettings = async (
  projectName: string,
  pageToken?: string
): Promise<any[]> => {
  const headers = {
    authorization: `Basic ${Buffer.from(API_TOKEN).toString("base64")}`,
  };
  const response = await fetch(
    `https://circleci.com/api/v1.1/project/${CVS}/${OWNER_NAME}/${projectName}/settings${pageToken ? `?page-token=${pageToken}` : ""
    }`,
    { headers }
  );
  const data = await response.json();
  return data.aws;
};

Update: submitted this as a PR against xhiroga’s repo, in case anyone wants it working. Thank you again.

1 Like

With Jira integration, each project has a token generated by a CircleCI plugin in Jira and then added to each project’s settings page via the Jira section.

  1. Do we need to rotate this secret?
  2. How do we rotate this secret in the CircleCI plugin in Jira? The secret appears to be hardcoded and I can’t see how to regenerate it.

@z00b ,
Would it be possible to confirm if “Deploy Keys” are on the list of things that need to be rotated
I know a few others have asked but it would be great to get a clear list of elements that need to be rotated as part of this advisory (With where to find them on projects and if possible scripts to enumerate where they all exist on an org wide basis. Also if so update to the original advisory to make it completely clear as to what needs rotating. Given I don’t think it was made clear that “secrets” included SSH keys as part of the original advisory. I feel adding that adding this to the messaging would be a great help to people just picking up on this / over the next few days :raised_hands:)
Thanks for any help on this front!

The lack of engagement by CircleCI staff on this thread is concerning, to say the least

5 Likes

For those working with circleCI and GitHub, here is a script we did today

QuickStart

python main.py list-ctx
python main.py list-env
python main.py list-keys
python main.py replace-keys

with the following main.py file

import requests
import argparse
from config import API_TOKEN, GH_TOKEN, ORG # CircleCi api Token, Github api token, Github organisation name

ORG_SLUG = f"github/{ORG}"

auth_header = { 'Circle-Token': API_TOKEN }

github_headers = {
    "X-GitHub-Api-Version": "2022-11-28",
    "Accept": "application/vnd.github+json",
    "Authorization": f"Bearer {GH_TOKEN}"
}

def list_contexts():
    url = "https://circleci.com/api/v2/context"
    req_params = {
        "owner-slug": ORG_SLUG,
        "owner-type": "organization",
    }

    page_token = ""

    contexts = []

    while True:
        params = {**req_params, "page-token": page_token}
        response = requests.get(url, params=params, headers=auth_header).json()
        contexts += response.get("items", [])
        page_token = response.get("next_page_token", None)

        if page_token is None:
            return contexts

def list_context_variables(context_id):
    url = f"https://circleci.com/api/v2/context/{context_id}/environment-variable"
    
    req_params = {}

    page_token = ""

    env_vars = []

    while True:
        params = {**req_params, "page-token": page_token}
        response = requests.get(url, params=params, headers=auth_header).json()
        env_vars += response.get("items", [])
        page_token = response.get("next_page_token", None)

        if page_token is None:
            return env_vars

def list_github_repositories():
    url = f"https://api.github.com/orgs/{ORG}/repos"

    req_params = {
        "per_page": 100
    }

    page_idx = 1

    repos = []

    while True:
        params = {**req_params, "page": page_idx}
        response = requests.get(url, params=params, headers=github_headers).json()
        repos += response
        if len(response) == 0:
            return repos
        page_idx += 1

def list_project_env_variables(project):
    url = f"https://circleci.com/api/v2/project/gh/{ORG}/{project}/envvar"
    
    req_params = {}

    page_token = ""

    env_vars = []

    while True:
        params = {**req_params, "page-token": page_token}
        response = requests.get(url, params=params, headers=auth_header).json()
        env_vars += response.get("items", [])
        page_token = response.get("next_page_token", None)

        if page_token is None:
            return env_vars

def list_project_keys(project):
    url = f"https://circleci.com/api/v2/project/gh/{ORG}/{project}/checkout-key"
    
    req_params = {}

    page_token = ""

    keys = []

    while True:
        params = {**req_params, "page-token": page_token}
        response = requests.get(url, params=params, headers=auth_header).json()
        keys += response.get("items", [])
        page_token = response.get("next_page_token", None)

        if page_token is None:
            return keys 

def list_github_deploy_keys(project):
    url = f"https://api.github.com/repos/{ORG}/{project}/keys"

    req_params = {
        "per_page": 100
    }

    page_idx = 1

    keys = []

    while True:
        params = {**req_params, "page": page_idx}
        response = requests.get(url, params=params, headers=github_headers).json()
        keys += response
        if len(response) == 0:
            return keys
        page_idx += 1

def delete_github_deploy_key(project, key):
    url = f"https://api.github.com/repos/{ORG}/{project}/keys/{key['id']}"

    response = requests.delete(url, headers=github_headers)

    if response.status_code != 204:
        print("Could not delete github key", key['title'], response.json())


def get_project_checkout_keys(project, fingerprint):
    url = f"https://circleci.com/api/v2/project/gh/{ORG}/{project}/checkout-key/{fingerprint}"

    response = requests.get(url, headers=auth_header).json()

    return response

def delete_project_checkout_keys(project, fingerprint):
    url = f"https://circleci.com/api/v2/project/gh/{ORG}/{project}/checkout-key/{fingerprint}"

    response = requests.delete(url, headers=auth_header)

    if response.status_code != 200:
        print("Could not delete circleci key for project", project) 

def create_project_checkout_keys(project, key_type):
    url = f"https://circleci.com/api/v2/project/gh/{ORG}/{project}/checkout-key"

    response = requests.post(url, headers=auth_header, json={"type": key_type})

    if response.status_code != 201:
        print("Could not create circleci key for project", project, response.json()) 

def main_list_project_env_var():
    repos = list_github_repositories()
    print("project;var")
    for repo in repos:
        env_vars = list_project_env_variables(repo["name"])
        for var in env_vars:
            print(f'{repo["name"]};{var["name"]}')

def main_list_project_checkout_keys():
    repos = list_github_repositories()
    print("project;type;fingerprint;created_at")
    for repo in repos:
        keys = list_project_keys(repo["name"])
        for key in keys:
            print(f'{repo["name"]};{key["type"]};{key["fingerprint"]};{key["created_at"]}')


def main_list_contexts():
    contexts = list_contexts()

    print("context;var;last_modified")
    for ctx in contexts:
        env_vars = list_context_variables(ctx["id"])
        for var in env_vars:
            print(f'{ctx["name"]};{var["variable"]};{var["created_at"]}')

def main_replace_project_checkout_keys():
    repos = list_github_repositories()
    for repo in repos:
        project = repo["name"]
        gh_keys = list_github_deploy_keys(project)

        for key in gh_keys:
            delete_github_deploy_key(project, key)

        ci_keys = list_project_keys(project)
        for key in ci_keys:
            delete_project_checkout_keys(project, key["fingerprint"])
            create_project_checkout_keys(project, key_type=key['type'])
            

command_dict = {
    "list-ctx": main_list_contexts,
    "list-env": main_list_project_env_var,
    "list-keys": main_list_project_checkout_keys,
    "replace-keys": main_replace_project_checkout_keys
}

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("cmd", choices=list(command_dict.keys()))
    args = parser.parse_args()
    
    command_dict[args.cmd]()
2 Likes

Hello CircleCI!

Do you have any script already done, or a docker image, that lists all the secrets that could have been compromised?

We have lots of repositories, so we need a tool to dump everything we have configured, rotate credentials and then run again to be sure everything was rotated.

I see some scripts but not sure if they are listing everything that could have been compromised.

Thanks!

For the various teams at CircleCI, working day & night on this - Thank you!
It was not an easy decision to go public without having all the details, even though this now means additional pressure to work the case faster # YouDidTheRightThing.

3 Likes

When does CircleCI plan to notify customers by email?

1 Like

Will CircleCI be updating the official blog post with additional guidance from questions/answers here? Or is this the place to look for the most accurate and updated information?

3 Likes