[CircleCI Security Alert] Rotate any secrets stored in CircleCI

Hi @trevormarshall ,

When you hit the “Stop Building” in the CircleCI project, or archive/delete a repository in the VCS, the project build data and projects settings are indeed still present.

Though the build data:

  • build history (pipelines you’ve run and their output)
  • caches, workspaces, and artifacts

will age out automatically, based on your Organization plan and settings (See Data Retention Policy), the project settings will indeed remain, unless you manually remove them or send us an explicit request for the whole project to be deleted.

As long as the project still exist in CircleCI, you can access the “Project Settings” either by:

  • Constructing the URL as you suggested (https://app.circleci.com/settings/project/<vcs>/<org_name>/<project_name>)

or

  • Using the CircleCI v1 and v2 APIs
    • For SSH checkout keys > https://circleci.com/api/v2/project//<vcs>/<org_name>/<project_name>/checkout-key

    • For environment variables > https://circleci.com/api/v2/project/<vcs>/<org_name>/<project_name>/envvar

    • For the Jira integration and other legacy integrations > https://circleci.com/api/v1.1/project/<vcs>/<org_name>/<project_name>/settings

Can anyone please give an update on what actually went down and why you’re now confident things are ok? Can we really allow Circle CI access to our critical systems, credentials and code again?

I’m sorry to say but the following phrase: We have confidence in the security of the CircleCI platform, and customers can continue to build. doesn’t cut it for our team. Confidence in a security posture is one thing, but you got pwned, we have all been pwned because of it. So what exactly has changed to mitigate the issue? I really think release this detail will help restore customer confidence.

We still feel it’s better we at least lock CircleCI out of our systems, until we receive greater transparency on what’s gone down.

I’m not ignorant to the fact you can’t always give our all details for your own security reasons, but I’m sorry to say “trust us” and “have confidence” isn’t really working for many people at this point and I think enough time has gone by to disclose more information, especially if as you say, things are under control.

Please release more details sooner rather than later.

2 Likes

Hi - I am unable to see my organisation after I sign in after my token has been revoked.

Ticket number #125492

We have received an email from AWS (“CircleCI Security Alert to Rotate Access Keys”) listing a single Access Key, out of many we had exposed on CircleCI. Why is that? Shouldn’t all exposed keys be included in this advisory?

Today we published incident report for our security incident disclosed on January 4 on our blog here:

Hi @pallanmercari, our incident report just went live on our blog. You can find it here: CircleCI incident report for January 4, 2023 security incident

Hey @mavink78, you can find more details on the incident in our report that just went live on our blog, here: CircleCI incident report for January 4, 2023 security incident

Hey @dserodio, our incident report just went live on our blog, please check it out here! CircleCI incident report for January 4, 2023 security incident

As described in the incident report and our previous blog post, we worked with AWS to notify customers whose tokens may have been impacted, and they were sent an email with the same subject line you’ve described.

We can’t speak for AWS for what they consider to be potentially impacted on their systems, so out of caution we recommend folks to review our security best practices listed in the incident report, as well as our previous recommendations to rotate secrets on CircleCI and target systems.

1 Like

I have a question about the response to the incident announced by Circle ci, and I would appreciate it if someone with knowledge could help me.

In the release it says "we have added additional step-up authentication steps and controls.”

I was wondering how Circle CI detect when a session that has been authenticated by SSO has been stolen, and what is the basis for detecting when additional authentication is required.

What are the specific measures?

I see there is no mention of what happens to users whose signing certificates for app stores were exposed. I have built a mobile app with CircleCi and signing keys were stored in environment variables as I have presumed(naively) it’s secure place to store them.
Now here is the pickle - signing certificates for apps as far as I understand are locked to a specific app. It’s not possible to change certificates without asking ALL the users to download a new app. Any suggestions on what to do in that case?

From the blog post:

On January 4, 2023, at 6:30 PM PST / January 5, 2023, at 02:30 UTC, we sent disclosure emails

Both emails I did not receive (not in spam either). The first email I received about this issue was the one from Saturday 14 January. Hopefully you guys can look into why some users didn’t receive those emails.

Is there something I can do on my end to ensure I’ll receive them in the future?

1 Like

We will not be disclosing how we detect. There are plenty of best practices out there to help guide you though.

We’re not able to provide a good answer as to the signing certificates for your apps. This would depend on the development of your application and the processes you used. The likely mitigation would entail deploying an updated version with a new key, but the particulars on this would depend on your signing process for each app distribution (Apple, Google, etc.) which provide some ways to mitigate this.

We identified an issue with some accounts in which kept them from getting the initial emails, which is also why you received the Incident Report. This has been resolved, and you will not need to do anything else. If you have any questions, or experiencing any issues, please create a new topic.

1 Like