Thanks for that.
By this you mean the blog post steps (was 6 now 7) Project Settings > SSH Keys … am I correct, please?
Also from the most recent update
Note: SSH keys will also need to be rotated from the target environment.
Can you give an example of what is meant by this please to avoid confusion?
Thanks
Yes, your SSH keys on CircleCI as well as on whichever environment connected to CircleCI, to be safe.
Apologies for the delay. If your project has been archived and no longer on CircleCI, then your secrets are already removed. I verified on one of my own, personal projects that was archived.
What about renames on Bitbucket or GitHub? We’ve seen that we can retrieve old environment-variables if we rename the repository back to what it was renamed from. We also see that if a new project takes the same name as an old project on Bitbucket/GitHub, the build history for the previous project is there. We therefore create a support request to CircleCI to clean up stuff on their side whenever we delete/archive our projects for this exact reason.
To be 100% clear you mean SSH keys at a connected repository level inside the target env and not a personal account level?
So if my repository vcs is Bitbucket, i would need to rotate repo level keys and tokens, but there is not a need to rotate personal Bitbucket SSH keys for example.
Hello.
Based on our testing, github repos formerly setup in circle, which are then deactivated (“stop building”), but then re-setup later retain their environment variable values. For any org that has been using circle for a long time, there are likely many compromised secrets contained “invisibly” in deactivated projects. In the case of shared or long-lived secrets, its quite possible that those secrets are still usable.
I think CircleCI needs to spell this out explicitly as I don’t think it’s been clear in the public statements to date.
Would it be possible for Support to produce a list of formerly-active projects within an account? (From the UI, it seems projects never setup or projects formerly setup are indistinguishable.)
Thanks for your efforts on this.
Came here for this, thank you for posting. We have noticed the same and none of the tools we’ve found so far are looking for these.
In these cases you need to check for old env vars by crafting a url by hand:
https://app.circleci.com/settings/project/github/_github-org_/_github-repo_/environment-variables
Also:
Perhaps Circle could comment on whether these settings that were kept (when they should have been discarded) were also compromised.
Cheers.
Sorry for the delay, yes please rotate any SSH keys at the repository level associated with your vcs (Bitbucket in this case) and in CircleCI as described in the blog.
Hey @erikahansen, I’m not quite clear on your question. Do you mean something along the lines of “Do renaming repositories on Bitbucket or GitHub clear keys and variables”?
We would recommend removing any existing variables, contexts and ssh keys that have been associated with CircleCI, past or present, out of caution.
Additionally, please let me know if you need help creating a support ticket for any additional questions or requests that we might be able to help with!
Can we get some clarity on whether an attacker was able to access actual machines running builds? So far all questions about whether source code was leaked were answered with:
We recommend viewing the security audit logs of your VCS for any unauthorized access.
If actual build agents/runners were (potentially) compromised that means source code could have been leaked without it showing up in out Github audit logs. Additionally, any builds created while the attacker had access might have had malicious code injected into them.
We have hundreds of renamed/archived/deleted repositories, and the old CircleCI project for an equivalent Bitbucket/GitHub project doesn’t appear unless we create a repository on Bitbucket/GitHub with the old name as it was before it was deleted. We need a list of all CircleCI projects to make sure we have control of all secrets, including old projects that is retained in CircleCI’s systems but isn’t shown in the web GUI.
We have project that was depreciated couple of years back and pipeline also was deleted then. Unfortunately the employees working on this project left the company already. Do we still need to rotate the passwords?
We can’t provide any details regarding the incident right now, but we expect to provide an incident report on Tuesday, Jan 17th (PST)
Hi @rahulgkrishna, we recommend following the steps outlined in the blog to rotate or revoke env vars, ssh keys and other secrets from projects in CircleCI and target systems (like project repositories in Github and Bitbucket, for example) out of caution.
As of today, apparently all of our SSH Deploy Keys are still enabled on both CircleCI and Github.
I see on the Rotating Secrets for January 4th Incident article that you have updated the Contexts API to include the last "updated_at" date and time stamp. This is very useful to make sure the secret rotation was completed, will you do the same for Project Environment Variables and SSH Deploy keys too?
Hey @glenjamin, sorry for the delay! Based on the latest instructions from the blog post (step 7) all users should rotate all SSH keys stored in CircleCI and target environments (like github, bitbucket).
The only action we’ve taken on behalf of users is rotating oauth tokens and revoking personal and project API tokens.
Hey @anon, thanks for the feedback here. I can share this with the folks who work on the blog.
If you haven’t already, would you mind opening up a support ticket? Our team might be able to help you out in regards to getting a list of projects.
Opening a support ticket would be your best bet in potentially getting a list of all CircleCI projects.
For any renamed or archived repositories on target environments (say github or bitbucket), please either rotate or revoke env vars, ssh keys and other secrets out of caution. If you’ve deleted a repository on target environments, any secrets set in that target environment will be deleted as well.
Sorry, but that statement doesn’t help anyone. There are two possibilities here:
Based on the vague statements published so far we’ve already assumed that builds created during the incident can’t be trusted, but it would be good to be more upfront about that.