I did some investigation on the failed assume role. It seems that circleci changed the private key for signing the OIDC JWT token but didn’t update https://oidc.circleci.com/org/<project_id>/.well-known/jwks-pub.json
Tried throwing the OIDC JWT token to https://jwt.io/ and validate it. They are saying “signature verification failed” in console and invalid signature in the UI.
OIDC issues should resolve shortly. No action is required by users.
Have CircleCI personal access tokens been automatically revoked/deleted? Multiple people have reported that theirs have just stopped working, and they have nothing listed @ https://app.circleci.com/settings/user/tokens.
This doesn’t seem to apply to the tokens for service accounts we rotated yesterday. So perhaps only tokens that haven’t been explicitly rotated since the breach?
I see the updated blog post suggests that we rotate those tokens, but it doesn’t say they’ll be deleted automatically.
Folks, as per @philglass’ observation, in the last couple of hours we’ve taken the step of removing all personal and project API tokens created before we released the security alert. This means you can focus on other efforts as part of your own remediation.
We apologise for the impact this might have in having to recreate tokens, but took this action in the belief that it is in the best interests of our users, minimising risk whilst also reducing the burden on our users having to respond.
We will update the blog in due course.
Hello CircleCi,
Firstly, thank you for providing the access to audit logs.
Would you please guide the us with some Indicators of Compromise that we have look in the audit logs for potential unauthorized access to our resources?
Thank you!
They have been saying all along - there is nothing to look for in the CircleCI audit logs. We need to check all the services/systems that had secrets in CircleCI for “suspicious activity” associated to those creds/tokens. Yes, I agree, that is not easy or specific enough.
It is very costly and hard to verify to have individual users revoke their OAuth tokens.
CircleCI should be able to do this by changing client secret or otherwise addressing this with GitHub directly.
Is there a reason CircleCI hasn’t clicked on “Revoke all user tokens” in their GitHub OAuth app to do this for everyone?
It’s actually very difficult to do this properly on the customer / client side because we have to somehow convince every user of our org who might have created an OAuth token for CircleCI to go and do this - 100s of users potentially - some of whom might be on leave etc.
Any update on this?
Has CircleCI just revoked all deploy SSH keys by any chance?
Should be resolved, are you still seeing the problem?
I am not aware of deploy key revocation on our side at this time, only the personal/project API keys as previously posted
Ah yep sorry - false alarm. Thanks for the quick reply 
In case it helps anybody else, I’ve produced this tool for listing my CircleCi secrets: GitHub - rupert-madden-abbott/circleci-audit
Looks like CircleCI have their own tool now as well but it’s not clear to me what it covers.
My tool handles:
We have made audit logs available to all customers, with these specific changes
This can be found in the “Security” tab of the “Org Settings” page.
For more information on self-serve audit logs please see our [documentation]Audit logs - CircleCI).
Has CircleCI taken action to remove user SSH checkout keys?
Earlier today I made a report using the API, and now those keys are not showing up.
This is problematic, as I need to assume those keys are compromised, but I only have the key fingerprints from earlier today - which isn’t enough to track them down and ensure they’re revoked at the GitHub end (the API response doesn’t note which user the keys belonged to).
Anyone who didn’t already list these keys earlier today won’t be able to get a list at all.
Joining in on the above questions about SSH keys.
We rotated all of our deploy keys yesterday (deleted the key on the Github side, deleted the key on the CircleCI side, clicked Add Deploy Key again).
We have just found that the new keys have been deleted on the Github side for every project (they’re still present in each project on the Circle side).
It does seem best in this case to revoke user tokens on the CircleCI end, otherwise we are talking about an error-prone manual process which is massively difficult for large orgs.
I had the exact same thing happen yesterday. I assumed it was because I generated new SSH keys before revoking my oAuth relationship between CircleCI and Github. After regenerating the keys with my new oAuth session things look better.
But I would like confirmation that this isn’t the root cause so I can be on the lookout for our keys getting removed again.
To follow up on this briefly, I was able to match up some of my user keys via their fingerprints, and found that they’d been deleted on the CircleCI side, but were still valid on the GitHub side.
