I did some investigation on the failed assume role. It seems that circleci changed the private key for signing the OIDC JWT token but didn’t update https://oidc.circleci.com/org/<project_id>/.well-known/jwks-pub.json
Tried throwing the OIDC JWT token to https://jwt.io/ and validate it. They are saying “signature verification failed” in console and invalid signature in the UI.
Folks, as per @philglass’ observation, in the last couple of hours we’ve taken the step of removing all personal and project API tokens created before we released the security alert. This means you can focus on other efforts as part of your own remediation.
We apologise for the impact this might have in having to recreate tokens, but took this action in the belief that it is in the best interests of our users, minimising risk whilst also reducing the burden on our users having to respond.
They have been saying all along - there is nothing to look for in the CircleCI audit logs. We need to check all the services/systems that had secrets in CircleCI for “suspicious activity” associated to those creds/tokens. Yes, I agree, that is not easy or specific enough.
It’s actually very difficult to do this properly on the customer / client side because we have to somehow convince every user of our org who might have created an OAuth token for CircleCI to go and do this - 100s of users potentially - some of whom might be on leave etc.
Has CircleCI taken action to remove user SSH checkout keys?
Earlier today I made a report using the API, and now those keys are not showing up.
This is problematic, as I need to assume those keys are compromised, but I only have the key fingerprints from earlier today - which isn’t enough to track them down and ensure they’re revoked at the GitHub end (the API response doesn’t note which user the keys belonged to).
Anyone who didn’t already list these keys earlier today won’t be able to get a list at all.
I had the exact same thing happen yesterday. I assumed it was because I generated new SSH keys before revoking my oAuth relationship between CircleCI and Github. After regenerating the keys with my new oAuth session things look better.
But I would like confirmation that this isn’t the root cause so I can be on the lookout for our keys getting removed again.