Rob Zuber here, CTO at CircleCI. I want to apologize to those of you experiencing forked cache breakages over the past few days. Firstly, I apologize for breaking your builds and interrupting your development. I know first hand how much havoc a broken build system can wreak on already aggressive development schedules. Second, I want to apologize for not communicating regarding changes to the behavior of the system, making those breakages that much more problematic and frustrating.
The change we rolled out around cache behavior was a quickly-deployed fix around a discovered security vulnerability. This “quick fix” was to close the initial hole, while we worked out a more tailored and complete resolution of the issue. We had intentionally held off public notification and discussion until we could ensure we had rolled out a complete fix. I apologize for the additional frustration and confusion that decision caused.
While we anticipated the change slowing down some builds, we did not expect it to break builds outright and stop workflows. I apologize for that as well. In our effort to get the exposure covered quickly, we failed to think through all of the ramifications.
We are actively working on additional changes to restore caching functionality on forks in a way that doesn’t re-open the vulnerability. In the meantime, if you are using caches to store assets between jobs in a workflow, this can be achieved with our workspaces functionality. More information is available here.
We will also be going through a full retrospective with our team to evaluate the process and decisions that got us here and ensure that we do better next time.