All of a sudden started working again this saturday… The only thing on the circleci status page says that there was a rollout for a fix that only affected 2% of build jobs? I find it hard to believe ours fell into the 2%
As far as I know, our percentage rollout system uses a consistent hash of your project’s ID. So if your project was part of the 2% rollout, when every job would have been affected. Our goal with this system is to make it easier for us to reason about the system when we are deploying changes that we want to keep a close eye on. In particular, we want to ensure that if we re-run a job that exposed a problem, the re-run should also run with the same set of features enabled.
Sorry for any problems that this caused for you and your team.
thanks for the reply… I’m assuming we ran into a different issue then because not all of our jobs were failing. Only our Xcode build step that would timeout and run forever… with no changes they started working again this saturday but i still don’t have the root cause or able to relay this to manager/company etc for root cause.