April 25 Machine Outage

Announcements
1

On April 25 at 05:17 UTC the performance of a database query degraded, preventing the timely allocation of machine resources. This degradation impacted MacOS, machine executor and remote Docker jobs, and led to the failure to initiate these job types. Regular service resumed at 10:18 UTC when the problematic query was identified and a solution was implemented along with other improvements to the reliability of builds that use virtual machines. We are continuing to investigate the cause of the sudden change in database behavior.

2 Likes