On April 25 at 05:17 UTC the performance of a database query degraded, preventing the timely allocation of machine resources. This degradation impacted MacOS, machine executor and remote Docker jobs, and led to the failure to initiate these job types. Regular service resumed at 10:18 UTC when the problematic query was identified and a solution was implemented along with other improvements to the reliability of builds that use virtual machines. We are continuing to investigate the cause of the sudden change in database behavior.
Related topics
| Topic | Replies | Views | Activity | |
|---|---|---|---|---|
| 2018-05-17 Workflows Outage Postmortem | 1 | 1235 | July 13, 2018 | |
| Postmortem: March 26 - April 10 Workflow Delay Incidents | 0 | 1627 | April 29, 2019 | |
| Postmortem: Incidents of October 22nd–29th | 0 | 1401 | November 15, 2019 | |
| Postmortem: May 21, 2021 - Delay in starting Docker Jobs. Machine & remote Docker environments blocked | 0 | 1598 | June 3, 2021 | |
| 2018-04-21 MacOS Planned Maintenance | 1 | 1505 | April 19, 2018 |