# \#incident **URL:** https://discuss.circleci.com/tag/incident/126.md [Latest](https://discuss.circleci.com/latest.md) · [Categories](https://discuss.circleci.com/categories.md) · [Tags](https://discuss.circleci.com/tags.md) --- ## [Post Incident Report: Oct 9, 2026 - Delay in data appearing in UI and API](https://discuss.circleci.com/t/post-incident-report-oct-9-2026-delay-in-data-appearing-in-ui-and-api/54919)
**Author:** [@amitm](https://discuss.circleci.com/u/amitm)\ **Replies:** 0\ **Last updated:** [October 12, 2026, 10:40pm UTC](https://discuss.circleci.com/t/post-incident-report-oct-9-2026-delay-in-data-appearing-in-ui-and-api/54919 "2026-10-12T22:40:18Z")
Summary Between 11:15 UTC and 23:47 UTC on October 9, CircleCI customers experienced hours-long delays and failures across pipelines, workflows and jobs, along with delays in the CircleCI UI, API and notifications. Fro… --- ## [Post Incident Report: Oct 1 to 8, 2026 - Delays and failures starting jobs](https://discuss.circleci.com/t/post-incident-report-oct-1-to-8-2026-delays-and-failures-starting-jobs/54915)
**Author:** [@anthonycastro](https://discuss.circleci.com/u/anthonycastro)\ **Replies:** 0\ **Last updated:** [October 9, 2026, 10:30pm UTC](https://discuss.circleci.com/t/post-incident-report-oct-1-to-8-2026-delays-and-failures-starting-jobs/54915 "2026-10-09T22:30:04Z")
Summary Between October 1 and October 8, 2026, CircleCI customers experienced five separate incidents in which jobs waited longer than normal to start, or failed to start. Three of the incidents were driven by limited in… --- ## [Post Incident Report: Oct 6, 2026 - Delays in pipelines, workflows, UI data, and notifications](https://discuss.circleci.com/t/post-incident-report-oct-6-2026-delays-in-pipelines-workflows-ui-data-and-notifications/54908)
**Author:** [@amitm](https://discuss.circleci.com/u/amitm)\ **Replies:** 0\ **Last updated:** [October 7, 2026, 10:19pm UTC](https://discuss.circleci.com/t/post-incident-report-oct-6-2026-delays-in-pipelines-workflows-ui-data-and-notifications/54908 "2026-10-07T22:19:29Z")
Summary On October 6, 2026, from 15:30 to 20:30 UTC, CircleCI customers experienced delays creating pipelines and starting workflows and jobs. From 15:48 to 16:38 UTC, no workflows or jobs started, and pipelines triggere… --- ## [Post Incident Report: July 20, 2026 - Errors with Github API Delaying Workflows](https://discuss.circleci.com/t/post-incident-report-july-20-2026-errors-with-github-api-delaying-workflows/54816)
**Author:** [@amitm](https://discuss.circleci.com/u/amitm)\ **Replies:** 0\ **Last updated:** [July 23, 2026, 9:23pm UTC](https://discuss.circleci.com/t/post-incident-report-july-20-2026-errors-with-github-api-delaying-workflows/54816 "2026-07-23T21:23:15Z")
Summary On July 20, 2026 from 00:22 to 01:45 UTC, CircleCI customers using our GitHub integrations experienced failures running pipelines and experienced workflows getting stuck. During this incident, some pipelines trig… --- ## [Post Incident Report: 2026-07-13 Network Routing Issues Cause Intermittent macOS Job Failures](https://discuss.circleci.com/t/post-incident-report-2026-07-13-network-routing-issues-cause-intermittent-macos-job-failures/54785)
**Author:** [@rob\_hartsock](https://discuss.circleci.com/u/rob_hartsock)\ **Replies:** 0\ **Last updated:** [July 16, 2026, 2:00am UTC](https://discuss.circleci.com/t/post-incident-report-2026-07-13-network-routing-issues-cause-intermittent-macos-job-failures/54785 "2026-07-16T02:00:47Z")
Summary From 18:00 UTC on July 13, 2026 to 23:18 UTC on July 14, 2026, some customers running jobs on CircleCI’s macOS fleet experienced jobs that hung or failed to progress during GitHub fetch steps. For these customers… --- ## [Post Incident Report: 2026-07-14 Bitbucket OAuth Change Causes Login Failures and Workflow Issues](https://discuss.circleci.com/t/post-incident-report-2026-07-14-bitbucket-oauth-change-causes-login-failures-and-workflow-issues/54781)
**Author:** [@rob\_hartsock](https://discuss.circleci.com/u/rob_hartsock)\ **Replies:** 0\ **Last updated:** [July 14, 2026, 10:36pm UTC](https://discuss.circleci.com/t/post-incident-report-2026-07-14-bitbucket-oauth-change-causes-login-failures-and-workflow-issues/54781 "2026-07-14T22:36:00Z")
Summary From 01:33 UTC to 05:18 UTC on July 14, 2026, customers with a Bitbucket-linked identity were unable to log in to CircleCI, and some customers experienced workflows that failed to start or update, due to a change… --- ## [Post Incident Report: July 2, 2026 - Outage impacting web experience and pipelines starting](https://discuss.circleci.com/t/post-incident-report-july-2-2026-outage-impacting-web-experience-and-pipelines-starting/54759)
**Author:** [@shelly](https://discuss.circleci.com/u/shelly)\ **Replies:** 0\ **Last updated:** [July 10, 2026, 7:03pm UTC](https://discuss.circleci.com/t/post-incident-report-july-2-2026-outage-impacting-web-experience-and-pipelines-starting/54759 "2026-07-10T19:03:50Z")
Summary On July 2, 2026, from 15:12 UTC to 16:44 UTC, some CircleCI customers experienced failures starting pipelines and difficulty accessing the CircleCI UI, including an inability to log in. Customers triggering pipel… --- ## [Post Incident Report: June 23, 2026 - Delays starting Docker jobs](https://discuss.circleci.com/t/post-incident-report-june-23-2026-delays-starting-docker-jobs/54750)
**Author:** [@anthonycastro](https://discuss.circleci.com/u/anthonycastro)\ **Replies:** 0\ **Last updated:** [June 27, 2026, 12:15am UTC](https://discuss.circleci.com/t/post-incident-report-june-23-2026-delays-starting-docker-jobs/54750 "2026-06-27T00:15:08Z")
Summary On Tuesday, June 23, 2026, our engineering team identified that a single Docker job cluster was operating below expected performance levels. Our infrastructure is organized into routing groups, each responsible f… --- ## [Post Incident Report: December 2, 2025 - Jobs not starting, Jobs stuck in running state, Pipelines page not loading](https://discuss.circleci.com/t/post-incident-report-december-2-2025-jobs-not-starting-jobs-stuck-in-running-state-pipelines-page-not-loading/54291)
**Author:** [@shelly](https://discuss.circleci.com/u/shelly)\ **Replies:** 0\ **Last updated:** [December 18, 2025, 9:37pm UTC](https://discuss.circleci.com/t/post-incident-report-december-2-2025-jobs-not-starting-jobs-stuck-in-running-state-pipelines-page-not-loading/54291 "2025-12-18T21:37:31Z")
Summary All timestamps are in UTC On December 2, 2025, at 16:20, we deployed a code change to workflow status computation that introduced a race condition in how workflows are terminated. This caused the following impac… --- ## [Post Incident Report: May 1, 2025 - Delays in starting some jobs](https://discuss.circleci.com/t/post-incident-report-may-1-2025-delays-in-starting-some-jobs/53307)
**Author:** [@shelly](https://discuss.circleci.com/u/shelly)\ **Replies:** 0\ **Last updated:** [May 22, 2025, 4:02pm UTC](https://discuss.circleci.com/t/post-incident-report-may-1-2025-delays-in-starting-some-jobs/53307 "2025-05-22T16:02:14Z")
Summary On May 1, 2025, from 22:20 UTC to May 2, 2025 02:00 UTC, CircleCI customers experienced delays in starting most jobs. Jobs affected were contained to the following resource classes: Docker large, Docker medium, D… --- ## [Post Incident Report: April 4, 2025 - CircleCI UI Loading & build triggering issues](https://discuss.circleci.com/t/post-incident-report-april-4-2025-circleci-ui-loading-build-triggering-issues/53208)
**Author:** [@amitm](https://discuss.circleci.com/u/amitm)\ **Replies:** 0\ **Last updated:** [May 2, 2025, 2:20pm UTC](https://discuss.circleci.com/t/post-incident-report-april-4-2025-circleci-ui-loading-build-triggering-issues/53208 "2025-05-02T14:20:47Z")
Summary On April 4, 2025, from 00:16 to 01:49 UTC (approximately 1 hour and 33 minutes), CircleCI experienced a service disruption affecting both our user interface and build capabilities. During this time, customers wer… --- ## [Post Incident Report: April 4, 2025 - Delays in starting workflows](https://discuss.circleci.com/t/post-incident-report-april-4-2025-delays-in-starting-workflows/53113)
**Author:** [@shelly](https://discuss.circleci.com/u/shelly)\ **Replies:** 0\ **Last updated:** [April 16, 2025, 1:41pm UTC](https://discuss.circleci.com/t/post-incident-report-april-4-2025-delays-in-starting-workflows/53113 "2025-04-16T13:41:43Z")
Summary On April 3, 2025, from 22:08 UTC to 23:45 UTC, CircleCI customers experienced increased latency and some failures with starting and canceling workflows and jobs. During this time customers may have experienced de… --- ## [Post Incident Report: January 23, 2025 - Some commit status updates were not published](https://discuss.circleci.com/t/post-incident-report-january-23-2025-some-commit-status-updates-were-not-published/52697)
**Author:** [@shelly](https://discuss.circleci.com/u/shelly)\ **Replies:** 0\ **Last updated:** [January 31, 2025, 9:42pm UTC](https://discuss.circleci.com/t/post-incident-report-january-23-2025-some-commit-status-updates-were-not-published/52697 "2025-01-31T21:42:42Z")
Summary On January 23, 2025, from 19:48 UTC to 20:43 UTC, customers using CircleCI GitHub OAuth and Bitbucket projects stopped receiving commit status updates. This was due to a code change deployed at 19:48 UTC that neg… --- ## [Post Incident Report: January 22, 2025 - Delays starting and canceling workflows and jobs, and viewing them in the UI](https://discuss.circleci.com/t/post-incident-report-january-22-2025-delays-starting-and-canceling-workflows-and-jobs-and-viewing-them-in-the-ui/52696)
**Author:** [@shelly](https://discuss.circleci.com/u/shelly)\ **Replies:** 0\ **Last updated:** [January 31, 2025, 9:37pm UTC](https://discuss.circleci.com/t/post-incident-report-january-22-2025-delays-starting-and-canceling-workflows-and-jobs-and-viewing-them-in-the-ui/52696 "2025-01-31T21:37:52Z")
Summary From January 21, 2025 at 23:50 UTC to January 22, 2025 at 00:56 UTC, CircleCI customers experienced increased latency with starting and canceling workflows and jobs, and experienced delays and difficulty viewing … --- ## [Incident Report: 2023-03-14 - Delays Starting Jobs](https://discuss.circleci.com/t/incident-report-2023-03-14-delays-starting-jobs/47555)
**Author:** [@naoya](https://discuss.circleci.com/u/naoya)\ **Replies:** 0\ **Last updated:** [March 24, 2023, 1:34am UTC](https://discuss.circleci.com/t/incident-report-2023-03-14-delays-starting-jobs/47555 "2023-03-24T01:34:20Z")
Incident Report: 2023-03-14 - Delays Starting Jobs Summary: On March 14, 2023, at approximately 18:00 UTC, all customers’ pipelines began to experience delays in starting, ultimately resulting in jobs not running. This i… --- ## [We’re improving our macOS Gen2 fleet capacity - what you need to know](https://discuss.circleci.com/t/we-re-improving-our-macos-gen2-fleet-capacity-what-you-need-to-know/43777)
**Author:** [@alexazl](https://discuss.circleci.com/u/alexazl)\ **Replies:** 1\ **Last updated:** [April 26, 2022, 6:40pm UTC](https://discuss.circleci.com/t/we-re-improving-our-macos-gen2-fleet-capacity-what-you-need-to-know/43777 "2022-04-26T18:40:31Z")
We want to let you know about a change we’re making to help improve capacity across our macOS Gen2 fleet. There is no action required on your end. What’s happening: As a result of global supply chain issues, we are exp… --- ## [Incident Report: November 8, 2021 - Jobs stuck in a "not running" state](https://discuss.circleci.com/t/incident-report-november-8-2021-jobs-stuck-in-a-not-running-state/41890)
**Author:** [@yannCI](https://discuss.circleci.com/u/yannCI)\ **Replies:** 0\ **Last updated:** [November 11, 2021, 1:53pm UTC](https://discuss.circleci.com/t/incident-report-november-8-2021-jobs-stuck-in-a-not-running-state/41890 "2021-11-11T13:53:47Z")
Incident Report: November 8, 2021 - Jobs stuck in a “not running” state Summary Customer jobs were blocked in a “not running” state from approximately 18:50 UTC to 19:43 UTC on Monday, November 8, 2021, due to a database… --- ## [Incident Report: October 27, 2021 - VMs failing to be created](https://discuss.circleci.com/t/incident-report-october-27-2021-vms-failing-to-be-created/41769)
**Author:** [@JonathanC503](https://discuss.circleci.com/u/JonathanC503)\ **Replies:** 0\ **Last updated:** [November 2, 2021, 9:31pm UTC](https://discuss.circleci.com/t/incident-report-october-27-2021-vms-failing-to-be-created/41769 "2021-11-02T21:31:05Z")
Summary Customers experienced lengthy delays in job starts and stuck builds from 17:00 UTC October 27 to 01:16 UTC October 28. All jobs were affected, starting with machine jobs and continuing into docker executor as th… --- ## [Incident Report: September 27, 2021 - Delays in Workflows](https://discuss.circleci.com/t/incident-report-september-27-2021-delays-in-workflows/41443)
**Author:** [@JonathanC503](https://discuss.circleci.com/u/JonathanC503)\ **Replies:** 0\ **Last updated:** [October 5, 2021, 9:40pm UTC](https://discuss.circleci.com/t/incident-report-september-27-2021-delays-in-workflows/41443 "2021-10-05T21:40:12Z")
Incident Report: September 27, 2021 - Delays in Workflows Summary: On September 27, 2021, from 07:05 to 14:46 UTC CircleCI customers experienced delays in workflows. Due to our efforts to fix the initial incident, some c… --- ## [Postmortem: May 21, 2021 - Delay in starting Docker Jobs. Machine & remote Docker environments blocked](https://discuss.circleci.com/t/postmortem-may-21-2021-delay-in-starting-docker-jobs-machine-remote-docker-environments-blocked/40274)
**Author:** [@JonathanC503](https://discuss.circleci.com/u/JonathanC503)\ **Replies:** 0\ **Last updated:** [June 3, 2021, 2:11pm UTC](https://discuss.circleci.com/t/postmortem-may-21-2021-delay-in-starting-docker-jobs-machine-remote-docker-environments-blocked/40274 "2021-06-03T14:11:05Z")
Summary: Starting on May 21, 2021, at 13:30, UTC CircleCI customers running Docker, Machine executor (which includes Windows, Mac, Arm, and GPU) and remote Docker jobs experienced delayed or blocked pipelines. This incid… --- ## [Failure to allocate a remote Docker Engine](https://discuss.circleci.com/t/failure-to-allocate-a-remote-docker-engine/30348)
**Author:** [@csscif](https://discuss.circleci.com/u/csscif)\ **Replies:** 1\ **Last updated:** [May 14, 2019, 7:01pm UTC](https://discuss.circleci.com/t/failure-to-allocate-a-remote-docker-engine/30348 "2019-05-14T19:01:59Z")
This issue has been occurring for the past hour… Allocating a remote Docker Engine Waiting for a Docker Engine assignment: ................................................................................................… --- ## [Convenience Images Update: 2018.11.27 Node variants incident](https://discuss.circleci.com/t/convenience-images-update-2018-11-27-node-variants-incident/27025)
**Author:** [@rose](https://discuss.circleci.com/u/rose)\ **Replies:** 0\ **Last updated:** [December 8, 2018, 1:15am UTC](https://discuss.circleci.com/t/convenience-images-update-2018-11-27-node-variants-incident/27025 "2018-12-08T01:15:43Z")
During the week of Sunday, November 25th, an upstream change affected a subset of CircleCI convenience images, causing Node variants of some images to fail in CircleCI jobs. What happened We create Node variants of our … --- ## [April 25 Machine Outage](https://discuss.circleci.com/t/april-25-machine-outage/22017)
**Author:** [@mharriscircleci](https://discuss.circleci.com/u/mharriscircleci)\ **Replies:** 0\ **Last updated:** [April 26, 2018, 7:49pm UTC](https://discuss.circleci.com/t/april-25-machine-outage/22017 "2018-04-26T19:49:07Z")
On April 25 at 05:17 UTC the performance of a database query degraded, preventing the timely allocation of machine resources. This degradation impacted MacOS, machine executor and remote Docker jobs, and led to the failu… --- ## [Our build console is down. Anybody else?](https://discuss.circleci.com/t/our-build-console-is-down-anybody-else/10718)
**Author:** [@foobarnum](https://discuss.circleci.com/u/foobarnum)\ **Replies:** 13\ **Last updated:** [March 1, 2017, 9:57pm UTC](https://discuss.circleci.com/t/our-build-console-is-down-anybody-else/10718 "2017-03-01T21:57:07Z")
1-2 day standard support doesn't help when all of our development is stopped --- ## [Fixed: Unable to Login due to revoked GitHub token](https://discuss.circleci.com/t/fixed-unable-to-login-due-to-revoked-github-token/5856)
**Author:** [@bear](https://discuss.circleci.com/u/bear)\ **Replies:** 6\ **Last updated:** [August 25, 2016, 9:44am UTC](https://discuss.circleci.com/t/fixed-unable-to-login-due-to-revoked-github-token/5856 "2016-08-25T09:44:07Z")
We are experiencing frequent but non-deterministic issues when logging in because we are incorrectly revoking github tokens. When this happens, you will receive an email from us saying your token has been revoked. This c…