# Post Incident Review: July 2nd Errors loading CircleCI

**URL:** <https://discuss.circleci.com/t/post-incident-review-july-2nd-errors-loading-circleci/51497>\
**Category:** Announcements\
**Tags:** post-incident-review\
**Created:** [July 19, 2024, 4:04pm UTC](https://discuss.circleci.com/t/post-incident-review-july-2nd-errors-loading-circleci/51497 "2024-07-19T16:04:06Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![kami](https://sea2.discourse-cdn.com/circleci/user_avatar/discuss.circleci.com/kami/32/35940_2.png) [@kami](https://discuss.circleci.com/u/kami)\
**Post date:** [July 19, 2024, 4:04pm UTC](https://discuss.circleci.com/t/post-incident-review-july-2nd-errors-loading-circleci/51497/1 "2024-07-19T16:04:06Z")

</div>

**Summary and Impact:**

On July 2, 2024, at 07:34 UTC, a deployment change in production using the MicroFrontends (MFEs) migration Ingress guide caused Kong routing to break for the [circleci.com](http://circleci.com/) host. This led to the unavailability of APIs and UI. The issue was resolved by reverting the change in web-ui-v1, restoring Kong routing.

‌

**Customer Impact Analysis:**

The incident had a major impact as customers were unable to access any UI or ingest pipelines. An estimated 16.5k pipelines were affected based on traffic data.

‌

**Background:**

The incident was related to merging MicroFrontends (MFEs) in the web-ui-consolidated monorepo. A change in Kong routing led to the breakdown of APIs and UI functionality.

‌

**What Happened:**

Issues arose due to how Kong handles route priorities, leading to unexpected behavior. The incident was exacerbated by a bug in the Kong router, impacting multiple routes.

‌

**Lessons Learned and Future Steps:**

- Conduct incident bot testing and improve incident response protocols.
- Investigate route priorities and upgrade Kong for better routing.
- Enhance internal documentation on route priorities to prevent similar incidents.

‌

**Timeline:**

- 07:35:46 UTC: Customer Impact Start
- 07:39:53 UTC: Initial investigation initiated
- 08:05:45 UTC: Helm Rollback applied, issue resolved
- 08:06:00 UTC: Customer Impact End
- 08:38:00 UTC: Incident End

‌

We are committed to improving our systems and processes to prevent such incidents in the future

---

<div class="post-metadata">

**Author:** ![kami](https://sea2.discourse-cdn.com/circleci/user_avatar/discuss.circleci.com/kami/32/35940_2.png) [@kami](https://discuss.circleci.com/u/kami)\
**Post date:** [July 19, 2024, 4:04pm UTC](https://discuss.circleci.com/t/post-incident-review-july-2nd-errors-loading-circleci/51497/2 "2024-07-19T16:04:32Z")

</div>

Closed on July 19, 2024, 4:04pm UTC