🌿freegardner

Science

Microsoft's Dormant Gateway Bug: How Three Quiet Faults Became One Global Outage

16 Nov 2026 · via Arstechnica

Microsoft's Dormant Gateway Bug: How Three Quiet Faults Became One Global Outage
AI-generated image

The Defect That Slept Through Every Test

A change deployed to Microsoft’s cloud gateway starting in July introduced a flaw that sat undetected for months, and that flaw — not an attack, not a model, not a hacker — took down much of Microsoft 365 and Exchange Online for nearly a full day. The company’s preliminary Post-Incident Report places the disruption between November 12 at 16:20 UTC and November 13 at 16:10 UTC, just under 24 hours, with additional monitoring continuing through November 14. [4] The final status is listed plainly as a technical issue unrelated to cyber activity or AI agents. [4] The mechanism is the part worth sitting with. Under certain conditions, the gateway could exhaust its CPU and memory and stop responding. Pre-release testing never caught it. The reason is almost an irony of progress: the test hardware was less powerful than machines already running in production, and the higher CPU core counts on newer servers apparently helped trigger the lockout.

When the Backup Door Locks Too

Microsoft keeps two independent gateway systems for reaching Microsoft 365, so that if one path jams, the other carries the load. About 15 hours before the outage, traffic from Teams was shifted from the alternate gateway onto the primary system, where Exchange Online access is maintained. [4] The switchover was routine — a standard fail-safe during network maintenance and upgrades — but it changed both the volume and the mix of requests arriving at the primary gateway.

That shift is the second factor, and it is where the story turns from a single fault into a cascade. A separate, existing fault had left the gateway server fleet running at 14 percent below optimal capacity. That missing headroom would normally have absorbed the demand. Combined with the altered traffic pattern and the sleeping July defect, it produced lock contention as requests kept piling up. As US business operations reached their daily peaks, the gateway servers began to freeze.

Backup systems that route requests away from unresponsive servers did exactly what they were built to do — and met the same lockouts on the gateways presumed healthy. The result snowballed until more than 80 percent of the affected gateway servers were frozen. Regional isolation mechanisms, designed to contain exactly this kind of spread, appear to have failed.

Microsoft's Dormant Gateway Bug: How Three Quiet Faults Became One Global Outage (Image 1)

The Restart That Could Not Work

Recovery dragged on because the remedy was not a restart. Rebooting the gateway servers brought back large request queues still waiting for access, which immediately re-created the same resource exhaustion. Rolling back was equally useless: the July bug had never been discovered, so there was no earlier “working” state to return to. Engineers had to patch and return servers to service gradually instead.

Routing around the damage was possible for some services. Teams, OneDrive and other Office applications could be steered away from the affected infrastructure. Exchange Online could not make the same move, held back by scaling limits. Microsoft extended delivery timeouts on affected emails so that delayed messages would not bounce back as undeliverable, and says it has added capacity and traffic-distribution guardrails to reduce the chance of a repeat.

A full final incident report is expected on November 20.


Sources

  1. Exchange Online — unclear
Microsoft's Dormant Gateway Bug: How Three Quiet Faults Became One Global Outage (Image 2)
  1. Teams — Quote source (original article)
  2. OneDrive — Quote source (original article)
  3. Ars Technica — Quote source (original article)

Mentioned organisations (context, not sources)

← back to the garden