Osegi
Web & Dev·2 min read

Microsoft Outlook outage: a July code change caused it

Microsoft says the Exchange Online and Outlook outage came from a defect it shipped in July. It sat quietly for months, and a rollback was not an option.

Rows of servers in a data center with warning lights, illustrating a large cloud email outage.
Image: theregister

The Microsoft Outlook outage that left Microsoft 365 users without email last week traces back to a code change from July. For months it did nothing. Then, on November 12, it took down most of a network gateway fleet.

According to a preliminary post incident report Microsoft published on November 15, the update had passed load testing and rolled out without any visible impact. The problem: "Our load testing tended to focus on weaker hardware with fewer cores." The defect caused lock contention (processes fighting over the same resource) inside a Windows process, which used up CPU and memory. Microsoft says it gets worse on machines with more cores and newer hardware.

What triggered the Exchange Online outage?

Two ordinary things happened together. Teams failed over to the primary gateway that supports Exchange Online, which created a load surge. At the same time, almost 14 percent of the gateway's server capacity was out of service and on its way back. Microsoft says there was enough headroom for either event alone. Together they hit the defect.

Then came a cascade. When a server stopped responding, traffic moved to the next healthy one, which hit the same bug. Within two hours the defect had spread across the fleet, and at the lowest point more than 80 percent of servers were unresponsive.

Why was it so hard to fix?

Restarting servers did not help, because they were overwhelmed again straight away. Microsoft's automated recovery "was not designed to restore service from this broad unresponsive state," and a simple rollback was off the table because the defect was months old. Engineers wrote a patch for the patch: an update returning the code to its previously known good state in Windows, applied to groups of servers one by one.

Teams, OneDrive, SharePoint and Office apps could fail out of the broken gateways. Exchange Online could not, for scale reasons, which is why email stayed down longest.

The timeline: the Teams failover came at 0107 UTC on November 12, Microsoft's monitor flagged connectivity problems at 1603 UTC, and the first support case arrived at 1633 UTC. The fix was applied at 0720 UTC on November 13. Microsoft confirmed healthy service at 1610 UTC and sent its final customer update at 0415 UTC on November 14. A final report is due on November 20.

Apps worth a look

SUMO: the number puzzle

Tap the right tiles, add them up, hit the target. Easy to learn, hard to master. Free, for all ages.

Tags:microsoftoutlookexchange onlinemicrosoft 365outageazure

More from this day

See all →

More in: Web & Dev

See all →