Osegi
Web & Dev·2 min read

Google Cloud blames PLC misconfiguration for London outage

Google Cloud says a PLC misconfiguration and a voltage-sensing flaw stopped backup generators from starting during the October 20 outage in London.

Rows of backup diesel generators outside a large datacenter building at dusk.
Image: theregister

Google Cloud has traced the October 20 outage in its europe-west2-b availability zone in London to a misconfigured programmable logic controller (PLC) that never told the backup generators to start. The disruption hit services at Lloyds, HSBC, Royal Mail and Revolut.

In a November 4 update to its status page, the company said the site uses a 2N utility power architecture, meaning complete duplicate power paths for every critical load. A central high-voltage generator bank is managed by a dual-redundant PLC. The facility is fed by two independent utility feeders, called A and B, plus a standby connection configured for manual campus-wide transfer.

A cable failure at the wrong moment

Google says the root cause was a failure in an external underground cable run by its power supplier, which connects to feeder A. Feeder B was out of action at the time: its circuit breaker had been physically opened for repairs after problems found during a quarterly generator load test on 14 October 2026. Feeder A was therefore carrying the entire critical load.

When feeder A failed, the uninterruptible power supplies, which can hold the zone for five minutes, did their job. But a "voltage-sensing design flaw and PLC misconfiguration" meant the generator plant received no automatic start signal. According to Google, the PLC logic indicated a healthy utility source was available even though both distribution feeds were de-energized, so it issued no start command. The UPSes drained and the zone lost power.

Fixes and follow-ups

After reconnecting to the utility, Google reprogrammed the PLC to prevent a repeat. It says it has made multiple changes to the electrical setup at the zone and tested that they work.

Two action items follow. A "Communications Enhancement" will build an automated alert-to-communication mechanism to speed up notifications for location-impacting incidents. Google also plans to expand platform-wide testing outcomes for infrastructure-level events.

For customers, the incident is a reminder that redundancy on paper depends on control logic working under unusual conditions, such as maintenance windows that reduce resilience. The faster-notification pledge also suggests customers were not told quickly enough as the outage unfolded.

Apps worth a look

SUMO: the number puzzle

Tap the right tiles, add them up, hit the target. Easy to learn, hard to master. Free, for all ages.

Tags:google cloudoutagelondondatacenterplceurope-west2

More from this day

See all →

More in: Web & Dev

See all →