Osegi
Security·2 min read

FAA ground stop: a router swap at a remote Arkansas site

A new router at a remote radio site in El Dorado, Arkansas, sat next to all 22 US air traffic centers on the network map. The FAA's nationwide ground stop followed.

Air traffic control tower at a US airport with grounded passenger jets waiting on the tarmac.
Image: Ars Technica

The FAA nationwide ground stop that stranded more than 1 million passengers began, by the agency's own early account, at a remote radio site in Arkansas where a contractor was swapping a router. The government is expected to announce the official root cause soon.

FAA Administrator Bryan Bedford said investigators are "very close" to an answer, and Transportation Secretary Sean Duffy plans to present the findings at a news conference as soon as Friday, November 20. The ground stop on November 13 was only the third nationwide one the FAA has ever issued. It lasted more than four hours. More than 12,000 flights were canceled that day, and the total passed 20,000 over the weekend.

What caused the FAA ground stop?

According to an FAA post on November 15, data began flooding the FAA Telecommunications Infrastructure (FTI) network, with 1,000 connection attempts every 4 to 5 seconds. FTI carries flight data, voice and video to air traffic control centers. The flood caused instability in aircraft communications and radar feeds at 17 of the 22 en-route centers.

The early investigation pointed to a radar site in Amarillo, Texas, and a radio site in El Dorado, Arkansas. Contractor L3Harris was installing a new Juniper Networks router at El Dorado to replace a 25-year-old Cisco one, and had recently put in a similar router in Texas. The FAA has since paused the Juniper rollout and switched the installed units off, going back to the old Cisco routers.

Who was overseeing the upgrade?

That is the open question. The PASS union, which represents FAA technicians, says none of its members were involved in the upgrades. CNN reported, citing the union and two sources, that an L3Harris engineer did the work without direct FAA oversight, and the FAA technician only escorted him in.

Philip Mann, a former FAA technician who spent 17 years at the agency and now advises on aviation safety, says the problem was basic risk management. The site looks remote on a map, but on the network diagram that router sat right next to all 22 centers, so the potential impact was nationwide. In that case, he says, you take the maximum precautions, including working at night when traffic is light. Duffy has reportedly acknowledged that the FAA mixed up a facility's physical location with how far its service reaches.

Apps worth a look

SUMO: the number puzzle

Tap the right tiles, add them up, hit the target. Easy to learn, hard to master. Free, for all ages.

Tags:faaground stopl3harrisjuniper networksair traffic controlrouter outage

More from this day

See all →

More in: Security

See all →