AT&T’s 1990 Long-Distance Collapse: The Switch Bug That Propagated Across a Network
A subtle recovery-code defect in AT&T's 4ESS switching software turned a local reset into a nationwide cascade, showing how identical code can turn redundancy into correlated failure.
The outage began with an ordinary recovery event, not an extraordinary attack
On January 15, 1990, AT&T’s long-distance network suffered one of the most consequential software outages of the pre-Internet era. A 4ESS switch in New York entered a normal recovery sequence after trouble in trunk-interface equipment. The network had been designed to report temporary unavailability to neighboring switches and then resume service. The initiating event was therefore the sort of local equipment fault the system was expected to absorb, not a catastrophe that should have threatened the national network.[1] For reliability engineering, the broader lesson is fault domains must remain independent enough that recovery traffic or a shared software defect cannot synchronize an entire network. A mature design assumes local faults will occur and asks explicitly how far they can propagate before isolation or fallback takes over.
Recovery code can be more dangerous than application code
Recovery logic runs precisely when a system is already stressed. It needs to be simpler, more isolated and more thoroughly tested than ordinary feature code, because a mistake there can convert a contained fault into a synchronized failure.
A recently deployed software change created a timing-sensitive logic error
AT&T had installed new software across its 4ESS fleet in the weeks before the incident. Contemporary reporting said Bell Labs engineers reproduced a logic error after the outage, and later accounts described a subtle control-flow mistake that appeared only when particular messages arrived in a narrow sequence. The code worked during normal call processing and during many individual recovery tests. What mattered was the interaction between closely timed messages and the recovery path, a classic example of a defect hidden in concurrency and timing rather than in a simple deterministic calculation.[2] That distinction matters because critical infrastructure is judged by continuity of service, not by the sophistication of any single component. In practice, fault domains must remain independent enough that recovery traffic or a shared software defect cannot synchronize an entire network has to be treated as an architectural requirement rather than an operational aspiration.
The rare path was the real production path
Large systems spend most of their lives outside the exceptional states that engineers fear most. Reliability work has to search intentionally for those rare interleavings instead of treating them as too improbable to deserve dedicated tests.
Identical software across the network made the fault correlated
The 4ESS switches were built for high availability, but they shared the same new software. When one recovering switch sent messages to its neighbors, those neighbors could enter the same bad state and reset. Their own recovery traffic then exposed more peers to the defect. The result was not the independent failure of many machines; it was a correlated failure of many machines carrying the same vulnerable logic. Standardization reduced operational complexity while simultaneously creating a common-mode risk across the network.[3] The case therefore rewards a systems view. Engineers have to examine software, network topology, operating procedures, alarms, recovery access and organizational authority together, because fault domains must remain independent enough that recovery traffic or a shared software defect cannot synchronize an entire network only works when all of those layers support it.
Redundancy is not independence
Duplicating hardware does not provide meaningful fault tolerance when every replica has the same software defect and receives the same triggering input. Diversity or stronger containment boundaries may be required for genuinely independent failure modes.
Self-healing behavior became the mechanism of propagation
A striking feature of the incident is that the switches were trying to recover. The network’s automatic mechanisms announced failure, restarted processing and exchanged signaling information as designed. But recovery messages interacted with the faulty code in peer switches, producing another cycle of resets. A mechanism intended to restore service therefore helped distribute the trigger. This is a recurring pattern in distributed systems: automation that is safe in isolation can become destabilizing when every node responds to the same condition at once.[4] The enduring point is less about one historical product than about failure containment. Fault domains must remain independent enough that recovery traffic or a shared software defect cannot synchronize an entire network is what prevents a routine defect, bad update or hostile action from becoming a public-service emergency.
Automation needs dampers as well as triggers
Backoff, rate limits, circuit breakers and staged recovery are not cosmetic safeguards. They keep restorative behavior from becoming a positive-feedback loop when many components react simultaneously.
The blast radius reached millions of calls because the control plane was shared
News reports described tens of millions of calls that could not be completed during roughly nine hours of disrupted service. The failure affected ordinary long-distance traffic as well as services that depended on AT&T’s signaling infrastructure. The incident showed that a control-plane defect can have an impact far larger than the physical equipment where it begins. Once shared routing and signaling state becomes unstable, healthy links and switches may still be unable to deliver useful service.[5] For reliability engineering, the broader lesson is fault domains must remain independent enough that recovery traffic or a shared software defect cannot synchronize an entire network. A mature design assumes local faults will occur and asks explicitly how far they can propagate before isolation or fallback takes over.
Engineers restored service by removing the common trigger
AT&T engineers traced the problem to the new software and applied overrides that restored stable network behavior. The operational response is significant because it illustrates the value of a known-good fallback. Once a fleet-wide deployment is suspected, recovery depends on the ability to stop propagation and move quickly to a safer software state. A rollback mechanism is especially important in infrastructure where every minute of instability affects millions of users and where troubleshooting traffic itself may burden the failing system.[1] That distinction matters because critical infrastructure is judged by continuity of service, not by the sophistication of any single component. In practice, fault domains must remain independent enough that recovery traffic or a shared software defect cannot synchronize an entire network has to be treated as an architectural requirement rather than an operational aspiration.
The incident became an early lesson in distributed common-mode failure
Later reliability literature repeatedly cited the AT&T outage because the network contained extensive redundancy yet still failed widely. The shared software and signaling protocol created dependencies that were not visible if each switch was considered separately. Modern cloud regions, carrier cores and service meshes face the same design question: which components are truly independent, and which merely look redundant while sharing binaries, configuration, control channels or recovery logic that can fail together?[3] The case therefore rewards a systems view. Engineers have to examine software, network topology, operating procedures, alarms, recovery access and organizational authority together, because fault domains must remain independent enough that recovery traffic or a shared software defect cannot synchronize an entire network only works when all of those layers support it.
Why the AT&T collapse still matters to software engineers
The 1990 collapse belongs in software history because it demonstrates how a tiny defect can become a systems failure only when architecture gives it permission to spread. The lesson is not that large networks should avoid automation or shared software. It is that recovery paths, fleet-wide deployments and control-plane messages deserve adversarial testing of propagation behavior. High availability is ultimately a property of interactions among components, not a count of how many redundant boxes exist in a diagram.[5] The enduring point is less about one historical product than about failure containment. Fault domains must remain independent enough that recovery traffic or a shared software defect cannot synchronize an entire network is what prevents a routine defect, bad update or hostile action from becoming a public-service emergency.
Works Cited
- 01
- 02Washington Post — AT&T Back to Normal After Network Failure washingtonpost.com
- 03Telephone World / RISKS Digest — The Crash of the AT&T Network in 1990 telephoneworld.org
- 04Sterling — The Hacker Crackdown, AT&T Network Crash Chapter lysator.liu.se
- 05
CodeHistory is a living archive. Citations document the evidence used for this edition; later evidence may refine the account.