GitHub 2018: Twenty-Four Hours of Database Degradation and Replication Repair
GitHub's 2018 network partition triggered cross-site database failover, divergent MySQL histories, and a 24-hour recovery that prioritized data integrity over speed.
A network partition triggered a database failover that created divergent histories
On October 21, 2018, GitHub experienced a connectivity problem between data centers that disrupted the quorum used by its database failover tooling. Orchestrator began promoting MySQL primaries in the US West Coast site, and when connectivity returned, application writes were directed there even though a brief set of writes still existed only in the East Coast.[1] The result was not ordinary replica lag. GitHub now had database histories that had diverged across sites. The central recovery problem became how to preserve every accepted user write without merging incompatible timelines incorrectly.
Availability automation can create a consistency problem
Automatic failover is valuable, but a partition changes what the system can know. If two regions accept different writes, bringing service back is no longer just a routing decision; it becomes a data-reconciliation problem.
GitHub deliberately chose data integrity over a faster return to green
GitHub’s post-incident analysis says the degraded period lasted 24 hours and 11 minutes and that the team consciously prioritized consistency over restoring full usability quickly.[2] Engineers paused webhook delivery and other processes, rebuilt database clusters, and avoided a rapid failback that could have discarded writes. The site therefore remained partially available while showing stale or inconsistent metadata. This was a failure, but it also demonstrates mature incident prioritization: when recovery goals conflict, preserving user data can be more important than minimizing the visible outage duration.
Degraded service can be a safety mechanism
Temporarily disabling writes, queues or derived processing can contain uncertainty. A system that stays fully active during reconciliation may create more state than operators can safely reason about.
The topology manager did what it was designed to do under incomplete information
GitHub used Orchestrator to manage MySQL replication and automated failover. The tool maps replication topology, detects failures and can promote new primaries.[3] The incident did not prove that automation was a mistake. It showed that failover policies must be designed around network partitions and cross-site write semantics. Automation reduces human reaction time, but it also executes assumptions instantly. If those assumptions do not fully describe a partitioned world, a fast automated response can produce a topology that is available yet harder to reconcile.
Consensus protects the controller, not every database write
A highly available control plane can agree on which server should lead while application data still contains writes that never crossed the partition. Control-plane consensus and data-plane consistency solve related but different problems.
GitHub’s later architecture described stronger coordination around MySQL leadership
GitHub subsequently documented a high-availability design in which Orchestrator nodes use Raft consensus and communicate primary changes to local Consul installations.[4] That architecture illustrates the broader response to incidents like 2018: make failover state explicit, auditable and consistent across the infrastructure that routes traffic. High availability is not one mechanism. It is a chain from failure detection to leader election, service discovery, application routing and replication. A weakness at any point can turn a local fault into a global consistency event.
Recovery architecture must include service discovery
Promoting a database is insufficient if applications, queues and background jobs do not agree on the new primary. The routing layer is part of database correctness.
Backups solved capacity problems even though no user data was ultimately lost
GitHub rebuilt affected MySQL clusters from backups and then replicated forward from the West Coast state. The company noted that backups were taken every four hours and stored remotely, and that restoring multiple terabytes took substantial time because data had to be transferred, decompressed, verified and loaded.[5] This is a critical recovery lesson: a valid backup can still have a recovery-time problem. At large scale, bandwidth and rebuild throughput become part of the disaster-recovery design. A team must know not only that backups exist but how long the largest cluster will take to recreate.
The incident exposed the hidden backlog behind database recovery
Database consistency was only part of the work. GitHub had paused webhooks, Pages builds and other processing to protect state, which created millions of queued events. After databases were healthy, operators still had to drain those backlogs without overwhelming downstream systems or partners. This second phase is common in distributed-service recovery. Queues preserve work during an outage, but they also store future load. Capacity planning therefore needs a recovery mode that can process accumulated work faster than new work arrives, or degraded service can continue long after the original database fault is repaired.
No data loss does not mean the incident was harmless
GitHub reported that repository data remained available and that user data was not lost, but users still experienced stale metadata, delayed webhooks and unavailable Pages builds. Reliability therefore cannot be measured by durability alone. A service may preserve every bit while failing to present a coherent, timely view of those bits. Distributed systems must define acceptable staleness, consistency and recovery time as separate objectives. The October 2018 incident is valuable because GitHub made those tradeoffs explicit rather than collapsing them into a single uptime percentage.
Why GitHub 2018 belongs in software-failure history
GitHub’s 2018 database incident belongs in software history because it shows the uncomfortable edge of automated high availability: failover can preserve service while creating divergent histories that require careful repair. The company chose to remain degraded rather than risk user data, rebuilt large clusters from backups, reconstructed replication topologies and then worked through enormous queues of deferred processing. The lasting lesson is that multi-region resilience is not simply “have another data center.” It requires a consistency model for partitions, a tested plan for reconciling accepted writes, and recovery capacity large enough to rebuild the system without turning a short network fault into a day-long restoration.
Works Cited
- 01GitHub — October 21 Incident Report github.blog
- 02GitHub — October 21 Post-Incident Analysis github.blog
- 03
- 04
- 05
CodeHistory is a living archive. Citations document the evidence used for this edition; later evidence may refine the account.