GitHub 2022: The Database Service Incident That Exposed Cross-Service Dependencies
GitHub’s March 2022 outages showed how one shared MySQL cluster with too little headroom could impair Git, pull requests, Actions, Packages, Codespaces, Pages, and API writes together.
GitHub entered March 2022 with too much traffic concentrated on mysql1
GitHub disclosed a series of March 2022 incidents centered on resource contention in a database cluster called mysql1.[1] The cluster received heavy read and write traffic from many parts of GitHub. During the March 16 incident, peak load and poorly performing queries drove the proxy layer to its maximum connection count. Write operations failed across Git, webhooks, pull requests, API requests, issues, Packages, Codespaces, Actions and Pages. The database did not merely support one feature; it was a shared state dependency underneath much of the product.
Shared databases create organizationally invisible coupling
Teams can own separate services while still depending on the same primary write path. Service boundaries in code do not guarantee independent failure domains.
Failover restored service but did not remove the underlying load pattern
GitHub recovered the March 16 incident by failing over to a healthy replica, but the next day the same peak pattern returned.[1] A proactive failover then exposed a different connectivity problem on the new primary. The sequence illustrates the limits of failover as a universal cure. High availability can replace a failed machine; it cannot automatically fix a workload whose query behavior and connection demand exceed the capacity of every equivalent primary.
Failover is a recovery mechanism, not a capacity strategy
If the logical bottleneck moves with the primary role, promoting a replica may only reset the clock until traffic recreates the same pressure.
One index fixed a major query problem but headroom remained too small
GitHub identified a poorly performing query pattern and added an index, reducing the immediate load.[2] Yet its March availability report emphasized the broader issue: the primary database had too little operational headroom, so a few expensive queries could threaten availability. This is a mature diagnosis because it separates trigger from vulnerability. The bad query mattered, but a system with adequate margin would have degraded less dramatically. Capacity planning is therefore about uncertainty, not merely average utilization.
Headroom is protection against the workload you did not predict
Running a critical database permanently hot makes every query regression, traffic burst or maintenance task a potential outage.
Downstream queues remained unhealthy after database writes recovered
GitHub Actions accumulated a large queue while core services were impaired, and the company had to re-enable processing gradually after the database incidents.[2] This extended visible impact beyond the database recovery itself. Distributed systems often store failure as backlog: jobs, webhooks, CI runs and notifications remain waiting after the root service returns. Recovery planning must therefore include the maximum drain rate of downstream queues, or the incident tail can last longer than the initiating failure.
Queues preserve work but also preserve load
A durable queue converts immediate loss into future demand. Systems need enough burst capacity to process accumulated work without triggering another overload.
GitHub had already been partitioning its main database as it grew
GitHub noted that database partitioning and additional clusters were ongoing parts of its scaling strategy.[1] Its engineering writing on MySQL high availability describes classic primary-replica topology and the operational machinery around failover.[3] The March incidents showed why scale eventually pressures architecture as well as hardware. If too many unrelated writes must serialize through one logical primary, adding replicas helps reads but does not remove the write bottleneck.
The company accelerated sharding and tightened alert thresholds
GitHub lowered database performance alert thresholds, accelerated efforts to shard mysql1, and created focused engineering work around observability, release processes and operational procedures.[2] Those actions attack both technical concentration and organizational response. Sharding can reduce blast radius and distribute writes, while earlier alerts buy time before the database reaches saturation. Neither helps if teams lack a clear model of which services depend on which shard, so dependency mapping remains essential.
A service architecture is only as independent as its state architecture
GitHub’s products appear modular to users—Actions, Pages, Packages, Codespaces, pull requests—but many workflows converge on shared repository and account state. Database partitioning work at GitHub illustrates the long-term effort required to untangle that state without breaking product semantics.[4] Reliability reviews should therefore trace shared databases, queues and identity systems, not stop at API boundaries. Cross-service dependencies are often strongest where state is shared.
Why GitHub 2022 belongs in software-failure history
GitHub’s 2022 database incidents belong in software history because they reveal how a mature, service-oriented platform can still have a concentrated write dependency. Resource contention in mysql1 propagated into Git operations, pull requests, APIs, Actions and several other products, and recovery generated downstream queue pressure after the database itself improved. The lasting lesson is that software modularity does not automatically create reliability isolation. To reduce blast radius, organizations must partition state, preserve capacity headroom and understand which supposedly separate services converge on the same critical database path.[5] The broader reliability principle is to make recovery a first-class production path. Teams should rehearse the exact operations they expect to use during failure, verify that monitoring and administrative tooling survive the same dependency loss, and measure how quickly backlogs and replacement capacity return to steady state. A recovery plan that exists only as documentation has not yet been tested as software.
Works Cited
- 01GitHub — An Update on Recent Service Disruptions github.blog
- 02GitHub — Availability Report: March 2022 github.blog
- 03
- 04
- 05GitHub Docs — GitHub Status and Availability githubstatus.com
CodeHistory is a living archive. Citations document the evidence used for this edition; later evidence may refine the account.