Slack January 2021: When the Workday Began with a Global Collaboration Outage
Slack’s first major workday of 2021 became a global outage when network packet loss, autoscaling behavior, and a provisioning bottleneck reinforced one another.
The outage began as subtle network degradation rather than an obvious server crash
On January 4, 2021, Slack’s external monitoring began showing rising error rates and latency during the Americas morning. Engineers initially saw broad signs of network trouble while the service still appeared mostly functional.[1] A normal traffic mini-peak around 7:00 a.m. PST arrived on top of the packet loss, causing requests from Slack’s web tier to back-end services to take longer. Worker threads accumulated waiting requests, system resources saturated, and Slack became unavailable. The failure illustrates how moderate network impairment can turn into application overload when concurrency and latency interact.
Latency can consume capacity without increasing CPU
A server waiting on slow back-end calls may show deceptively low CPU while its worker threads are exhausted. Scaling signals must reflect the actual constrained resource.
Autoscaling initially moved in the wrong direction
Slack scaled its web tier using CPU and Apache worker-thread utilization. Because network calls were slow, CPU utilization briefly fell, prompting some automated downscaling before thread pressure caused a much larger scale-up.[1] This is a classic control-system problem: the metric was normally correlated with load but became misleading during a network fault. Automation acted on a signal whose meaning had changed. Resilient autoscaling therefore needs multiple indicators and explicit behavior for degraded dependencies, not just normal traffic patterns.
Healthy automation needs failure-aware signals
A scaling policy should ask whether low CPU means spare capacity or blocked work. Queue depth, latency and active-thread utilization often tell a more complete story.
The emergency scale-up overloaded the service responsible for provisioning servers
Slack attempted to add roughly 1,200 web servers in minutes. Its internal provision-service had to talk to other Slack systems and AWS APIs over the same degraded network, causing it to hit Linux open-file limits and an AWS quota.[1] Thousands of instances were created but not fully configured, filling autoscaling-group limits without adding useful serving capacity. Recovery automation had become another overloaded dependency. This is why capacity systems must be tested at emergency scale, not only at ordinary growth rates.
Recovery services need more headroom than serving services
Provisioning, deployment and orchestration may be used most aggressively during incidents. Their capacity model should assume bursts larger than normal business operations.
Monitoring failed because it crossed the same network boundary
Slack’s dashboard and alerting service became unavailable during the incident because its front end and database were in different VPCs connected through the affected Transit Gateway.[1] Engineers still had logs and direct metric backends, but diagnosis became slower. Slack later planned to colocate those components to remove the dependency. Its public status record shows service gradually recovering between roughly 8:45 and 10:40 a.m. PST.[2]
Observability should degrade more slowly than production
Monitoring is part of the control system. If it fails first, responders must operate the platform with less information precisely when decisions carry the most risk.
AWS Transit Gateway scaling was the underlying network trigger
Slack reported that one AWS Transit Gateway did not scale quickly enough for the abrupt packet-per-second increase that followed the holiday lull and the first major workday back.[1] AWS engineers manually increased capacity during the incident. Transit Gateway exists to connect VPCs and networks through a shared hub.[3] Shared hubs simplify architecture, but they also create aggregation points whose scaling behavior must match the largest correlated traffic shift, not the daily average.
Load balancer panic mode helped Slack serve through imperfect health checks
Slack’s Envoy deployment supported a panic-routing behavior that could send traffic across hosts even when a large fraction were failing health checks.[4] During widespread network degradation, strict health filtering can remove so much capacity that the load balancer makes the outage worse. Panic modes acknowledge that an imperfect instance may be better than no instance. Graceful degradation often requires relaxing normal assumptions while preserving enough service to recover.
The incident changed both technical controls and seasonal planning
Slack committed to load-testing provision-service more aggressively, revisiting health checks and autoscaling, removing the monitoring dependency on the Transit Gateway, and requesting preemptive TGW scaling around the post-holiday traffic jump.[1] The lesson is organizational as well as technical: recurring calendar patterns can be infrastructure events. Capacity plans should model unusual but predictable transitions, such as everybody returning to work at once after the quietest week of the year.
Why Slack January 2021 belongs in software-failure history
Slack’s January 2021 outage belongs in software history because no single dramatic bug explains it. Network packet loss changed application latency; autoscaling briefly misread CPU; emergency scale-up overloaded provisioning; monitoring shared the degraded network; and a shared connectivity hub needed more capacity. The incident is a textbook feedback failure in a distributed SaaS system. The lasting lesson is to test how automation behaves when dependencies are slow rather than dead, because partial failure is exactly where reasonable local decisions—downscale, replace unhealthy hosts, add capacity—can combine into a global outage.[5]
Works Cited
- 01Slack Engineering — Slack’s Outage on January 4th 2021 slack.engineering
- 02Slack Status — January 4, 2021 Incident slack-status.com
- 03AWS Docs — What Is AWS Transit Gateway? docs.aws.amazon.com
- 04Slack Engineering — Migrating Millions of Concurrent Websockets to Envoy slack.engineering
- 05Slack Engineering — All Hands on Deck slack.engineering
CodeHistory is a living archive. Citations document the evidence used for this edition; later evidence may refine the account.