Discord 17% session drop caused by single-threaded Erlang bottleneck
Discord released a technical post-mortem detailing how a single-threaded Erlang supervisor process within its real-time infrastructure caused a significant session outage. Despite horizontal scaling across 25,000 Selective Forwarding Units, the mailbox of this single process grew to one million messages, creating a bottleneck that halted RPC command processing.
Key Takeaways
- A single-threaded Erlang supervisor process failed to drain a mailbox that reached 1,000,000 messages during a recovery wave.
- The outage impacted 17% of total Discord sessions, leaving users stuck on the 'Awaiting Endpoint' status for over three hours.
- Discord's infrastructure fanout involves 15 voice syncer instances communicating with a global fleet of 25,000 Selective Forwarding Units (SFUs).
- Horizontal scaling failed to prevent the outage because every healthy replica contained the same single-threaded process bottleneck.
Why It Matters
This incident highlights a critical vulnerability in high-scale streaming architectures: the 'hidden singleton' within distributed systems. While Discord scaled its Selective Forwarding Units to massive levels, the underlying Erlang supervisor architecture created a serial bottleneck that horizontal replicas could not bypass. For the broader B2B streaming industry, this serves as a technical warning that parallelization at the Kubernetes layer does not guarantee concurrency at the process level. Operators of real-time video infrastructure must now audit their stateful session management to ensure recovery waves do not trigger similar 'thundering herd' failures. Watch for Discord's transition toward PartitionSupervisor models to decouple its connection pools and improve concurrent message handling.
Additional Context
The March 25, 2026, outage triggered a multi-day period of instability for Discord, with subsequent incidents occurring through March 28 that affected API reliability and bot connectivity. Per InfoQ (May 2026), the engineering team identified that a routine configuration change caused a simultaneous shutdown of session management servers, which then overwhelmed downstream service discovery. This pattern of recovery becoming the primary source of failure is a known risk in stateful distributed systems, where the act of rebuilding millions of concurrent sessions generates more load than the steady-state infrastructure is designed to handle.
Industry data from Confluent's 2026 Data Streaming Report indicates that 72% of IT leaders now cite insufficient real-time infrastructure as the primary barrier to scaling complex digital services. Discord’s reliance on Erlang—a language chosen for its fault tolerance—illustrates that even robust technologies require architectural evolution to manage the massive fanout required by modern WebRTC applications. According to technical analysis by statusfield.com (March 2026), these types of cascading failures are becoming more frequent as streaming platforms move away from hyperscale cloud providers toward custom, hybrid infrastructures to manage rising egress costs and latency demands.
In response to the bottleneck, Discord engineers Bo Ingram and Stephen Birarda detailed a shift toward replacing the standard Holster pool supervisors with PartitionSupervisors. This change allows the system to balance outbound HTTPS requests across multiple independent supervisors, preventing a single process mailbox from halting the entire RPC command chain. As real-time video platforms continue to scale, the industry is increasingly moving toward 'resilience-by-design' frameworks that prioritize process isolation and sub-second recovery metrics to mitigate the impact of massive session reconnections.
Read full article at medium.com
Enjoy our coverage?
Add StreamingMeme as a preferred source on Google to see more of our streaming news at the top of your Search results.
Add as preferred source