Netflix details eBPF and Kafka pipeline for real-time service mapping
Netflix engineering has released a technical breakdown of their real-time service topology mapping system. The architecture utilizes eBPF flow logs, Kafka, and reactive streams with backpressure to provide developers with near real-time dependency visibility at massive scale.
Key Takeaways
- Architecture utilizes eBPF flow logs for kernel-level connection visibility and IPC metrics for application-specific endpoint data.
- Streaming pipeline leverages reactive streams with backpressure to prevent memory crashes during traffic spikes or GC pauses.
- Physical separation of graph layers allows the Network and IPC layers to be optimized and queried independently.
- Three-stage aggregation process resolves multi-hop network paths into direct application-to-application edges.
- System achieves sub-second query response times while ensuring topology data remains fresh within minutes, replacing hourly batch jobs.
Why It Matters
Real-time topology mapping is becoming a critical infrastructure requirement as streaming platforms transition toward live sports and 24/7 events that demand immediate incident response. By shifting from batch-processed 'archaeology' to streaming-first visibility, Netflix addresses the context gap inherent in traditional metrics and logs. This architecture demonstrates how eBPF can provide ground-truth connectivity without the overhead of manual instrumentation. As industry observability trends shift toward unified context, this model serves as a blueprint for maintaining reliability in highly fragmented, large-scale microservice environments. Watch for Netflix to integrate deployment change events into this graph to enable automated root cause analysis.
Additional Context
The rollout of the Service Topology system coincides with Netflix's aggressive expansion into live broadcasting, including the World Baseball Classic in March 2026 and a significant increase in live event frequency compared to 2024. Per InfoQ, June 2026, the underlying processing pipeline runs on Apache Pekko Streams (a fork of Akka) and persists data in a custom graph database layer built on Netflix’s internal distributed key-value storage. This infrastructure is essential for managing a footprint that now exceeds 2 trillion events per day across thousands of brokers. External reporting highlights that Netflix is managing this scale by using a two-tier Kafka cluster layout, separating ingest-focused 'Fronting' clusters from read-optimized 'Consumer' clusters. Per Factor House, June 2026, this multi-cluster strategy is specifically designed to limit blast radius and align with AWS stateless infrastructure. While traditional observability typically relies on the 'three pillars' of logs, metrics, and traces, analysts at Medium noted in June 2026 that the industry is shifting toward service maps to solve the problem of being 'signal rich but context poor.' Furthermore, according to Dev.to, March 2026, Netflix’s Data Mesh platform has matured to support over 1,200 SQL processors created by non-infrastructure teams, democratizing the ability to build real-time monitoring pipelines. This internal democratization, paired with the service map's gRPC API, allows diverse engineering teams to perform 'time-travel' queries to view historical topology states without the high storage costs associated with traditional snapshots. The system's sub-second response time requirement is a direct response to the operational pressures of maintaining 99.99% availability for 300 million global subscribers.
Read full article at netflixtechblog.com
Get this in your inbox → Subscribe
Enjoy our coverage?
Add StreamingMeme as a preferred source on Google to see more of our streaming news at the top of your Search results.
Add as preferred source