Netflix details eBPF and Kafka pipeline for real-time service mapping

Real-time topology mapping is becoming a critical infrastructure requirement as streaming platforms transition toward live sports and 24/7 events that demand immediate incident response. By shifting from batch-processed 'archaeology' to streaming-first visibility, Netflix addresses the context gap inherent in traditional metrics and logs. This architecture demonstrates how eBPF can provide ground-truth connectivity without the overhead of manual instrumentation. As industry observability trends shift toward unified context, this model serves as a blueprint for maintaining reliability in highly fragmented, large-scale microservice environments. Watch for Netflix to integrate deployment change events into this graph to enable automated root cause analysis.

The rollout of the Service Topology system coincides with Netflix's aggressive expansion into live broadcasting, including the World Baseball Classic in March 2026 and a significant increase in live event frequency compared to 2024. Per InfoQ, June 2026, the underlying processing pipeline runs on Apache Pekko Streams (a fork of Akka) and persists data in a custom graph database layer built on Netflix’s internal distributed key-value storage. This infrastructure is essential for managing a footprint that now exceeds 2 trillion events per day across thousands of brokers.

External reporting highlights that Netflix is managing this scale by using a two-tier Kafka cluster layout, separating ingest-focused 'Fronting' clusters from read-optimized 'Consumer' clusters. Per Factor House, June 2026, this multi-cluster strategy is specifically designed to limit blast radius and align with AWS stateless infrastructure. While traditional observability typically relies on the 'three pillars' of logs, metrics, and traces, analysts at Medium noted in June 2026 that the industry is shifting toward service maps to solve the problem of being 'signal rich but context poor.'

Furthermore, according to Dev.to, March 2026, Netflix’s Data Mesh platform has matured to support over 1,200 SQL processors created by non-infrastructure teams, democratizing the ability to build real-time monitoring pipelines. This internal democratization, paired with the service map's gRPC API, allows diverse engineering teams to perform 'time-travel' queries to view historical topology states without the high storage costs associated with traditional snapshots. The system's sub-second response time requirement is a direct response to the operational pressures of maintaining 99.99% availability for 300 million global subscribers.