Netflix’s Per-Second Index Turns Raw Footage Into Searchable Intelligence

Netflix describes a production-oriented multimodal video search architecture designed to index and retrieve moments from large volumes of raw footage by fusing outputs from multiple AI models (e.g., character, scene, dialogue) into a unified, time-aligned representation. The system persists raw annotations in an internal annotation service backed by Apache Cassandra, performs offline temporal bucketing and intersection via Kafka-triggered processing, and indexes enriched per-second records into Elasticsearch to enable low-latency hybrid text+vector search and ranking. The post also outlines query planning, result deduplication/clustering, and future work including natural-language querying, adaptive ranking from user feedback, and workflow-specific personalization.