Researchers develop MLT-Dedup to slash video repetition by 91 percent

The immediate implication is a significant reduction in egress and storage costs for UGC platforms facing exponential volume growth. By optimizing indexing capacity fivefold, platforms can broaden candidate coverage without a proportional increase in server spend. Within the broader ecosystem, this shifts deduplication from simple file-matching to sophisticated temporal segment analysis, crucial for rights management as AI-assisted video editing proliferates. To track progress, watch for the integration of sparse-embedding retrieval in commercial vector databases like Milvus to handle 100-billion-scale video search workloads.

MLT-Dedup was developed by TikTok's Content Intelligence team and accepted to the Applied Data Science track at the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining, held August 9 through 13, 2026, in Jeju Island, South Korea. The paper's author list spans TikTok offices in Singapore and San Jose, with collaborators from the National University of Singapore's School of Computing, signaling that the work represents a production deployment rather than a purely academic exercise. The KDD ADS track specifically selects papers demonstrating real-world impact at scale, which aligns with the team's claim of live A/B testing on a large short-video platform.

The framework's architecture combines a Multi-Level Video Encoder that produces both sparse 768-dimensional clip-level embeddings for retrieval and fine-grained 256-dimensional frame-level features for precise matching, achieving 87.59 Recall@5 with sub-200-millisecond added latency in online serving. The trade-off is a 23% increase in GPU capacity and 14% more VRAM compared to the Swin-Large baseline, a cost that the authors argue is offset by the fivefold index expansion and 91% repetition reduction. The DiF-SiM matching module locates duplicated temporal segments between video pairs, enabling policy-driven decisions rather than binary remove-or-keep outcomes.

Within the video deduplication research landscape, MLT-Dedup addresses a gap that prior systems left unresolved: retrieving sufficient candidates under fixed memory budgets while maintaining precision against partial edits, watermarks, cropping, and temporal shifts. The paper's ablation results show that combining ML-VE with DiF-SiM yields the full 91% repetition reduction, whereas either component alone achieves only 72% or 24% respectively, demonstrating that the multi-level representation and the differential matching module are complementary rather than redundant. For platforms evaluating deduplication pipelines, this suggests that investment in both retrieval efficiency and fine-grained temporal matching is necessary to approach production-grade deduplication at UGC scale.