CDN77 details ZooKeeper performance hit caused by stale data parts
CDN77 engineering details an incident involving a performance degradation in their ClickHouse data pipeline caused by unexpected ZooKeeper load. The issue was triggered by a specific combination of stale client machines sending unmerged data, which led to excessive partitioning and heavy coordination traffic.
Key Takeaways
- ZooKeeperWait metrics spiked to thousands of seconds across three redundant instances due to excessive coordination traffic.
- Root cause was two client machines rejoining the network after months-long outages, sending large queues of unmerged, tiny batches.
- ClickHouse partitioning on a five-minute unit for detail tables amplified the issue, creating 288 replicated parts for every historical day inserted.
- Standard 'too many parts' insert limits (5,000) were never tripped, as the issue manifested in the background replication and TTL merge processes.
- Implementation of out-of-range timestamp rejection in root MaterializedViews now prevents incoherent data from entering the production pipeline.
Why It Matters
In high-concurrency streaming and CDN environments, metadata coordination is often the silent bottleneck that fails before compute or storage resources reach their limits. This incident underscores that even minor edge disruptions—like two machines rejoining a network—can trigger a coordination storm if the database partitioning logic isn't sanitized for historical drift. For engineers, it highlights the 'long tail' risk of time-series data where stale timestamps multiply background management tasks exponentially. As industry reliance on ClickHouse for real-time analytics grows, implementing timestamp boundaries and migrating to ClickHouse Keeper (C++ native) for lower-overhead coordination will become table stakes for maintaining pipeline stability. Watch for CDN77 to further refine its automated data sanitization and expand its internal 'perfect storm' testing scenarios.
Additional Context
The incident at CDN77 aligns with broader challenges in the ClickHouse ecosystem regarding ZooKeeper scalability. Per OneUptime in March 2026, many production clusters are suffering from 'metadata bloat' where thousands of small tables or excessive partitions create a ZooKeeper bottleneck that presents as mysterious replication lag. To address these observability gaps, ClickHouse version 26.1, released in February 2026, introduced a specialized 'system.aggregated_zookeeper_log' and an embedded web dashboard for real-time Keeper introspection. These tools are designed to surface the specific 'coordination wait' patterns that CDN77's team had to infer manually during their March incident.
In the wider CDN market, infrastructure reliability has shifted from a pure performance metric to a critical risk-management priority. According to research from BlazingCDN in May 2026, the global market is increasingly split between 'legacy' providers optimized for broad global delivery and specialized media CDNs like CDN77 that prioritize deep unit economics and high-volume data analytics. This operational shift comes as Sandvine reported in 2024 that video continues to represent roughly 38% of global internet traffic. For providers, this sustained volume means that even minor pipeline inefficiencies can cause massive technical debt. Consequently, industry leaders are increasingly adopting 'self-healing' data pipelines that include mandatory incident reporting and automated rejection of non-conforming edge logs to avoid the type of 'perfect storm' coordination failures described by CDN77.
Read full article at cdn77.com
Enjoy our coverage?
Add StreamingMeme as a preferred source on Google to see more of our streaming news at the top of your Search results.
Add as preferred source