AutoMQ and Amazon FSx ONTAP cut Kafka streaming costs by 94%
AutoMQ has integrated Amazon FSx for NetApp ONTAP as a Write-Ahead Log (WAL) layer for its Diskless Kafka, achieving sub-10ms latency and 94% cost savings. This solution targets streaming professionals by offering low-latency, Multi-AZ resilience, and reduced cross-AZ data transfer costs for Kafka message queues. It compares favorably to traditional Kafka deployments on AWS in terms of performance and cost.
Key Takeaways
- Achieved average write latency of 5.98ms and P99 latency of 12.87ms in us-east-1 benchmarking.
- Integrated FSx for NetApp ONTAP to provide multi-AZ resilience without cross-AZ data transfer fees.
- Eliminated inter-broker replication traffic by using a single-replica shared storage model on S3.
- Reduced projected monthly costs for 1 GBps ingestion from $317,092 to approximately $18,345.
- Decoupled compute and storage, allowing Kafka brokers to scale as stateless nodes.
Why It Matters
This technical development directly addresses the high cost of cross-AZ data replication, which often serves as the primary driver of cloud bills for high-throughput video streaming platforms. By replacing local-disk replication with a shared file system (FSx) and low-cost object storage (S3), providers can achieve the low latency required for real-time recommendation engines and ad-insertion metadata without the traditional 3x storage overhead. For the broader ecosystem, this validates a shift toward diskless architectures that treat cloud storage as the source of truth. Engineers should track the adoption of similar diskless patterns in upcoming Apache Kafka 4.0 releases and KIP-1150 to gauge long-term industry standardization.
Additional Context
The move toward diskless architectures reflects a broader industry pivot as the Apache Kafka community formalizes support for cloud-native storage patterns. Per reports from Kai Waehner in December 2025, diskless Kafka and projects like Apache Iceberg are creating a new foundation for unified, cost-effective storage that merges real-time and historical data. This trend is gaining traction as enterprises prioritize reducing the total cost of ownership (TCO) for data streaming, which 86% of IT leaders now cite as a top strategic investment, according to Confluent's 2025 Data Streaming Report. Since Confluent's acquisition of WarpStream in 2024 and its subsequent $11 billion acquisition by IBM in March 2026, the market for diskless solutions has bifurcated between managed SaaS offerings and independent providers like AutoMQ.
Technological maturity in the underlying cloud infrastructure has accelerated this transition. In April 2026, AWS expanded its second-generation Amazon FSx for NetApp ONTAP to four additional regions, including London and Sao Paulo, providing up to 72 GBps of throughput and multi-AZ high availability. This infrastructure expansion is critical for global streaming platforms that require regional compliance and minimal recovery times. Furthermore, the integration of S3 Access Points for FSx for ONTAP, introduced by AWS in 2025, allows stored data to be used natively by AI and analytics services like Amazon Bedrock. This accessibility shifts the role of streaming infrastructure from a passive data transporter to an active context provider for real-time AI inference and automated content personalization.
Read full article at aws.amazon.com
Enjoy our coverage?
Add StreamingMeme as a preferred source on Google to see more of our streaming news at the top of your Search results.
Add as preferred source