AWS launches OpenSearch engine promising 70% reduction in log storage costs
AWS has launched a specialized analytics engine for its managed OpenSearch Service that utilizes Apache Parquet and DataFusion to reduce telemetry storage costs by a reported 70%. While the update addresses the high infrastructure demands of AI and streaming log monitoring, analysts note that the requirement for new domain deployment and the lack of DSL support may complicate adoption for existing users.
Key Takeaways
- Storage costs are reduced by a reported 70% by moving from traditional JSON-based inverted indexes to columnar Apache Parquet storage.
- The new engine doubles data ingestion throughput and improves price-performance by 4x using Apache DataFusion and Calcite for query routing.
- Adoption requires deploying a new domain as the engine cannot be enabled on existing indices or domains within the managed service.
- Workloads relying on OpenSearch Domain Specific Language (DSL) must be rewritten, as the new engine only supports SQL and Piped Processing Language (PPL).
Why It Matters
This release directly addresses the 'observability tax' that streaming and AI companies face as telemetry volumes outpace infrastructure budgets. By decoupling storage format from the query interface, AWS allows teams to retain logs longer for incident response without the typical cost scaling of Lucene-heavy architectures. For the streaming ecosystem, this shifts the focus from 'data sampling' to 'complete retention,' but the lack of DSL support creates a significant migration hurdle for mature DevOps pipelines. Watch for whether third-party observability vendors like Datadog or New Relic adjust their ingestion pricing to counter this competitive AWS native advantage.
Additional Context
The launch comes as enterprises struggle with an explosion in operational data. Per a June 2026 report from Dynatrace, AI workloads have driven a 93% increase in log volume over the last 12 months. This surge has forced organizations to exclude an average of 86% of their log data to stay within budget, often leaving teams 'blind' during security or performance incidents. The same study found that companies now spend an average of $2.5 million annually on log management, storage, and indexing across approximately seven different tools. The competitive landscape for log analytics has intensified as a result. While AWS is optimizing its managed service, rival Elastic has aggressively marketed its own cost-efficiency gains. In February 2024, Elastic published findings claiming that customers moving from AWS OpenSearch to Elastic Cloud achieved storage cost reductions of up to 58% through more efficient resource utilization. Additionally, as of June 2026, market data from PeerSpot indicates that Datadog maintains a lead in observability mindshare at 4.6% compared to OpenSearch Service at 1.1%, despite the latter's deep integration with the AWS infrastructure stack. Technically, the industry is moving toward vendor-neutral governance to spur innovation. In September 2024, AWS transferred the governance of the OpenSearch project to the Linux Foundation, establishing the OpenSearch Software Foundation (OSSF). This move was intended to foster broader community contributions to the core engine, such as the SIMD optimizations and concurrent segment search mentioned in 2024 development summaries, which laid the groundwork for the high-performance throughput seen in the latest log-optimized release.
Read full article at infoworld.com
Get this in your inbox → Subscribe
Enjoy our coverage?
Add StreamingMeme as a preferred source on Google to see more of our streaming news at the top of your Search results.
Add as preferred source