Apache Iceberg V4 spec to deprecate equality deletes for deletion vectors
Apache Iceberg contributors are currently discussing the deprecation of equality deletes in the upcoming V4 specification, citing performance benefits of transitioning to deletion vectors. This work coincides with Flink's implementation of conversion tooling and broader structural updates to the lakehouse format to support AI-centric data types and collation standards.
Key Takeaways
- Iceberg V4 proposal seeks to remove equality deletes, favoring O(1) position checks via V3 deletion vectors.
- Maximilian Michels reported Flink's 'ConvertEqualityDeletes' task is complete, removing a key blocker for deprecation.
- Iceberg Rust 0.10.0 RC4 entered voting, serving as the high-performance core for the PyIceberg ecosystem.
- First-class vector support proposal for Iceberg aims to standardize dense numeric types like float[768] for AI workloads.
- New RC0 for the Apache Iceberg Terraform Provider brings infrastructure-as-code capabilities to catalog management.
Why It Matters
The removal of equality deletes fundamentally resolves the conflict between streaming-write efficiency and read-time performance. For the streaming video industry, which relies on high-velocity telemetry and user-state Change Data Capture (CDC), this transition ensures that 'frozen' snapshots remain performant for large-scale analytics without the constant 'tax' of row-level joins during scans. By forcing a move to deletion vectors, Iceberg is effectively maturing from a flexible data lake format into a predictable, warehouse-grade storage layer. Industry leaders must now watch the adoption of these specs by major engines like Snowflake and Trino, as lack of V4 alignment could create a significant performance divide across the lakehouse ecosystem.
Additional Context
The push toward Iceberg V4 coincides with broader industry efforts to standardize open table formats for AI and real-time streaming. Per Snowflake, May 2026, the general availability of Support for Iceberg V3 introduced foundational components like row lineage and native 'variant' types, which are already being utilized by data platforms to manage semi-structured metadata. Oracle also extended vector embedding support to external Iceberg tables in March 2026, highlighting a growing demand for the format to handle high-dimensional AI data without requiring intensive data movement. Concurrently, the metadata layer is undergoing a radical redesign to support high-frequency commits typical of low-latency streaming applications. Recent community discussions, per ByteIota in July 2026, suggest V4 will include 'Adaptive Metadata Trees' to replace the existing manifest hierarchy. This change aims to collapse several metadata files into single-file commits, directly addressing the 'small file problem' that often plagues Flink-based streaming architectures. These structural updates, alongside the formalization of the Iceberg REST Catalog protocol, are positioning the project to serve as the universal interoperability layer between diverging cloud data warehouses and independent processing engines.
Read full article at dev.to
Get this in your inbox → Subscribe
Enjoy our coverage?
Add StreamingMeme as a preferred source on Google to see more of our streaming news at the top of your Search results.
Add as preferred source