# ADR-001: Adoption of Apache Iceberg for Lakehouse Storage Format * **Status:** Accepted * **Date:** 2026-03-15 * **Deciders:** Principal Data Architect, Platform Engineering Team ## Context & Problem Statement Our analytical lakehouse currently faces multiple concurrency and table maintenance challenges with Hive metastore and raw parquet partitions: - Concurrent batch writes and streaming micro-batches occasionally cause dirty reads and partition locks. - Schema evolution (such as renaming or dropping columns) requires costly full-table rewrites. - Hidden partitioning is required so data consumers do not need to construct explicit partition pruning WHERE clauses. - Multi-engine querying across Spark, Trino, Flink, and Snowflake is a strict business requirement. ## Decision Outcome We will adopt **Apache Iceberg** as the standard table format for all Bronze, Silver, and Gold lakehouse tables on AWS S3 / Cloud Storage. All ETL pipelines and streaming sinks (Spark Structured Streaming, Flink) will write Iceberg v2 metadata with hidden partitioning, snapshot isolation, and ACID transactions. ## Consequences ### Positive Drivers - ✅ Full ACID transactions with serializable snapshot isolation, eliminating dirty reads. - ✅ Safe, fast schema evolution (rename, drop, add, reorder) without data rewrites. - ✅ Time travel queries and point-in-time rollbacks for disaster recovery. - ✅ Vendor-neutral multi-engine interoperability across Spark, Trino, DuckDB, Snowflake, and BigQuery. ### Trade-offs & Negative Impacts - ⚠️ Requires automated maintenance jobs (compaction, snapshot expiry, orphan file cleanup). - ⚠️ Team must learn Iceberg catalog management (AWS Glue / Project Nessie / Polaris). ## Alternatives Considered ### Delta Lake (Open Source) Excellent Spark support, but slightly more complex cross-engine integration with non-Databricks engines at the time of evaluation. ### Apache Hudi Great for CDC record-level updates, but higher metadata complexity and steeper learning curve for our team. --- *Generated by InsightfulSaga Architecture Decision Record (ADR) Studio*