Data Architecture Modern Data Stack

Data Lakehouse

An architecture that combines the low-cost storage and flexibility of a data lake with the reliability, governance, and performance of a data warehouse using open table formats.

⏱ 10 min read

What it is

A data lakehouse is an architectural approach that collapses the traditional two-tier data lake + data warehouse architecture into a single tier, by adding a transactional metadata layer (an open table format) on top of low-cost object storage to provide data warehouse-like semantics directly on lake storage. Key capabilities include ACID transactions, schema enforcement and evolution, time travel (query historical versions of data), streaming ingestion, and BI query performance comparable to a traditional warehouse.

The three dominant open table formats are Delta Lake (Databricks; transaction log stored in _delta_log JSON/checkpoint files alongside Parquet data files), Apache Iceberg (Netflix origin; vendor-neutral, supported by Spark, Flink, Trino, Athena, Snowflake external tables, DuckDB), and Apache Hudi (Uber origin; optimized for low-latency record-level upserts and CDC ingestion). Delta Lake is the default on Databricks; Iceberg has the broadest multi-engine support and is the recommended choice for new implementations.

Why it exists

The classic two-tier architecture (data lake + data warehouse) suffers from three problems: data duplication (the same data lives in both S3 and Snowflake/Redshift, at doubled storage cost), ETL complexity (pipelines are needed to move data from the lake to the warehouse, introducing latency and failure modes), and inconsistency (the lake and warehouse may diverge due to failed ETL runs). The lakehouse eliminates the ETL hop between lake and warehouse by making the lake itself a reliable, queryable analytical store. BI tools connect directly to Iceberg/Delta Lake tables via Spark SQL, Trino, or Athena, eliminating the warehouse tier entirely for teams whose performance requirements are met by optimized open table format queries.

When to use

When you are building a new data platform and want to avoid the cost and complexity of maintaining separate lake and warehouse tiers.
When ML and BI workloads access the same underlying data — data scientists use Spark against the same Iceberg tables that BI tools query via SQL.
When storage cost is a significant concern and you want to keep analytical data in cheap object storage rather than expensive warehouse storage.
When you need time travel (point-in-time queries, audit) or GDPR right-to-erasure (row-level deletes) on analytical data.
When you want to avoid vendor lock-in — Iceberg tables on S3 can be queried by Spark, Trino, Athena, DuckDB, and even Snowflake external tables without copying data.
For streaming + batch unified pipelines where Flink writes streaming data to Iceberg tables that are also read by batch Spark jobs.

When not to use

When a managed cloud warehouse (Snowflake, BigQuery) already meets performance, governance, and cost requirements — adding lakehouse complexity without clear benefit is over-engineering.
When your team lacks Spark/Flink expertise — operating a self-managed lakehouse requires significantly more engineering than a fully managed SaaS warehouse.
For very small data volumes (under 100GB) where the overhead of distributed compute and metadata management is unnecessary.
When sub-second BI query latency is required — a warehouse with dedicated compute (Snowflake XS cluster) will outperform a Trino/Iceberg query at the tail latency.

Typical architecture


  Data Lakehouse Architecture:

  Ingestion Layer        Storage Layer            Query / Serving
  ┌────────────────┐    ┌──────────────────────┐  ┌──────────────────┐
  │ Fivetran       │    │  Object Storage       │  │ Databricks SQL   │
  │ Debezium CDC   │───▶│  (S3 / ADLS / GCS)   │◀─│ Apache Spark     │
  │ Kafka + Flink  │    │                       │  │ Trino / Athena   │
  └────────────────┘    │  Open Table Format:   │  │ DuckDB           │
                        │  ┌─────────────────┐  │  └──────────────────┘
                        │  │  Delta / Iceberg│  │
                        │  │  Metadata Layer │  │  ┌──────────────────┐
                        │  │  - Transaction  │  │  │ BI Tools         │
                        │  │    log          │  │  │ Tableau / Looker │
                        │  │  - Schema       │  │◀─│ Power BI         │
                        │  │    registry     │  │  └──────────────────┘
                        │  │  - Snapshot     │  │
                        │  │    history      │  │  ┌──────────────────┐
                        │  └─────────────────┘  │  │ ML Platforms     │
                        │  Parquet Data Files   │◀─│ MLflow / SageMaker│
                        └──────────────────────┘  └──────────────────┘

Pros and cons

Pros

Eliminates data duplication between lake and warehouse — one copy of data in cheap object storage.
ACID transactions on object storage: concurrent writers and readers are safe; partial writes don't corrupt tables.
Time travel: query any historical snapshot of a table, enabling reproducible ML experiments and compliance audits.
Open format avoids vendor lock-in; data is queryable by any engine supporting Iceberg/Delta without copying.
Unified platform for batch ETL, streaming ingestion, BI queries, and ML feature engineering.

Cons

Higher operational complexity than a managed warehouse; requires expertise in Spark, distributed storage, and open table format internals.
Table maintenance (compaction, vacuuming, snapshot expiration) must be actively managed or query performance degrades.
Query performance for interactive BI is lower than a purpose-built warehouse with dedicated caching and result reuse.
Immature tooling compared to traditional warehouses; features like column-level security and fine-grained access control are more complex to configure.
Vendor ecosystem fragmentation: Delta Lake is optimized for Databricks; Iceberg on EMR behaves differently than on Databricks.

Implementation notes

For new implementations, choose Apache Iceberg for maximum engine portability. Databricks natively supports Iceberg via the UniForm feature (tables written as Delta are also readable as Iceberg). Configure Iceberg with the AWS Glue Catalog or a Hive Metastore as the catalog; this allows Athena, EMR, and Glue ETL jobs to share the same table definitions. Partition Iceberg tables using hidden partitioning: PARTITIONED BY (days(event_ts), truncate(16, customer_id)) rather than requiring consumers to know the partition scheme.

Databricks SQL Warehouse is the managed lakehouse query engine that provides sub-second BI query latency through result caching, pre-warmed Photon execution engine instances, and predictive I/O. For open-source deployments, Trino with Iceberg REST catalog provides comparable query performance. Schedule regular maintenance jobs: Iceberg rewrite_data_files (file compaction), rewrite_manifests (manifest compaction), and expire_snapshots (remove old versions beyond the retention period). Automate these with a weekly Airflow DAG or Databricks job cluster.

Common failure modes

Snapshot accumulation: Time travel is enabled but expire_snapshots is never run; metadata files accumulate indefinitely; table plan time grows from milliseconds to minutes as Iceberg scans thousands of snapshot manifests.
Small files at scale: A streaming pipeline writes micro-batches every 30 seconds; after a month, a table has 100,000 small Parquet files; compaction is not configured; queries perform 50x worse than expected.
Concurrent write conflicts: Two Spark jobs attempt to overwrite the same partition simultaneously; Iceberg's optimistic concurrency control fails one job; the failed job has no retry logic and the partition is silently not updated.
Catalog inconsistency: Parquet files are deleted from S3 directly (e.g., via S3 lifecycle policy) without going through Iceberg's table management; Iceberg metadata still references deleted files; queries fail with FileNotFoundException.
Schema evolution breakage: A consumer reads an Iceberg table with a hardcoded schema; a producer adds a new required column; the consumer's hardcoded schema omits the column and fails to deserialize new rows.

Decision checklist

Have you chosen a table format (Iceberg vs Delta) based on engine compatibility and your primary query engines?
Is automated table maintenance (compaction, snapshot expiration) scheduled as a recurring job?
Is the catalog (Glue, Hive Metastore, Nessie) shared across all query engines to avoid duplicate table registration?
Are schema changes (new columns, type changes) tested against all downstream consumers before being applied?
Is data access governed at the catalog level with row/column filtering policies (Apache Ranger, AWS Lake Formation)?
Have you benchmarked BI query latency against your SLA requirements to confirm a lakehouse can replace a dedicated warehouse?

Example use cases

Databricks lakehouse for e-commerce: Fivetran ingests source data into Bronze Delta tables on S3. dbt-Spark transforms data through Silver to Gold Delta tables. Databricks SQL Warehouse serves BI queries with sub-second latency via result caching. The same Gold tables feed ML feature engineering pipelines in Databricks Notebooks.
Streaming + batch unified: Apache Flink writes CDC events from Debezium into Iceberg tables on S3 in near real time. Apache Spark batch jobs run nightly transformations on the same Iceberg tables. Trino provides interactive SQL queries with no data copy between streaming and batch layers.
Multi-cloud open format: A company stores its canonical data as Iceberg tables on S3. AWS Athena is used for ad-hoc queries. Snowflake external tables read the same Iceberg metadata for BI reports. DuckDB is used by data scientists for local development. No data is copied between systems.

Data Lake — The storage foundation of the lakehouse; understanding lake challenges motivates the lakehouse approach.
Data Warehouse — The traditional alternative; understand when a managed warehouse is still preferable.
ETL vs ELT — The lakehouse architecture favors ELT (transform in-place) and reduces the need for warehouse-specific ETL.