Data Lake

What it is

A data lake is a centralized storage repository that holds raw data at any scale — structured, semi-structured, and unstructured — in its native format until it is needed. Unlike a data warehouse (schema-on-write), a data lake applies schema at query time (schema-on-read), allowing data to be ingested without upfront modeling. The underlying storage is typically cloud object storage (AWS S3, Azure Data Lake Storage Gen2, Google Cloud Storage) with compute and query engines (Spark, Presto/Trino, Athena) running against the data on demand.

Modern data lakes use open table formats — Delta Lake (Databricks), Apache Iceberg (Netflix, Apple), and Apache Hudi (Uber) — layered on top of Parquet/ORC files in object storage. These formats add ACID transactions, schema evolution, time travel (query historical snapshots), and incremental processing capabilities to raw object storage, addressing the reliability and consistency problems of the original "raw files on S3" data lake approach. They are a key enabler of the data lakehouse architecture.

Why it exists

Traditional data warehouses require schema design before data can be ingested. In fast-moving environments where data scientists need to explore raw event logs, clickstream data, IoT sensor feeds, and semi-structured JSON before knowing what questions to ask, the schema-on-write model is a bottleneck. A data lake accepts any data in any format immediately, making it available for exploration while transformation and modeling work proceeds in parallel. Object storage also provides virtually unlimited capacity at very low cost ($0.02–0.025/GB/month for S3 Standard), far cheaper than cloud warehouse storage, making it practical to retain years of raw data that would be cost-prohibitive in a warehouse.

When to use

• Storing raw event logs, clickstream data, IoT sensor readings, and semi-structured data (JSON, Avro) that may not have a defined schema at ingestion time.
• As the raw ingestion layer feeding a downstream data warehouse — a "landing zone" that preserves the original data before transformation.
• Machine learning model training where data scientists need access to full raw datasets to experiment with feature engineering.
• Long-term archival of data that must be retained for compliance but is rarely queried and would be expensive to store in a warehouse.
• Multi-modal data storage: combining structured tables, JSON documents, images, audio, and video in a single storage tier.
• Ad-hoc, exploratory analysis by data scientists using Spark or Jupyter notebooks before productionizing queries.

When not to use

• As a replacement for a data warehouse for structured BI reporting — without dimensional modeling and a semantic layer, query performance and metric consistency suffer.
• When governance and access control are critical — a raw, unorganized data lake with no catalog or lineage tracking quickly becomes a data swamp where no one knows what data exists or whether it is trustworthy.
• For operational transactional workloads — object storage has high per-operation latency and is not suitable for OLTP access patterns.
• When all your data is structured and already modeled — the added complexity of a data lake is not justified if a data warehouse meets all requirements.

Typical architecture

  Medallion Architecture (Bronze / Silver / Gold):

  Sources           Landing          Refined        Curated
  ┌──────────┐     ┌────────────┐   ┌───────────┐  ┌───────────────┐
  │ App DB   │────▶│  BRONZE    │──▶│  SILVER   │─▶│    GOLD       │
  │ Events   │────▶│  (Raw)     │   │ (Cleaned, │  │ (Aggregated,  │
  │ IoT Data │────▶│  S3 / ADLS │   │  validated│  │  business     │
  │ Files    │────▶│  Parquet/  │   │  enriched)│  │  ready, ready │
  └──────────┘     │  JSON/CSV  │   │  Delta    │  │  for DWH/BI)  │
                   └────────────┘   │  Lake     │  └───────────────┘
                                    └───────────┘
                   Open Table Format (Delta / Iceberg / Hudi):
                   - ACID transactions on object storage
                   - Schema evolution (add/rename columns)
                   - Time travel (query as-of date)
                   - Incremental reads (only changed files)
          

Pros and cons

Implementation notes

Adopt the Medallion architecture (Bronze/Silver/Gold) to organize lake data into quality tiers. Bronze contains raw ingested data in its original format, archived immutably. Silver applies validation, deduplication, and light transformations. Gold is curated, aggregated data ready for BI consumption or warehouse loading. Use Apache Iceberg for new data lake tables due to its broad engine support (Spark, Flink, Trino, DuckDB, Snowflake external tables) and strong schema evolution. Iceberg's hidden partitioning avoids the common mistake of encoding partition columns into file paths (which requires rewriting all data to change partitioning).

File size management is critical for query performance: aim for Parquet files between 128MB and 512MB. Small files (many files under 1MB) degrade both write and query performance. Use Iceberg's rewrite_data_files procedure or Delta Lake's OPTIMIZE command to periodically compact small files. For GDPR right-to-erasure, use Iceberg's row-level delete capability to delete specific rows without rewriting entire partitions, then run expire_snapshots to remove old versions that contained the deleted rows.

Common failure modes

• Data swamp: Hundreds of datasets land in the lake with no catalog entries, no owners, and no quality documentation; data scientists spend more time finding and validating data than analyzing it.
• Small file problem: A streaming pipeline writes one Parquet file per event or per minute; after months, a table contains millions of tiny files; every query takes minutes just for file listing before any data is read.
• Concurrent writer corruption: Two Spark jobs write to the same S3 prefix simultaneously without using an open table format; one job overwrites the other's output, silently losing data.
• Uncontrolled PII ingestion: Raw data from production databases is ingested into the Bronze zone without PII masking; a data scientist with lake access inadvertently exposes customer data in a notebook exported to a shared drive.
• Schema drift: The source application adds a new JSON field or renames an existing one; the Spark job that reads the schema fails silently or loses the new field because the schema is hardcoded rather than inferred from the data catalog.

Decision checklist

• Have you adopted an open table format (Iceberg, Delta, Hudi) for ACID transactions and schema evolution rather than raw file writes?
• Is the Medallion architecture (Bronze/Silver/Gold) or equivalent zone structure defined with clear data quality standards per zone?
• Is a data catalog (AWS Glue Catalog, Apache Atlas, DataHub) populated with metadata for all datasets in the lake?
• Is PII classified and masked at Bronze ingestion, or is column-level access control enforced?
• Do you have a file compaction strategy to prevent small file accumulation?
• Are data quality checks (row counts, null rates, freshness) automated and monitored?

Example use cases

• Media streaming analytics: A streaming service ingests 500GB/day of clickstream events as JSON into S3 Bronze. Spark ETL converts events to Parquet in Silver, partitioned by date and event type. Data scientists query Silver in Jupyter notebooks using PySpark to build recommendation model features.
• IoT platform: An industrial IoT platform ingests sensor readings from 100,000 devices into ADLS Gen2 using Event Hubs Capture (Avro format). Azure Databricks Delta Lake pipelines transform readings into curated Gold tables. Power BI connects to Gold for operational dashboards.
• Compliance archive: All application logs, audit trails, and transaction records are retained in immutable S3 Bronze for 7 years at $0.004/GB/month (Glacier). Queries for compliance investigations use Athena with Iceberg time travel to reconstruct the exact state of data at any point in the past.

Related patterns

• Data Warehouse — Often co-exists with the data lake in a two-tier architecture where the lake feeds the warehouse.
• Data Lakehouse — The architectural evolution that adds warehouse-like ACID semantics directly on the lake storage layer.
• Data Cataloging — Essential companion to prevent the data lake becoming a data swamp.

Further reading

• Apache Iceberg Documentation
• Delta Lake Documentation
• Medallion Architecture — Databricks